Discovering Earth System Datasets with the NCI Data Catalogue and Intake Indexing Tools

We are hosting a hands-on tutorial introducing the NCI Data Catalogue and its two indexing systems: Intake-ESM and Intake-Spark. The session also introduces NCI’s scalable data platform for interactive exploration and analysis of large-scale climate, weather, and geoscience datasets. Through hands-on exercises, participants will explore the NCI Data Catalogue, understand its indexing schemes, and apply them directly in data analysis and AI/ML workflows.

The NCI Data Catalogue provides a unified framework for discovering and accessing a wide range of scientific datasets across NCI’s published data collections, including - but not limited to - geoscience, climate, weather and earth system data. It uses structured metadata for consistent dataset discovery, while its indexing layer enables fast and scalable search across large data holdings.

Built with the Python Pangeo ecosystem, these tools support efficient, memory-friendly workflows. Intake-Spark provides Parquet-based indexing for high-performance Spark queries, while Intake-ESM provides lightweight CSV-based indexing designed for Earth system and geoscience workflows.

For any questions, please contact training.nci@anu.edu.au.

Who should attend

This tutorial is ideal for researchers working with large-scale scientific and earth system datasets who want to improve data discovery and analysis efficiency.

Prerequisites

1. Basic experience with Python (Programming with Python).

2. Basic experience with bash or similar Unix shells. (Software Carpentry - The Unix Shell )

3. Experience using NCI ARE service is recommended. You can find relevant documentations here: ARE User Guide.

Learning Outcomes

After this training session, you will be able to

• Learn about NCI data services

• Understand NCI data catalogue and schemes

• Perform search, load, and filter datasets efficiently from the /g/data collection

• Can use data catalog and indexing tools in data analysis and machine learning workflows.

Topics Covered

• Welcome and Introduction to NCI’s Intake-Spark and Intake-ESM Indexing Schemes

• Overview of NCI’s Data Catalogue Services

• Working with the Intake-ESM Indexing Scheme

• Applying the Intake-ESM Scheme in AI/ML Workflows

• Exploring Earth system dataset with NCI’s Scalable Data Exploration Tools