We are hosting a hands-on tutorial introducing the NCI Data Catalogue and its two indexing systems: Intake-ESM and Intake-Spark. The session also introduces NCI’s scalable data platform for interactive exploration and analysis of large-scale climate, weather, and geoscience datasets. Through hands-on exercises, participants will explore the NCI Data Catalogue, understand its indexing schemes, and apply them directly in data analysis and AI/ML workflows.

The NCI Data Catalogue provides a unified framework for discovering and accessing a wide range of scientific datasets across NCI’s published data collections, including - but not limited to - geoscience, climate, weather and earth system data. It uses structured metadata for consistent dataset discovery, while its indexing layer enables fast and scalable search across large data holdings.

Built with the Python Pangeo ecosystem, these tools support efficient, memory-friendly workflows. Intake-Spark provides Parquet-based indexing for high-performance Spark queries, while Intake-ESM provides lightweight CSV-based indexing designed for Earth system and geoscience workflows.

For any questions, please contact training.nci@anu.edu.au.

Who should attend

This tutorial is ideal for researchers working with large-scale scientific and earth system datasets who want to improve data discovery and analysis efficiency.

Prerequisites

Basic experience with Python (Programming with Python).
Basic experience with bash or similar Unix shells. (Software Carpentry - The Unix Shell )
Experience using NCI ARE service is recommended. You can find relevant documentations here: ARE User Guide.

Learning Outcomes

After this training session, you will be able to

Learn about NCI data services
Understand NCI data catalogue and schemes
Perform search, load, and filter datasets efficiently from the /g/data collection
Can use data catalog and indexing tools in data analysis and machine learning workflows.

Topics Covered

Welcome and Introduction to NCI’s Intake-Spark and Intake-ESM Indexing Schemes
Overview of NCI’s Data Catalogue Services
Working with the Intake-ESM Indexing Scheme
Applying the Intake-ESM Scheme in AI/ML Workflows
Exploring Earth system dataset with NCI’s Scalable Data Exploration Tools

Contact the host of Discovering Earth System Datasets with the NCI Data Catalogue and Intake Indexing Tools

There were 0 errors found in the information you submitted: