Managing scientific datasets, their relationships, and metadata across research workflows can be complex and error-prone. The Scientific Dataset Catalog provides a centralized system for tracking datasets, their lineage, collections, and rich metadata throughout the research lifecycle.
This system helps you organize datasets into collections, track how datasets relate to each other (lineage relationships), store rich metadata, and provides a Python client for programmatic access to all these capabilities.
Want to integrate dataset catalog functionality into your Python workflows? β Quick Start | Full Documentation
Want to understand dataset metadata structure and relationships? β Schema Documentation
New to the catalog, or registering data from a pipeline? Install the catalog plugin for guided onboarding and registration scripts.
β Installation
Want to contribute to the codebase? β Development Guide
This repo ships a Claude Code plugin, catalog, distributed through the dataset-catalog marketplace defined in .claude-plugin/marketplace.json. It helps a scientist with no catalog experience understand the catalog, map their data onto it, write correct metadata, and register data from code that runs in their own pipeline. Claude writes the registration script; you own it and run it. Nothing in the plugin registers data on its own.
| Component | Type | What it's for |
|---|---|---|
catalog-docs |
MCP server | The catalog docs (Mintlify), installed with the plugin: concepts, field meanings, troubleshooting, auth how-to. Claude looks facts up here. |
catalog-guide |
skill | First-run guide (asks what you want to do, checks your setup, explains the catalog in your terms, gets you to one success in a dev or staging catalog; offered automatically on your first session), the rules that apply to every catalog task, and how to use the docs. |
registration-script |
skill | How to produce correct records from code: order of operations, search patterns for duplicates and lineage parents, checksums, governance decisions, the pipeline-safety rules, the script template, and how to read field types and allowed values from the installed client's models. |
script-writer |
agent | Writes or changes a registration script from the template and dry-runs it against a dev or staging catalog. |
metadata-reviewer |
agent | Read-only review of a draft record; returns blocking issues and warnings. |
registration-debugger |
agent | Diagnoses a failed registration and proposes a fix. |
| hooks | hooks | Session start: onboarding offer and a readiness check that names the catalog environment and flags problems. Before tool use: blocks a registration run against a production (or unrecognized) URL without a passing dry run against that URL, blocks writing a literal API token into a file, and auto-approves read-only catalog commands. After writes: lints generated scripts against the pipeline-safety rules. |
| Command | Does |
|---|---|
/catalog:onboard |
Starts the onboarding steps in catalog-guide. |
/catalog:register [path] |
Asks about your data one question at a time, then has script-writer produce a script and dry-run it against a dev or staging catalog. |
/catalog:debug [error] |
Hands a failure to registration-debugger. |
/catalog:whoami |
Checks CATALOG_API_URL/CATALOG_API_TOKEN are set, creates ./.venv with uv if there is no Python environment, then shows environment (from CATALOG_API_URL), identity, read/write permission, and token expiry. Never prints the token. |
- Python 3.11+ (
python3on yourPATH; the plugin's own scripts need nothing else). - Access to the catalog docs MCP (
https://docs.biohub.ai/authed/mcp); it requires login, so authenticate it from/mcpon first use. Without it (for example on an HPC node with no internet) the plugin still works and points you to the docs URL. - For registration scripts and the
catalogCLI:catalog-client1.3.0 or newer, installed from a release tag (theregistration-scriptskill has the steps). CATALOG_API_URLandCATALOG_API_TOKENexported from your shell init file (never typed into chat, never in Claude Codesettings.json); see the catalog docs auth pages for how to store the token. It is the same pair thecatalogCLI and SDK read, and whatever catalog it names is the one used: there is no default. The environment (dev,staging, orprod) is read from the URL's host; a host naming none is treated as production. Start with a dev or staging URL. Variable names are in.env.example.
Issue tokens at <catalog URL>/tokens in a logged-in browser.
From inside a Claude Code session:
/plugin marketplace add chanzuckerberg/dataset-catalog
/plugin install catalog@dataset-catalog
Or from your terminal (outside a session):
claude plugin marketplace add chanzuckerberg/dataset-catalog
claude plugin install catalog@dataset-catalogInstalled mid-session? Run /reload-plugins to load the skills, commands, agents, and hooks now. The onboarding offer comes from a session-start hook, so it appears at the start of your next session (or run /catalog:onboard right away). Confirm with claude plugin list. Organisation-managed settings can disable plugin hooks; the skills and commands still work without them.
From inside a Claude Code session:
/plugin marketplace update dataset-catalog
/plugin # opens the plugin menu β pick "Update" for catalog@dataset-catalog
Or from your terminal:
claude plugin marketplace update dataset-catalog
claude plugin update catalog@dataset-catalogmarketplace update pulls the latest marketplace definition; plugin update (or the /plugin menu) upgrades catalog@dataset-catalog to the newest published version. Run /reload-plugins inside a session to apply it (a new session picks it up automatically). Your onboarding status survives updates; uninstalling the plugin removes it. Plugin versions are managed by release-please β see the plugin CHANGELOG.
A plugin cannot ship permission rules, so add deny rules to your project's .claude/settings.json (or your user settings) to keep Claude from reading credential files or dumping the environment. Adjust paths to where you keep secrets, and check the result with /permissions:
{
"permissions": {
"deny": [
"Read(./.env)",
"Read(./.env.*)",
"Bash(printenv *)",
"Bash(env)"
]
}
}A deny rule cannot tell a production URL from a staging one, so production protection comes from two things: the plugin's guard hook (a real run against a prod or unrecognized URL needs a passing dry run against that URL first) and keeping production tokens out of the shell Claude Code runs in. Point your own CATALOG_API_URL at dev or staging; production runs happen in your pipeline with a production credential from its secrets store.
cd plugins/catalog
python3 -m unittest discover -s tests -t tests # stdlib only; CI runs the sameReady to start using the Python client? The fastest way to get up and running:
β Installation & Quick Start Guide
This will walk you through installation, getting an API token, and your first few API calls.
- Python Client Usage Guide - Comprehensive guide covering datasets, collections, lineage, async usage, and error handling
- Interactive Examples - Jupyter notebooks with step-by-step walkthroughs
- API Token Setup - How to generate and use API tokens
- Dataset Catalog API - The backend service this client connects to
- Schema Documentation - Detailed data models and relationships
- Development Setup - Local development and testing
- Issues & Feedback - Report bugs or request features
This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to opensource@chanzuckerberg.com.