Rename AttributeType/adapter terminology to Codec

[dimitri-yatsenko]

Summary

This PR implements a comprehensive redesign of the custom type system, renaming "AttributeType/adapter" terminology to "Codec" and providing a cleaner, more intuitive API. It also simplifies the testing infrastructure with testcontainers.

Codec API Redesign

Key Changes

• Renamed AttributeType base class to Codec - The new name better reflects the purpose: encoding Python objects for database storage and decoding them on retrieval
• Auto-registration via __init_subclass__ - Codecs automatically register when their class is defined; no decorator needed
• New get_dtype(is_external) method - Codecs dynamically return their underlying storage type based on whether external storage is used
• Unified naming convention - All built-in codecs use <name> or <name@store> syntax consistently
• Renamed type category ADAPTED to CODEC in internal code

Built-in Codecs

Codec	Internal	External	Description
<blob>	bytes	<hash@>	DataJoint serialization for Python objects
<hash@>	N/A	json	Content-addressed storage with MD5 dedup
<object@>	N/A	json	Path-addressed storage for files/folders
<attach>	bytes	<hash@>	File attachments
<filepath@store>	N/A	json	Reference to existing external files

New Codec API

import datajoint as dj

class GraphCodec(dj.Codec):
    name = "graph"  # Auto-registers on class definition
    
    def get_dtype(self, is_external: bool) -> str:
        return "<blob>"  # Delegates serialization to blob codec
    
    def encode(self, graph, *, key=None, store_name=None):
        return {'nodes': list(graph.nodes()), 'edges': list(graph.edges())}
    
    def decode(self, stored, *, key=None):
        G = nx.Graph()
        G.add_nodes_from(stored['nodes'])
        G.add_edges_from(stored['edges'])
        return G

Testing Infrastructure: Testcontainers

Tests now use testcontainers to automatically manage MySQL and MinIO containers. No manual docker-compose up required.

New Developer Workflow

# Clone and install
git clone https://github.com/datajoint/datajoint-python.git
cd datajoint-python
pip install -e ".[test]"

# Run all tests - containers start automatically
pytest tests/

Benefits

• Zero setup - Just pip install and pytest
• Dynamic ports - No conflicts with other services
• Automatic cleanup - Containers stop when tests finish
• Simpler CI - No docker-compose orchestration needed

Fallback: External Containers

For development/debugging with persistent containers:

docker compose up -d db minio
DJ_USE_EXTERNAL_CONTAINERS=1 pytest tests/

Dead Code & Terminology Cleanup

Removed

• Backward compatibility aliases (AttributeType, register_type, list_types, get_type, etc.)
• Backward compatibility codec aliases (ObjectType, AttachType, XAttachType, FilepathType)
• Unused build_foreign_key_parser_old() function
• Legacy feature switches (ADAPTED_TYPE_SWITCH, FILEPATH_FEATURE_SWITCH)
• Unused enable_filepath_feature test fixture
• Misleading comments about non-existent DJBlobType/ContentType
• object-type-spec.md (implementation complete, info now in object.md)
• pytest-env dependency (testcontainers handles configuration)

Renamed

• Type category ADAPTED → CODEC in declare.py and heading.py
• Test files:
  • schema_adapted.py → schema_codecs.py
  • test_adapted_attributes.py → test_codecs.py
  • test_type_composition.py → test_codec_chaining.py
• Test classes and functions to use Codec terminology

Updated Terminology

• All comments/docstrings: "AttributeType" → "Codec", "Adapter" → "Codec"
• Fixed content_registry.py docstring: SHA256 → MD5

Documentation Updates

New Documentation

• codec-spec.md - Detailed API specification for creating custom codecs
• codecs.md - User guide with examples (replaces customtype.md)
• README.md - Comprehensive developer guide with test/pre-commit instructions

Updated Documentation

• mkdocs.yaml - Navigation updated: customtype.md → codecs.md
• attributes.md - Fixed dead links, updated terminology
• docker-compose.yaml - Clarified it's optional for tests

Removed Documentation

• object-type-spec.md (redundant with object.md)
• customtype.md (replaced by codecs.md)

Other Changes in This Branch

Settings System

• Simplified settings to pure Pydantic without backward compatibility shims
• Added recursive config file search and secrets separation
• Removed deprecated save_* methods and set_password function

Type System

• Added type aliases (int8, int16, int32, int64, uint8, etc.)
• Added decimal(n,f) to core types
• Renamed core type 'blob' to 'bytes' for cross-database portability
• Added text type and documented type modifier policy

External Storage

• Refactored external storage to use fsspec for unified backend support
• Implemented content registry with MD5 hashing (removed SHA256)
• Added garbage collection module for external storage cleanup
• Implemented <object@> type for managed file/folder storage

Infrastructure

• Dropped support for Python < 3.10 and MySQL < 8.0
• Reorganized tests into unit/ and integration/ directories
• Version bump to 2.0.0a7

Test Plan

• All 471 tests pass
• Verified codec auto-registration works correctly
• Verified external storage chain (<blob@> → <hash@> → storage) works
• Verified testcontainers starts/stops containers automatically
• Verified external container mode works (DJ_USE_EXTERNAL_CONTAINERS=1)
• Verified renamed test files execute correctly

🤖 Generated with Claude Code