Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
bab8d1e
Add decimal(n,f) to core types (Tier 2)
claude Jan 1, 2026
21dd700
Rename core type 'blob' to 'bytes' for cross-database portability
claude Jan 1, 2026
2f38734
Fix dtype table: longblob is native type for raw bytes
claude Jan 1, 2026
4399f51
Update enum PostgreSQL mapping to use native ENUM type
claude Jan 1, 2026
7e32089
Document UTC timezone policy for datetime type
claude Jan 1, 2026
5d2c9f6
Add text to core types and document type modifier policy
claude Jan 1, 2026
e55c9a7
Add encoding and collation policy documentation
claude Jan 1, 2026
4fa802a
Document AttributeType naming conventions comprehensively
claude Jan 1, 2026
11d67a6
Redesign AttributeType naming with @ storage mode convention
claude Jan 1, 2026
d6bdd80
Fix content@ notation in json dtype description
claude Jan 1, 2026
1ad327d
Rename content type to hash for clarity
claude Jan 1, 2026
92647f1
Fix DataJoint syntax in HashRegistry definition
claude Jan 1, 2026
7d0a5a5
Clarify nullability and default value handling
claude Jan 1, 2026
af7c76c
Remove HashRegistry table, use JSON field scanning for GC
claude Jan 1, 2026
691edcc
Use MD5 instead of SHA256 for content hashing
claude Jan 1, 2026
905c463
Remove dead code: uuid_from_file and uuid_from_stream
claude Jan 1, 2026
57af0e8
Remove outdated storage-types-implementation-plan.md
claude Jan 1, 2026
40d7871
Fix type pattern conflicts and BLOB reference
dimitri-yatsenko Jan 1, 2026
5cb5ae4
Rename AttributeTypes to Codec Types in documentation
claude Jan 1, 2026
2dde3d9
Add Codec base class with __init_subclass__ auto-registration
claude Jan 1, 2026
8218fa7
Update Codec API with get_dtype(is_external) method
claude Jan 1, 2026
e7054dd
Unify codec names and fix external storage chain
dimitri-yatsenko Jan 1, 2026
898d0ed
Rename AttributeType/adapter terminology to Codec
dimitri-yatsenko Jan 1, 2026
f3e4489
Remove backward compatibility aliases from Codec API
dimitri-yatsenko Jan 1, 2026
4417ec1
Add Codec documentation with plugin specification
dimitri-yatsenko Jan 1, 2026
568d71a
Remove object-type-spec.md (implementation complete)
dimitri-yatsenko Jan 1, 2026
8271c86
Clean up dead code and outdated terminology
dimitri-yatsenko Jan 2, 2026
39cd993
Remove enable_filepath_feature fixture
dimitri-yatsenko Jan 2, 2026
11e492b
Rename tests to use Codec terminology
dimitri-yatsenko Jan 2, 2026
d4560e6
Implement testcontainers for pytest-managed container lifecycle
dimitri-yatsenko Jan 2, 2026
70f53d4
Fix test compatibility with testcontainers
dimitri-yatsenko Jan 2, 2026
fa47f47
Add pytest marks for test dependency management
dimitri-yatsenko Jan 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 66 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,30 +146,57 @@ DataJoint (<https://datajoint.com>).

### Prerequisites

- [Docker](https://docs.docker.com/get-docker/) for MySQL and MinIO services
- [Docker](https://docs.docker.com/get-docker/) (Docker daemon must be running)
- Python 3.10+

### Running Tests

Tests are organized into `unit/` (no external services) and `integration/` (requires MySQL + MinIO):
### Quick Start

```bash
# Install dependencies
# Clone and install
git clone https://github.com/datajoint/datajoint-python.git
cd datajoint-python
pip install -e ".[test]"

# Run unit tests only (fast, no Docker needed)
pytest tests/unit/
# Run all tests (containers start automatically via testcontainers)
pytest tests/

# Start MySQL and MinIO for integration tests
docker compose up -d db minio
# Install and run pre-commit hooks
pip install pre-commit
pre-commit install
pre-commit run --all-files
```

# Run all tests
### Running Tests

Tests use [testcontainers](https://testcontainers.com/) to automatically manage MySQL and MinIO containers.
**No manual `docker-compose up` required** - containers start when tests run and stop afterward.

```bash
# Run all tests (recommended)
pytest tests/

# Run with coverage report
pytest --cov-report term-missing --cov=datajoint tests/

# Run specific test file
pytest tests/integration/test_blob.py -v

# Stop services when done
# Run only unit tests (no containers needed)
pytest tests/unit/
```

### Alternative: External Containers

For development/debugging, you may prefer persistent containers that survive test runs:

```bash
# Start containers manually
docker compose up -d db minio

# Run tests using external containers
DJ_USE_EXTERNAL_CONTAINERS=1 pytest tests/

# Stop containers when done
docker compose down
```

Expand All @@ -183,24 +210,46 @@ docker compose --profile test up djtest --build

### Alternative: Using pixi

[pixi](https://pixi.sh) users can run tests with automatic service management:
[pixi](https://pixi.sh) users can run tests with:

```bash
pixi install # First time setup
pixi run test # Starts services and runs tests
pixi run services-down # Stop services
pixi run test # Runs tests (testcontainers manages containers)
```

### Pre-commit Hooks

Pre-commit hooks run automatically on `git commit` to check code quality.
**All hooks must pass before committing.**

```bash
pre-commit install # Install hooks (first time)
pre-commit run --all-files # Run all checks
# Install hooks (first time only)
pip install pre-commit
pre-commit install

# Run all checks manually
pre-commit run --all-files

# Run specific hook
pre-commit run ruff --all-files
pre-commit run codespell --all-files
```

Hooks include:
- **ruff**: Python linting and formatting
- **codespell**: Spell checking
- **YAML/JSON/TOML validation**
- **Large file detection**

### Before Submitting a PR

1. **Run all tests**: `pytest tests/`
2. **Run pre-commit**: `pre-commit run --all-files`
3. **Check coverage**: `pytest --cov-report term-missing --cov=datajoint tests/`

### Environment Variables

Tests use these defaults (configured in `pyproject.toml`):
For external container mode (`DJ_USE_EXTERNAL_CONTAINERS=1`):

| Variable | Default | Description |
|----------|---------|-------------|
Expand Down
13 changes: 9 additions & 4 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
# Development environment with MySQL and MinIO services
#
# Quick start:
# docker compose up -d db minio # Start services
# pytest tests/ # Run tests (uses localhost defaults)
# NOTE: docker-compose is OPTIONAL for running tests.
# Tests use testcontainers to automatically manage containers.
# Just run: pytest tests/
#
# Full Docker testing:
# Use docker-compose for development/debugging when you want
# persistent containers that survive test runs:
# docker compose up -d db minio # Start services manually
# pytest tests/ # Tests will use these containers
#
# Full Docker testing (CI):
# docker compose --profile test up djtest --build
services:
db:
Expand Down
2 changes: 1 addition & 1 deletion docs/mkdocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ nav:
- Blobs: design/tables/blobs.md
- Attachments: design/tables/attach.md
- Filepaths: design/tables/filepath.md
- Custom Datatypes: design/tables/customtype.md
- Custom Codecs: design/tables/codecs.md
- Dependencies: design/tables/dependencies.md
- Indexes: design/tables/indexes.md
- Master-Part Relationships: design/tables/master-part.md
Expand Down
Loading
Loading