Skip to content

Commit 537e5ff

Browse files
Arp-Gclaude
Arp-G
and
claude
committed
Add CI workflow, ARCHITECTURE.md, CLAUDE.md, and v5 README
CI: GitHub Actions with Postgres 16 + MySQL 8.4 services, cached deps and Dialyzer PLT, gates: format / credo --strict / compile warnings-as-errors / dialyzer / test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent d16564f commit 537e5ff

5 files changed

Lines changed: 325 additions & 12 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [v5, develop, master]
6+
pull_request:
7+
8+
permissions:
9+
contents: read
10+
11+
env:
12+
CSV2SQL_TEST_POSTGRES_URL: postgres://postgres:postgres@localhost:5432/postgres
13+
CSV2SQL_TEST_MYSQL_URL: mysql://root:mysql@localhost:3306/mysql
14+
15+
jobs:
16+
ci:
17+
name: Lint, Dialyzer, Test
18+
runs-on: ubuntu-latest
19+
20+
services:
21+
postgres:
22+
image: postgres:16
23+
env:
24+
POSTGRES_PASSWORD: postgres
25+
ports:
26+
- 5432:5432
27+
options: >-
28+
--health-cmd pg_isready
29+
--health-interval 5s
30+
--health-timeout 5s
31+
--health-retries 10
32+
mysql:
33+
image: mysql:8.4
34+
env:
35+
MYSQL_ROOT_PASSWORD: mysql
36+
ports:
37+
- 3306:3306
38+
options: >-
39+
--health-cmd "mysqladmin ping -pmysql"
40+
--health-interval 5s
41+
--health-timeout 5s
42+
--health-retries 10
43+
44+
steps:
45+
- uses: actions/checkout@v4
46+
47+
- uses: erlef/setup-beam@v1
48+
with:
49+
otp-version: "28"
50+
elixir-version: "1.20"
51+
52+
- name: Cache deps and build
53+
uses: actions/cache@v4
54+
with:
55+
path: |
56+
deps
57+
_build
58+
key: mix-${{ runner.os }}-${{ hashFiles('mix.lock') }}
59+
restore-keys: mix-${{ runner.os }}-
60+
61+
- name: Cache Dialyzer PLT
62+
uses: actions/cache@v4
63+
with:
64+
path: priv/plts
65+
key: plt-${{ runner.os }}-${{ hashFiles('mix.lock') }}
66+
restore-keys: plt-${{ runner.os }}-
67+
68+
- run: mix deps.get
69+
- run: mix format --check-formatted
70+
- run: mix credo --strict
71+
- run: mix compile --warnings-as-errors
72+
- run: mix dialyzer
73+
- run: mix test

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,6 @@ csv2sql-*.tar
3535
npm-debug.log
3636
/assets/node_modules/
3737

38+
39+
# Dialyzer PLTs
40+
/priv/plts/

‎ARCHITECTURE.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# csv2sql v5 — Architecture
2+
3+
Csv2sql bulk-loads CSV files into a database with automatic schema
4+
inference. This document describes the v5 engine; it is kept current as
5+
stages of the [rewrite roadmap](#roadmap) land.
6+
7+
## Design principles
8+
9+
1. **No global mutable state.** A run's configuration is one validated,
10+
immutable `%Csv2sql.Config{}` passed down the pipeline. Nothing reads
11+
application env at runtime; nothing is globally named.
12+
2. **Structural completion.** A run is done when its pipeline fold
13+
returns — there are no completion counters, drain protocols, or
14+
busy-wait loops (the failure modes that killed the v3 and v4.0
15+
architectures).
16+
3. **One event bus.** Every stage broadcasts typed event structs on a
17+
per-run `Phoenix.PubSub` topic. The LiveView UI, the CLI reporter, the
18+
report writer, and the test harness are all plain subscribers — there
19+
is no other progress/error channel.
20+
4. **Errors are data.** Cell-level checks are total functions; malformed
21+
input becomes structured `%Csv2sql.Error{}` values with a class
22+
(`:preflight | :run | :file | :row | :transient`) that defines the
23+
blast radius. A bad file fails alone; the run continues.
24+
5. **Adapters own dialects.** The engine speaks abstract column types
25+
(`{:varchar, n} | :text | :longtext | :integer | :bigint | :float |
26+
:boolean | :date | :datetime`); `Csv2sql.Adapter` callbacks (with
27+
overridable defaults) render SQL. No `ecto_sql`/repos — drivers
28+
(Postgrex/MyXQL) are used directly; core `ecto` provides embedded
29+
schemas + changesets for boundary data and config validation only.
30+
31+
## Process architecture
32+
33+
```
34+
Csv2sql.Application (one_for_one)
35+
├── Phoenix.PubSub (Csv2sql.PubSub — the event bus)
36+
├── Registry (Csv2sql.RunRegistry — per-run process names)
37+
├── DynamicSupervisor (Csv2sql.RunsSupervisor — one child per run)
38+
├── Task.Supervisor (Csv2sql.ProbeSupervisor — UI connection probes)
39+
└── Csv2sqlWeb.Endpoint (GUI mode)
40+
41+
Per run (temporary, max_restarts: 0 — nothing inside a run restarts):
42+
Run.Supervisor
43+
├── Run.Server state + events + cancel; monitors the pipeline
44+
└── core (Supervisor)
45+
├── DB pool Postgrex, owned by the run, dies with it
46+
├── Task.Supervisor file workers
47+
└── Run.Pipeline (Task) the orchestration fold
48+
```
49+
50+
- **Isolation:** every process registers in `Csv2sql.RunRegistry` keyed by
51+
`run_id` (`Csv2sql.Run.server_name/1` etc.), so concurrent runs coexist
52+
and a crashed run cannot poison the next.
53+
- **Failure handling:** worker crashes become file-level errors
54+
(`async_stream_nolink`); a pipeline crash is converted to `RunFailed` by
55+
`Run.Server` (which monitors it); a server crash tears down the run
56+
tree (`max_restarts: 0`).
57+
- **Cancellation:** `Run.Server.cancel/1` stops the core supervisor
58+
(pipeline + workers die, pool closes), emits `RunCancelled`, and the
59+
tree is torn down after a grace period.
60+
61+
## The pipeline
62+
63+
```
64+
Run.Pipeline.run/2
65+
discover files (stable lexicographic order)
66+
Task.Supervisor.async_stream_nolink(files, &FilePipeline.run/2,
67+
max_concurrency: file_workers)
68+
Server.complete(results) ← returning IS completion
69+
70+
FilePipeline.run/2 (per file)
71+
pass 1 analyze: stream → chunks → Task.async_stream(infer) → merge
72+
(associative: AND type flags, max lengths, min/max int bounds —
73+
constant memory regardless of file size)
74+
→ header repair (trim, blank→column_N, dedupe) → resolve types
75+
DDL drop (table_mode :drop) + CREATE TABLE IF NOT EXISTS
76+
pass 2 load: stream → chunks → Task.async_stream(encode + insert,
77+
max_concurrency: insert_workers)
78+
Backpressure is inherent: busy insert workers pause reading.
79+
ledger rows_read == rows_inserted + rows_rejected, per chunk, using
80+
driver-reported insert counts (never assumed)
81+
```
82+
83+
`mode: :analyze` stops after pass 1 — the dry-run behind schema preview.
84+
85+
## Events
86+
87+
All in `Csv2sql.Events`, all structs with `run_id` + `at`:
88+
89+
`RunStarted` · `FileDiscovered` · `AnalyzeStarted` · `AnalyzeCompleted` ·
90+
`SchemaCreated` · `LoadProgress` · `FileCompleted{ledger, timings}` ·
91+
`FileFailed{error}` · terminal: `RunCompleted{summary}` · `RunFailed` ·
92+
`RunCancelled`.
93+
94+
Subscribe with `Events.subscribe(run_id)`; `Csv2sql.start_run(config,
95+
subscribe: true)` subscribes the caller *before* the run starts so no
96+
event can be missed.
97+
98+
## Validation ladder
99+
100+
1. **Ledger** (always on, free): in-run double-entry accounting, checked
101+
per chunk. Implemented.
102+
2. **Counts** (planned): post-load `COUNT(*)` vs ledger, append-aware.
103+
3. **Fingerprints** (planned, default-on toggle): per-column O(1)
104+
aggregates over post-transform values vs one aggregate query per table.
105+
4. **Deep checksum** (planned, opt-in).
106+
107+
## Testing
108+
109+
- `Csv2sql.EngineCase`: per-test tmp dirs, `csv!/3` fixture builder,
110+
`unique_name/1` (collision-free tables → DB tests run `async: true`),
111+
`run_import/2` (subscribed run + collected events + auto table
112+
cleanup), `assert_event/2`, `db_query!/2`.
113+
- Doctests document and pin the pure functions (TypeCheck, Inference,
114+
Resolve, Encoder, Ledger, Db URL parsing).
115+
- Integration tests drive full runs and verify against
116+
`information_schema` and real row data.
117+
- DB-backed tests are tagged (`:postgres`) and excludable:
118+
`mix test --exclude postgres`.
119+
- **Local gotcha:** this machine exports `MIX_ENV=dev`, so run tests as
120+
`MIX_ENV=test mix test`.
121+
122+
## Roadmap
123+
124+
Stages (from the approved v5 plan): **1. walking skeleton ✅** ·
125+
2. adapter behaviour + MySQL + byte/param-aware insert engine ·
126+
3. full type system (dates via TimeParse compiler, booleans, sizing) ·
127+
4. error model (row isolation → errors.csv), encodings, delimiters ·
128+
5. validation ladder + preflight + run_report.json · 6. CLI + config
129+
sources/persistence · 7. LiveView UI + PWA · 8. Burrito packaging +
130+
Sentry + docs.

‎CLAUDE.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# CLAUDE.md
2+
3+
Guidance for Claude Code when working in this repository.
4+
5+
## Branch context (important)
6+
7+
- **`v5` (this branch)** is a ground-up rewrite: a single Phoenix 1.8 app
8+
(no umbrella) built per the approved v5 plan. It shares **no code** with
9+
the other branches.
10+
- `master`/`develop` hold the old working v3 umbrella app;
11+
`version-4.0` is an abandoned GenStage/Flow rewrite. Both are
12+
reference-only — do not port their global-state patterns here.
13+
- Untracked v4-era files (old CLAUDE.md, `.env.*.test`, `test/support/`)
14+
were preserved in a git stash before this branch was created:
15+
`git stash list` → "v4-workflow untracked files…".
16+
17+
## Commands
18+
19+
```bash
20+
MIX_ENV=test mix test # REQUIRED: this machine exports MIX_ENV=dev
21+
# (plain `mix test` compiles into _build/dev
22+
# and fails to find test/support modules)
23+
MIX_ENV=test mix test --exclude postgres # without a local Postgres
24+
25+
mix format # formatter
26+
mix credo --strict # lint (must stay clean)
27+
mix dialyzer # types (must stay clean; PLT in priv/plts)
28+
mix compile --warnings-as-errors
29+
mix phx.server # web UI (stage 7 — stock scaffold for now)
30+
```
31+
32+
DB-backed tests default to `postgres://postgres:postgres@localhost:5432/postgres`;
33+
override with `CSV2SQL_TEST_POSTGRES_URL`. Tables are per-test unique and
34+
dropped automatically — safe to run against a local server.
35+
36+
## Architecture
37+
38+
Read `ARCHITECTURE.md` first. The short version:
39+
40+
- `lib/csv2sql/` is the engine (zero web deps); `lib/csv2sql_web/` is the
41+
LiveView UI. Public API: `Csv2sql.start_run/2`, `await/2`,
42+
`cancel_run/1`, `run_state/1`.
43+
- Per-run supervision trees under `Csv2sql.RunsSupervisor`, names scoped
44+
in `Csv2sql.RunRegistry` — **never add a globally named process** to
45+
the run path, and never read `Application.get_env` in engine code;
46+
everything flows from `%Csv2sql.Config{}`.
47+
- Progress/errors travel ONLY as typed `Csv2sql.Events` structs on
48+
PubSub. New stages emit events; UIs/tests subscribe.
49+
- All boundary data is Ecto **embedded schemas** (core `ecto` only — no
50+
ecto_sql, no repos). DB access goes through `Csv2sql.Adapter`
51+
callbacks; drivers are Postgrex/MyXQL directly.
52+
- Cell-level functions are **total** — return tagged tuples, never raise
53+
on malformed data (upstream issue #24 class).
54+
55+
## Conventions
56+
57+
- `@moduledoc` on every module, `@spec` on public functions, doctests on
58+
pure functions. `mix credo --strict` and `mix dialyzer` must stay clean.
59+
- `alias Csv2sql.File, as: CsvFile` (it clashes with Elixir's `File`).
60+
- Tests: `use Csv2sql.EngineCase` for engine runs (tmp dirs, `csv!/3`,
61+
`run_import/2`, `assert_event/2`, `db_query!/2`, auto cleanup);
62+
`unique_name/1` for anything hitting the DB so `async: true` stays safe.
63+
- Commits are granular; PRs target `develop` per repo convention (the v5
64+
branch will merge when the rewrite reaches parity).
65+
66+
## Roadmap position
67+
68+
Stage 1 (walking skeleton) is complete: Postgres-only happy path,
69+
minimal types (int/bigint/float/varchar/text), ledger validation, full
70+
event bus, 44 tests. Next stages: 2 adapters+MySQL, 3 full type system,
71+
4 error model/encodings, 5 validation+report, 6 CLI+config, 7 UI+PWA,
72+
8 Burrito+Sentry. Details in ARCHITECTURE.md and the plan file.

‎README.md‎

Lines changed: 47 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,53 @@
1-
# Csv2sql
1+
# csv2sql
22

3-
To start your Phoenix server:
3+
> ⚠️ **v5 rewrite in progress.** This branch is a ground-up rewrite of
4+
> csv2sql. The released v3 (CLI escript + dashboard umbrella) lives on
5+
> [`develop`](../../tree/develop).
46
5-
* Run `mix setup` to install and setup dependencies
6-
* Start Phoenix endpoint with `mix phx.server` or inside IEx with `iex -S mix phx.server`
7+
A blazing fast, fully-automated CSV → database importer: point it at a
8+
directory of CSV files and it infers table schemas from the data, creates
9+
the tables, loads the rows in parallel, and validates what arrived.
710

8-
Now you can visit [`localhost:4000`](http://localhost:4000) from your browser.
11+
## v5 goals
912

10-
Ready to run in production? Please [check our deployment guides](https://phoenix.hexdocs.pm/deployment.html).
13+
- One app, two faces: a LiveView web UI **and** a headless CLI, shipped as
14+
a single cross-platform [Burrito](https://github.com/burrito-elixir/burrito)
15+
binary (macOS / Linux / Windows).
16+
- Robust plain-OTP engine: per-run supervision, concurrent runs, inherent
17+
backpressure, structural completion — no global state.
18+
- Structured errors that never blow up a run: bad files fail alone, bad
19+
rows are isolated to `<file>.errors.csv`, everything lands in a report.
20+
- Layered validation: in-run row ledger, count checks, and per-column
21+
fingerprint comparison.
22+
- Extensible DB adapters: PostgreSQL and MySQL first, engineered for more.
1123

12-
## Learn more
24+
## Status
1325

14-
* Official website: https://www.phoenixframework.org/
15-
* Guides: https://phoenix.hexdocs.pm/overview.html
16-
* Docs: https://phoenix.hexdocs.pm
17-
* Forum: https://elixirforum.com/c/phoenix-forum
18-
* Source: https://github.com/phoenixframework/phoenix
26+
| Stage | | |
27+
|---|---|---|
28+
| 1 | Walking skeleton — engine, events, Postgres, ledger, test harness | ✅ |
29+
| 2 | Adapter behaviour + MySQL + byte/param-aware inserts | ⬜ |
30+
| 3 | Full type system (dates, booleans, dynamic sizing) | ⬜ |
31+
| 4 | Error model, encodings, delimiters | ⬜ |
32+
| 5 | Validation ladder + preflight + report | ⬜ |
33+
| 6 | CLI + config system | ⬜ |
34+
| 7 | LiveView UI + PWA | ⬜ |
35+
| 8 | Burrito packaging + polish | ⬜ |
36+
37+
## Development
38+
39+
Requires Erlang/Elixir per [`.tool-versions`](.tool-versions) and a local
40+
PostgreSQL for the integration tests.
41+
42+
```bash
43+
mix setup # deps + assets
44+
MIX_ENV=test mix test # full suite (or --exclude postgres)
45+
mix credo --strict # lint
46+
mix dialyzer # types
47+
```
48+
49+
Architecture notes: [ARCHITECTURE.md](ARCHITECTURE.md).
50+
51+
## License
52+
53+
[MIT](https://github.com/Arp-G/csv2sql/blob/develop/LICENSE.md)

0 commit comments

Comments
 (0)