feat: Publish a `singer-python` and `pipelinewise-singer-python` replacement by edgarrmondragon · Pull Request #3701 · meltano/sdk

edgarrmondragon · 2026-07-15T22:18:32Z

No description provided.

sourcery-ai

Sorry @edgarrmondragon, your pull request is larger than the review limit of 150000 diff characters

read-the-docs-community · 2026-07-15T22:19:27Z

Documentation build overview

📚 Meltano SDK | 🛠️ Build #34277196 | 📁 Comparing 7e9e70c against latest (5d3f63b)

🔍 Preview build

14 files changed · + 1 added · ± 13 modified

+ Added

guides/singer-python.html

± Modified

codecov · 2026-07-15T22:20:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.52%. Comparing base (747fc9a) to head (7e9e70c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3701      +/-   ##
==========================================
+ Coverage   94.35%   94.52%   +0.16%     
==========================================
  Files          74       87      +13     
  Lines        6294     6462     +168     
  Branches      770      783      +13     
==========================================
+ Hits         5939     6108     +169     
+ Misses        266      265       -1     
  Partials       89       89

Flag	Coverage Δ
core	`83.39% <99.44%> (+0.45%)`	⬆️
end-to-end	`74.99% <75.19%> (-0.62%)`	⬇️
optional-components	`45.28% <56.95%> (+0.25%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed-hq · 2026-07-15T22:21:24Z

Merging this PR will not alter performance

✅ 14 untouched benchmarks

_{Comparing singer-python-package (7e9e70c) with main (747fc9a)}

The code in singer_sdk/singerlib now lives in packages/meltano-singer-python as the top-level `singer` package, published as the meltano-singer-python distribution. singer-sdk now depends on it and singer_sdk.singerlib re-exports the same objects as a stable alias.

…er package singer.metrics now hosts the meters, points, and exclusion filter, plus legacy singer-python factories (record_counter(endpoint=...), http_request_timer, job_timer). singer_sdk.metrics re-exports the core and keeps the SDK-flavored stream-based factories. BREAKING CHANGE: the metrics logger is now named singer.metrics instead of singer_sdk.metrics, and metric points carry a pid tag. singer.logging hosts ConsoleFormatter/StructuredFormatter, and singer.get_logger configures logging from SINGER_SDK_LOG_CONFIG (dictConfig, as generated by Meltano) or LOGGING_CONF_FILE (INI, pipelinewise-compatible).

Adds singer.bookmarks, singer.metadata, and extends singer.utils (now, load_json, check_config, parse_args) and singer.catalog (Catalog.load/dump) to port the legacy singer-python API. Message classes gain a legacy asdict() alongside to_dict(). singer.__init__ gains write_record/write_schema/write_state/write_version and should_sync_field, matching the legacy top-level API. The deprecated --properties CLI flag is intentionally not supported; use --catalog instead.

singer.batch now hosts BaseBatchFileEncoding and BatchMessage. singer_sdk.helpers._batch keeps SDKBatchMessage as an alias for BatchMessage, plus the fsspec-coupled StorageTarget/BatchConfig and the deprecated JSONLinesEncoding/ParquetEncoding wrappers. BatchFileFormat (jsonl/parquet) is left untouched in the SDK, since encoding.format is already a plain string and taps can pass format='arrow' without any enum change.

Extends the release workflow to build, attest, and publish meltano-singer-python alongside singer-sdk on the same tag. Adds a migration guide for singer-python/pipelinewise-singer-python users, and documents the singer.metrics logger rename.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

…log.from_entries write() had regressed to a no-op on breadcrumbs not already present in the compiled map, silently breaking get_standard_metadata() (which always starts from an empty mapping) and any first write to a new stream/field. Restored legacy semantics: create the breadcrumb entry if missing. Catalog.from_entries() called cls({"streams": ...}), which invokes dict.__init__ directly and does not go through from_dict — it built a single-key mapping instead of one keyed by tap_stream_id. Fixed to build the catalog the same way from_dict/add_stream do.

… replication-method/view-key-properties CatalogEntry.metadata is a MetadataMapping (already compiled, keyed by breadcrumb, with typed Metadata/StreamMetadata values), not the raw legacy list-of-entries form. Legacy taps calling metadata.to_map(catalog_entry.metadata) — as pipelinewise-singer-python callers do — got a TypeError, since to_map() assumed a raw list. to_map() now also accepts any breadcrumb-keyed mapping and compiles it to the same raw dict-of-dicts shape, converting typed values via to_dict() where available. StreamMetadata had no field for 'replication-method' or 'view-key-properties' (only its own 'forced-replication-method'), so both keys were silently dropped whenever stream metadata passed through Catalog.load()/from_dict() or MetadataMapping.to_list() -- a real data-loss bug for any orchestrator-set replication method. Added replication_method and view_key_properties fields.

Legacy singer-python code (and tests written against it) constructs CatalogEntry(metadata=[{'breadcrumb': [...], 'metadata': {...}}, ...]) directly, passing the raw list-of-entries form rather than a compiled MetadataMapping. CatalogEntry.metadata is typed as MetadataMapping, so this previously left .metadata as a plain list, breaking any code that expects mapping access (e.g. .root, breadcrumb indexing). __post_init__ now coerces a raw list the same way BatchMessage/BatchConfig already coerce their dict-shaped fields.

…ard_metadata() Verified against singer-io/singer-python's metadata.py directly. Two deviations found and fixed: - new() returned a defaultdict(dict), which silently auto-vivifies missing breadcrumbs on any access (e.g. 'x in mdata' or mdata[k]); upstream returns a plain {} with normal KeyError semantics. - get_standard_metadata() marked properties in valid_replication_keys as 'automatic' inclusion in addition to key_properties; upstream only does this for key_properties. CompiledMetadata stays as the concrete dict[Breadcrumb, dict[str, Any]] type: write()/delete()/get()/to_list()/new() all need dict mutation semantics that a read-only Mapping can't provide, matching upstream's own plain-dict design for this module.

tap-mysql, the first real consumer of meltano-singer-python, migrated every metadata.to_map()/metadata.get() call site to direct MetadataMapping/Metadata attribute access (catalog_entry.metadata.root.replication_method, catalog_entry.metadata['properties', name].inclusion), since CatalogEntry.metadata is always a MetadataMapping already. Keeping the functional dict-based module around as unused, untested surface added maintenance cost without a real consumer. BREAKING CHANGE: singer.metadata (new/to_map/to_list/write/get/delete/ get_standard_metadata) no longer exists. Use CatalogEntry.metadata (a MetadataMapping) directly instead.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

This reverts commit 4b0e120.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

sourcery-ai Bot reviewed Jul 15, 2026

View reviewed changes

edgarrmondragon force-pushed the singer-python-package branch 3 times, most recently from f696016 to 4d18542 Compare July 15, 2026 23:43

edgarrmondragon added 17 commits July 15, 2026 17:53

fix: Convert bookmark string to list

287dbe7

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Avoid metrics leaking into snapshots

f43c012

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Fix test-lowest test session

999ca4b

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Remove metadata list coercing

38ddf6a

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Fix package metadata

4c6242f

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Use singer_sdk.metrics logger

a9a78f5

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

refactor: Make stream optional in metrics.record_counter

20bd215

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

edgarrmondragon force-pushed the singer-python-package branch from 4d18542 to 20bd215 Compare July 15, 2026 23:54

edgarrmondragon added 5 commits July 15, 2026 22:15

Make metadata and schema fields of catalog entry optional

e153117

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Use typing.Self consistently

7231bcf

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

73def67

Merge branch 'main' into singer-python-package

7cfeaa4

Merge branch 'main' into singer-python-package

0306811

edgarrmondragon added 10 commits July 16, 2026 15:11

Merge branch 'main' into singer-python-package

1124706

refactor: Support arbitrary tags in new preset timers

dc155d8

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

a164ae0

Fix meter tests

6b47ae6

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Remove Message.asdict

3d1b412

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

693a290

Remove Catalog.dump

2a26e0d

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Make pyyaml a required dependency

f638dd1

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

The logging.config values we replace are always in dicts

a1b89cf

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Remove should_sync_field

89a157a

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

edgarrmondragon force-pushed the singer-python-package branch from 141474f to 89a157a Compare July 17, 2026 01:41

edgarrmondragon added 4 commits July 16, 2026 19:51

refactor: Simplify write_schema inputs

62fbb24

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

bfd7f1f

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

6236452

Bump baipp

2b53ae2

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

edgarrmondragon force-pushed the singer-python-package branch from bc3581b to 2b53ae2 Compare August 28, 2026 03:45

edgarrmondragon added 8 commits August 27, 2026 21:58

Restore __all__

9893d86

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Update imports

4b0e120

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Revert "Update imports"

26edd21

This reverts commit 4b0e120.

Update griffe search paths

ab03f26

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Resolve "external" singer package

cfb6806

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>

Merge branch 'main' into singer-python-package

aac82b5

Merge branch 'main' into singer-python-package

69ae26f

Merge branch 'main' into singer-python-package

7e9e70c

Navigation Menu

Uh oh!

feat: Publish a `singer-python` and `pipelinewise-singer-python` replacement - #3701

feat: Publish a `singer-python` and `pipelinewise-singer-python` replacement#3701
edgarrmondragon wants to merge 44 commits into
mainfrom
singer-python-package

edgarrmondragon commented Jul 15, 2026

Uh oh!

sourcery-ai Bot left a comment

Uh oh!

read-the-docs-community Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

edgarrmondragon commented Jul 15, 2026

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

read-the-docs-community Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

codecov Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

read-the-docs-community Bot commented Jul 15, 2026 •

edited

Loading

codecov Bot commented Jul 15, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 15, 2026 •

edited

Loading