Skip to content

fix(ingest/mongodb): add missing bytes and DatetimeMS type mappings - #19389

Open
JohnRTurner wants to merge 1 commit into
datahub-project:masterfrom
JohnRTurner:fix/mongodb-bson-type-mappings
Open

fix(ingest/mongodb): add missing bytes and DatetimeMS type mappings#19389
JohnRTurner wants to merge 1 commit into
datahub-project:masterfrom
JohnRTurner:fix/mongodb-bson-type-mappings

Conversation

@JohnRTurner

@JohnRTurner JohnRTurner commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds bytes and bson.datetime_ms.DatetimeMS entries to PYMONGO_TYPE_TO_MONGO_TYPE and _field_type_mapping in the MongoDB source.
  • BSON Binary fields (subtype 0) decode to plain bytes, and out-of-range datetimes decode to bson.datetime_ms.DatetimeMS (relevant since this connector sets datetime_conversion="DATETIME_AUTO"). Neither type was previously mapped, so affected fields fell back to "unknown"/NullTypeClass with an "Unrecognized column type" warning instead of being typed as binary/date.

Test plan

  • Verified via ./gradlew :metadata-ingestion:lintFix
  • Manually confirmed via BSON encode/decode that Binary subtype 0 decodes to bytes and an out-of-range datetime under DATETIME_AUTO decodes to bson.datetime_ms.DatetimeMS, and that both now resolve correctly through the updated dicts
  • Existing MongoDB unit tests pass (pytest tests/unit/test_mongodb_source.py, 14 passed)

🤖 Generated with Claude Code


Summary by cubic

Map BSON Binary and out-of-range datetimes to the correct types in the MongoDB source. Previously, bytes and bson.datetime_ms.DatetimeMS decoded values were unrecognized and emitted as unknown/NullType with warnings; now they map to binary/date, removing those warnings and aligning with datetime_conversion="DATETIME_AUTO".

  • Add bytes -> "binary"/BytesTypeClass and bson.datetime_ms.DatetimeMS -> "date"/TimeTypeClass entries in PYMONGO_TYPE_TO_MONGO_TYPE and _field_type_mapping.
  • Behavior change: fields decoded as bytes or bson.datetime_ms.DatetimeMS now emit binary/date schema types; expect fewer warnings and possible schema updates in emitted metadata. No configuration changes required.

Written for commit d7edbf0. Summary will update on new commits.

Review in cubic

BSON Binary fields decode to plain bytes and out-of-range datetimes
decode to bson.datetime_ms.DatetimeMS (used with datetime_conversion=
DATETIME_AUTO), but neither type was present in PYMONGO_TYPE_TO_MONGO_TYPE
or _field_type_mapping. Missing entries fell back to "unknown"/NullTypeClass
with a warning instead of the correct binary/date typing.
@github-actions github-actions Bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Aug 22, 2026
@cursor

cursor Bot commented Aug 22, 2026

Copy link
Copy Markdown

PR Summary

Overview
MongoDB schema inference now types BSON Binary (decoded as bytes) and out-of-range datetimes (bson.datetime_ms.DatetimeMS under DATETIME_AUTO) as binary and date instead of falling back to unknown/NullTypeClass with unrecognized-type warnings.

Adds those entries to PYMONGO_TYPE_TO_MONGO_TYPE and _field_type_mapping in the MongoDB source.

Reviewed by Cursor Bugbot for commit d7edbf0. Bugbot is set up for automated code reviews on this repo. Configure here.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

@maggiehays maggiehays added the needs-review Label for PRs that need review from a maintainer. label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants