fix(ingest/mongodb): add missing bytes and DatetimeMS type mappings - #19389
Open
JohnRTurner wants to merge 1 commit into
Open
fix(ingest/mongodb): add missing bytes and DatetimeMS type mappings#19389JohnRTurner wants to merge 1 commit into
JohnRTurner wants to merge 1 commit into
Conversation
BSON Binary fields decode to plain bytes and out-of-range datetimes decode to bson.datetime_ms.DatetimeMS (used with datetime_conversion= DATETIME_AUTO), but neither type was present in PYMONGO_TYPE_TO_MONGO_TYPE or _field_type_mapping. Missing entries fell back to "unknown"/NullTypeClass with a warning instead of the correct binary/date typing.
PR SummaryOverview Adds those entries to Reviewed by Cursor Bugbot for commit d7edbf0. Bugbot is set up for automated code reviews on this repo. Configure here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bytesandbson.datetime_ms.DatetimeMSentries toPYMONGO_TYPE_TO_MONGO_TYPEand_field_type_mappingin the MongoDB source.Binaryfields (subtype 0) decode to plainbytes, and out-of-range datetimes decode tobson.datetime_ms.DatetimeMS(relevant since this connector setsdatetime_conversion="DATETIME_AUTO"). Neither type was previously mapped, so affected fields fell back to"unknown"/NullTypeClasswith an "Unrecognized column type" warning instead of being typed as binary/date.Test plan
./gradlew :metadata-ingestion:lintFixBinarysubtype 0 decodes tobytesand an out-of-range datetime underDATETIME_AUTOdecodes tobson.datetime_ms.DatetimeMS, and that both now resolve correctly through the updated dictspytest tests/unit/test_mongodb_source.py, 14 passed)🤖 Generated with Claude Code
Summary by cubic
Map BSON Binary and out-of-range datetimes to the correct types in the MongoDB source. Previously,
bytesandbson.datetime_ms.DatetimeMSdecoded values were unrecognized and emitted as unknown/NullType with warnings; now they map to binary/date, removing those warnings and aligning withdatetime_conversion="DATETIME_AUTO".bytes-> "binary"/BytesTypeClass andbson.datetime_ms.DatetimeMS-> "date"/TimeTypeClass entries inPYMONGO_TYPE_TO_MONGO_TYPEand_field_type_mapping.bytesorbson.datetime_ms.DatetimeMSnow emit binary/date schema types; expect fewer warnings and possible schema updates in emitted metadata. No configuration changes required.Written for commit d7edbf0. Summary will update on new commits.