Skip to content

S3: "unsupported extension" warning for file names containing dots when default_extension is set #19690

Closed

Description

@rospe

Describe the bug

S3Source.get_fields uses pathlib.Path(...).suffix, which is purely lexical and returns everything after the last dot. For S3 files whose names contain dots — e.g. events.account.update-2026-05-27-<hash>.gz — stripping the .gz compression suffix leaves .update-2026-05-27-<hash> as the apparent format extension. That is not a real format, so the inferrer emits an "unsupported extension" warning.

Because the leftover value is non-empty, the default_extension fallback on the path_spec is never reached — so even with default_extension: json set, every such file warns and is not resolved.

To Reproduce

  1. Configure an S3 path_spec with default_extension: json.
  2. Point it at files whose names contain dots before the compression suffix, e.g. s3://.../events.account.update-2026-05-27-11-<hash>.gz.
  3. Observe an "unsupported extension" warning for every such file; default_extension is not applied.

Expected behavior

When a file name has no recognised format extension (csv, tsv, json, parquet, avro), the configured default_extension should be used so the file is ingested, instead of warning. Files with a recognised extension (including compressed *.json.gz, *.parquet.gz) should behave as before.

Version

DataHub ingestion (S3 source), current master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions