P2 - Discussion Request: File Metadata from submitters, to be used in dewrangle to link submitted dds and dfs

[brendagutman]

This is related to the long-term ingest process We are currently(7/23/26) still using the short-term solution.

Purpose:

1. To validate a datafile(study data) against a data dictionary via Airflow DAG, a datafile will need to be associated with a data dictionary.

Note - Study Metadata: Information about a study that is important to proper harmonization, or documentation of a study, may not be included in the submitted datafiles. i.e. consent, principal investigator,etc This information should be gathered in a structured manner to reduce hardcoding in the dbt pipeline, and increase data integrity. The strategy discussed in this GH Issue could double as the structured solution we need for this note's topic, as well.

Option 1:

• Use dewrangle directory structure and/or strict naming conventions to make associations between files.
• Pro: This option may not require any code changes. It may require modifications on the dewrangle or DAGs.
• Con: Depending on the number of files, naming and storage could become painful for users. Manual processes are error prone.

Option 2:

• Require a dataset metadata file that defines data domains (Subject,Family,etc). For each domain a data dictionary will be identified, as well as any datafile(s)
• Pros:
  • This option is the most robust, and hits 3 birds with one stone. Handles file associations, study metadata collection and structuring, as well as using a file format that is close to the one harmonizers would create to use the pipeline_utils.
    Note: Below, Family_format1 and Family_format2 highlight a common issue. Files within a domain that don't comply to the same data dictionary. i.e. Using the column 'participant_id' in one file and 'subject_id' in another.

study_id: brainpower_IV
dataset_id: BrainPower_IV_Syn_V3
principal_investigator: {pi_name}
consent_id: GRU
data_dictionary:
  Subject:
    identifier: participant_dd.csv
  Family_format1:
    identifier: family_dd.csv
  Family_format2:
    identifier: family_dd.csv
data_file:
  Subject:
    identifier:
      - participant.csv
  Family_format1:  
    identifier:
    - family_1.csv
    - family_2.csv
  Family_format2:
    identifier:
    - family_3.csv

Option 3:

• Require a single data dictionary file submission. The data dictionary should span every datafile(study data) submitted. The data dictionary should have a column that defines each of the following: A dataset identifier, a comma separated list of the datafiles each row is associated with.
• Cons: Some pipeline tooling is not prepared for dds that cover more than one table. Modifications would be required to account for the change in dd format.

dataset_identifier	datafile_list	variable_name	description	...
BrainPower_IV_SynV3	family_1.csv, family_2.csv, participant.csv	subject_id	A masked, unique, identifier for a subject	...
BrainPower_IV_SynV3	family_3.csv	participant_id	A masked, unique, identifier for a subject	...

[j-kanuch]

@brendagutman
Option 2 is my favorite, by far. I would tweak the structure of the metadata manifest to:

1. optionally include DD versioning (specify an object ID, or default to latest)
2. Potentially Rearrange the "data_dictionary" node structure to group by domains.

Perhaps something like:

domains:
  - name: Subject
    schema_variant: default
    data_dictionary: participant_dd.csv
    data_files:
      - participant.csv

  - name: Family
    schema_variant: format1
    data_dictionary: family_dd.csv
    data_files:
      - family_1.csv
      - family_2.csv

  - name: Family
    schema_variant: format2
    data_dictionary: family_dd.csv
    data_dictionary_version_id: <hash or version number or URI from Dewrangle>
    data_files:
      - family_3.csv