update and automate grant feed process - #1335
Conversation
…; template for email check logic started; do updates
… reporting failures; increase check email frequency to 6 hours
There was a problem hiding this comment.
🟡 Not ready to approve
Confirmed issues in the new delta deletion logic and process CLI option handling could cause incorrect Symplectic link deletions and prevent selecting staged input in supported modes.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR replaces the legacy Fuseki/SPARQL-based Aggie Enterprise → Symplectic grant-feed tooling with a pure-JS ETL integrated into the existing harvest/ + Dagster workflow, adds reporting/log-confirmation support, and removes the old grants-import/ implementation.
Changes:
- Add a new
harvest/lib/grant-feedlibrary (XML→CSV transform, generation→delta diff, GCS/email helpers, Symplectic log parsing, reporting DB loader). - Add new
experts harvest grant-feed ...CLI subcommands and wire them into Dagster jobs/schedules and the main harvest CLI. - Migrate XML parsing to
@ucd-lib/experts-commonsusingfast-xml-parser, and delete the legacygrants-import/scripts/templates and outdated docs.
File summaries
| File | Description |
|---|---|
| harvest/lib/grant-feed/transform.js | Implements fs-free XML→CSV row building to match the legacy pipeline output rules. |
| harvest/lib/grant-feed/symplectic-logs.js | Adds pure parsers for Symplectic import/delete log files and “meaningful log” checks. |
| harvest/lib/grant-feed/schema.sql | Introduces grant_feed reporting schema/tables/views for sent deltas + confirmations. |
| harvest/lib/grant-feed/reporting.js | Adds loaders to upsert weekly deltas and import/delete confirmations into Postgres. |
| harvest/lib/grant-feed/paths.js | Adds CasKFS path helpers for weekly grant-feed artifacts and Symplectic filename mapping. |
| harvest/lib/grant-feed/log-paths.js | Adds CasKFS path helpers for archived Symplectic logs (clean filenames). |
| harvest/lib/grant-feed/index.js | Exposes the public surface of the new grant-feed library. |
| harvest/lib/grant-feed/gcs.js | Adds GCS helpers for listing/downloading object generations for the weekly XML input. |
| harvest/lib/grant-feed/email.js | Introduces a pluggable (stubbed) email-ingest client interface for future automation. |
| harvest/lib/grant-feed/delta.js | Implements the generation-vs-generation delta algorithm as pure functions over row arrays. |
| harvest/lib/extract/cdl.js | Switches to using the shared xmlToJson from @ucd-lib/experts-commons. |
| harvest/bin/experts-harvest.js | Registers a new top-level grant-feed subcommand under experts harvest. |
| harvest/bin/experts-harvest-grant-feed.js | Adds dispatcher for grant-feed subcommands (git-style). |
| harvest/bin/experts-harvest-grant-feed-transform.js | Implements weekly transform CLI: resolve XML (CasKFS/GCS/local), write generation CSVs. |
| harvest/bin/experts-harvest-grant-feed-process.js | Implements orchestration CLI: transform + delta + upload + best-effort reporting load. |
| harvest/bin/experts-harvest-grant-feed-fetch-logs.js | Implements CLI to fetch Symplectic logs, archive meaningful ones, and load confirmations. |
| harvest/bin/experts-harvest-grant-feed-delta.js | Implements weekly delta CLI: find prior generation, compute delta, write delta CSVs. |
| harvest/bin/experts-harvest-grant-feed-check-email.js | Implements inbox polling CLI (currently stub/disabled) to stage weekly XML in CasKFS. |
| harvest/bin/experts-harvest-dagster.js | Adds a CLI command to launch the Dagster grant_feed_job. |
| grants-import/csv-templates/Prod_UCD_grants_to_delete.csv | Removes legacy template file (old pipeline). |
| grants-import/csv-templates/Prod_UCD_grants_persons.csv | Removes legacy template file (old pipeline). |
| grants-import/csv-templates/Prod_UCD_grants_metadata.csv | Removes legacy template file (old pipeline). |
| grants-import/csv-templates/Prod_UCD_grants_links.csv | Removes legacy template file (old pipeline). |
| grants-import/csv-templates/Prod_UCD_delete_user_grants_links.csv | Removes legacy template file (old pipeline). |
| grants-import/bin/experts-grant-feed.js | Removes legacy Fuseki-based transform/export script. |
| grants-import/bin/experts-grant-feed-process.js | Removes legacy process wrapper script. |
| grants-import/bin/experts-grant-feed-get-logs.js | Removes legacy log fetch + Slack posting script. |
| grants-import/bin/experts-grant-feed-delta.js | Removes legacy delta computation script. |
| doc/grant-process-steps.md | Removes outdated Fuseki-based step-by-step instructions. |
| doc/grant-feed.md | Updates grant-feed documentation to match the new harvest/Dagster-based ETL and log confirmation flow. |
| dagster/lib/schedules.py | Adds (registered) daily log-fetch schedules and defines (but notes on-hold) email-check schedules. |
| dagster/lib/jobs.py | Adds Dagster asset jobs for grant-feed ingest, email-check (on hold), and log fetch. |
| dagster/lib/assets.py | Adds Dagster assets for grant-feed ingest, log fetch, and (on-hold) email-check-triggered runs. |
| dagster/defs.py | Registers grant-feed ingest + log-fetch assets/jobs/schedules; keeps email-check unregistered (on hold). |
| commons/package.json | Adds fast-xml-parser dependency for shared XML parsing. |
| commons/package-lock.json | Locks fast-xml-parser + transitive deps. |
| commons/lib/xml-to-json.js | Re-implements XML→JSON using fast-xml-parser and shared logger. |
| commons/lib/config.js | Adds grantFeed configuration (GCS input, email placeholders, Symplectic SFTP/log paths) + schema registration. |
| commons/index.js | Exports xmlToJson from @ucd-lib/experts-commons for shared use. |
Review details
Files not reviewed (1)
- commons/package-lock.json: Generated file
- Files reviewed: 26/39 changed files
- Comments generated: 5
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
There was a problem hiding this comment.
🟡 Not ready to approve
The delta logic can emit incomplete metadata for multi-funder grants pulled in via link-only changes, and a couple of CLI/log-fetch behaviors need validation/determinism fixes before this can be safely relied on in production.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Files not reviewed (1)
- commons/package-lock.json: Generated file
Suppressed comments (1)
harvest/lib/grant-feed/delta.js:174
- addGrantsLinked() builds
newByIdwithnew Map(newGrants.map(g => [g.id, g])), which drops all-but-one metadata row for grants that legitimately have multiple rows (e.g. multiple funders). If a grant is pulled into the delta only because of link changes, this will send an incomplete metadata set for that grant.
- Files reviewed: 27/40 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
There was a problem hiding this comment.
🟡 Not ready to approve
There are confirmed correctness/operational issues in the new ETL tooling (reporting date_uploaded can be erased on re-runs, log folder selection is non-deterministic, and --env is not validated), which should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Files not reviewed (1)
- commons/package-lock.json: Generated file
Suppressed comments (2)
harvest/lib/grant-feed/reporting.js:49
- The metadata upsert overwrites
date_uploadedwith the new run’s value unconditionally. If you re-run the same week with--no-uploadafter a successful upload, this will setdate_uploadedback to NULL and lose the audit trail. It also makes re-runs non-idempotent with respect to upload time. Consider preserving the existing non-NULL timestamp and only filling it when it is currently NULL.
harvest/lib/grant-feed/delta.js:51 - rowKey() uses a literal SOH control character as the join delimiter. This is hard to see in diffs/editors and can cause confusing tooling behavior. Using an explicit escape (e.g.
\u0001) keeps the same semantics while making the intent obvious.
- Files reviewed: 27/40 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
There was a problem hiding this comment.
🟡 Not ready to approve
There are correctness and robustness issues in the new grant-feed pipeline (notably PI rows being emitted into the c-co-pis field and missing env validation / doc mismatches) that should be resolved before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Files not reviewed (1)
- commons/package-lock.json: Generated file
Suppressed comments (2)
harvest/lib/grant-feed/transform.js:57
- PERSON_ROLES includes 'Principal Investigator', but buildPersonRows always emits field-name 'c-co-pis'. That means any PI participant rows will be imported into the co-PI field, duplicating/misclassifying the PI already carried in grants_metadata via the 'c-pi' column. Either restrict PERSON_ROLES to Co-PI only, or make field-name depend on the participant role (e.g. c-pi vs c-co-pis).
doc/grant-feed.md:196 - The REMS sponsor URL examples are missing a slash ("http:/rems..."). This is easy to copy/paste incorrectly when debugging; it should be "http://rems..." (matching SPONSOR_IRI_PREFIX in transform.js).
|c-ucop-sponsor|http:/rems.ucop.edu/sponsor/[CODE]|URL with valid REMS sponsor code|
|c-flow-thru-funding|http:/rems.ucop.edu/sponsor/[CODE]|URL with valid REMS sponsor code|
- Files reviewed: 27/40 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
|
Added a comment to the code addressing the main issue of the last AI review. Ready for human review |
Addresses #1028