Repository navigation
Description
Summary
Records produced by JsonRetconMapping (hubs migrated from ingestion1 dumps) have no thumbnails in DPLA, even though the source data has them. The mapping puts the ingestion1 thumbnail URL into object instead of preview, and only preview reaches the index. This affects about 203,000 indexed items across Digital Maine, Library of Congress and Kentucky. Every item page shows the placeholder image.
This issue covers the mapping fix and re-running all three affected hubs on the ingest EC2 so the legacy thumbnail data actually reaches the index. Related: #828 (removing the unused object mappings from NARA, Smithsonian and Tennessee).
Cause
- The JSONL writer fills the index's
"object"field (the thumbnail) fromrecord.preview(src/main/scala/dpla/ingestion3/model/package.scala, the line marked// in dpla map 3, object is the thumbnail; that's been the case since 2017). JsonRetconMapping(added in Feature/json retcon #499, 2022) overrides`object`with the ingestion1_source.objectvalue and never overridespreview. In ingestion1,objectwas the thumbnail.- So the value goes into a field the index never reads. There's no warning or error, and
JsonRetconMappingTestasserts`object`, so the tests pass.
Impact
Measured against the 2022 JSONL in S3: records whose embedded ingestion1 source has an object, compared with the output.
| Hub | Source had thumbnail → output has none | Source thumbnails still resolve? | In index? |
|---|---|---|---|
| maine | 56,948 / 63,492 | Yes (30/30 sampled, digitalmaine.com/.../thumbnail.jpg) |
Yes |
| lc | 4,463 / 4,463 | Yes (3/3 sampled, cdn.loc.gov) |
Yes |
| kentucky | 141,563 / 141,608 | No: nyx.uky.edu timed out on 9/9 from a local network (EC2 not checked) |
Yes |
| artstor, washington | — | — | No |
Checked on dp.la: Maine b38cf47a4dad65d9069bbb580b810c0e, LC d4d689923afa428245def109f4313be9 and Kentucky 91920237723081663dde1438fa53c3c6 all show /static/placeholderImages/Image.svg.
Tasks
Code
-
JsonRetconMapping.scala: change theoverride def `object`tooverride def preview(same body). -
JsonRetconMappingTest.scala(~L265–291): assertpreviewinstead of`object`for all five retcon mappings. - Restore the Maine and Kentucky profiles so they can be re-mapped. Both were removed in Cleanup dead wood #608 ("Cleanup dead wood"), and both use the same shape as the existing
LocProfile:CHProviderProfiles.scala: add backMeProfile(MaineRetconMapping) andKyProfile(KentuckyRetconMapping), eachextends JsonProfilewithgetHarvester = classOf[DplaJsonlFileHarvester].CHProviderRegistry.scala: add back"maine" -> new MeProfileand"kentucky" -> new KyProfile.
-
sbt testpasses.
Re-run on the ingest EC2
No re-harvest: just re-map each hub's existing 2022 harvest. Use the standard SSM workflow and run sbt assembly after merging.
- lc: copy
s3://dpla-master-dataset/lc/harvest/20220720_122938-lc-OriginalRecord.avro/to$DPLA_DATA/lc/harvest/, then run./scripts/remap.sh lc. - maine: copy
s3://dpla-master-dataset/maine/harvest/20220725_111048-maine-OriginalRecord.avro/to$DPLA_DATA/maine/harvest/, then run./scripts/remap.sh maine. - kentucky: copy
s3://dpla-master-dataset/kentucky/harvest/20220720_122521-kentucky-OriginalRecord.avro/to$DPLA_DATA/kentucky/harvest/, then run./scripts/remap.sh kentucky. i3.conf only haskentucky.included_in_index = truefor this hub. That should be enough because the mapping hardcodes the provider, but if the remap needs another key, add a minimalkentuckyblock toingestion3-conf. - Verify before syncing:
- The mapping
_SUMMARYshows the same attempted/successful counts as 2022: lc 4,463; maine 63,492; kentucky 141,608. - The number of JSONL lines with a non-empty
"object"is about 4,463 (lc), about 56,948 (maine) and about 141,563 (kentucky). - DPLA IDs are unchanged: the JSONL
idset is identical to the 2022 JSONL. The mapping copies each record's existing DPLA ID (_source.id) through, so the IDs shouldn't change.
- The mapping
- Sync with
./scripts/s3-sync.shforlc,maineandkentucky. If anomaly detection flags the change in thumbnail coverage, that's expected. - After the next monthly index rebuild, confirm the Maine and LC items above show real thumbnails on dp.la, and that the Kentucky item now has
objectset in the index. Its thumbnail may still render as the placeholder whilenyx.uky.eduis unreachable.
Notes
- Kentucky's legacy thumbnail URLs may be dead, but they're the data the hub provided. This issue makes the index reflect that data correctly. When the source is unreachable the thumbnail API returns an error (502) and dp.la shows the placeholder, so the user-visible result is no worse than today. Repairing stale legacy data (Kentucky thumbnails, and similar legacy issues in Maine and LC) is a separate operation.
lc/enrichment/,maine/enrichment/andkentucky/enrichment/each contain a20251229_000000-*-MAP4_0.EnrichRecord.avrodirectory (written 2025-12-31, no matching JSONL). The re-run supersedes it as the latest enrichment.- Removing
objectfrom the trait, model and Avro schema, the Parquet export (batch-process-dpla-index) and the docs will be tracked separately.
Activity