Skip to content

Fix missing thumbnails for retcon hubs (Maine, LC, Kentucky): JsonRetconMapping maps thumbnail to object instead of preview #829

Closed

Description

@DominicBM

Summary

Records produced by JsonRetconMapping (hubs migrated from ingestion1 dumps) have no thumbnails in DPLA, even though the source data has them. The mapping puts the ingestion1 thumbnail URL into object instead of preview, and only preview reaches the index. This affects about 203,000 indexed items across Digital Maine, Library of Congress and Kentucky. Every item page shows the placeholder image.

This issue covers the mapping fix and re-running all three affected hubs on the ingest EC2 so the legacy thumbnail data actually reaches the index. Related: #828 (removing the unused object mappings from NARA, Smithsonian and Tennessee).

Cause

  • The JSONL writer fills the index's "object" field (the thumbnail) from record.preview (src/main/scala/dpla/ingestion3/model/package.scala, the line marked // in dpla map 3, object is the thumbnail; that's been the case since 2017).
  • JsonRetconMapping (added in Feature/json retcon #499, 2022) overrides `object` with the ingestion1 _source.object value and never overrides preview. In ingestion1, object was the thumbnail.
  • So the value goes into a field the index never reads. There's no warning or error, and JsonRetconMappingTest asserts `object`, so the tests pass.

Impact

Measured against the 2022 JSONL in S3: records whose embedded ingestion1 source has an object, compared with the output.

Hub Source had thumbnail → output has none Source thumbnails still resolve? In index?
maine 56,948 / 63,492 Yes (30/30 sampled, digitalmaine.com/.../thumbnail.jpg) Yes
lc 4,463 / 4,463 Yes (3/3 sampled, cdn.loc.gov) Yes
kentucky 141,563 / 141,608 No: nyx.uky.edu timed out on 9/9 from a local network (EC2 not checked) Yes
artstor, washington — — No

Checked on dp.la: Maine b38cf47a4dad65d9069bbb580b810c0e, LC d4d689923afa428245def109f4313be9 and Kentucky 91920237723081663dde1438fa53c3c6 all show /static/placeholderImages/Image.svg.

Tasks

Code

  • JsonRetconMapping.scala: change the override def `object` to override def preview (same body).
  • JsonRetconMappingTest.scala (~L265–291): assert preview instead of `object` for all five retcon mappings.
  • Restore the Maine and Kentucky profiles so they can be re-mapped. Both were removed in Cleanup dead wood #608 ("Cleanup dead wood"), and both use the same shape as the existing LocProfile:
    • CHProviderProfiles.scala: add back MeProfile (MaineRetconMapping) and KyProfile (KentuckyRetconMapping), each extends JsonProfile with getHarvester = classOf[DplaJsonlFileHarvester].
    • CHProviderRegistry.scala: add back "maine" -> new MeProfile and "kentucky" -> new KyProfile.
  • sbt test passes.

Re-run on the ingest EC2

No re-harvest: just re-map each hub's existing 2022 harvest. Use the standard SSM workflow and run sbt assembly after merging.

  • lc: copy s3://dpla-master-dataset/lc/harvest/20220720_122938-lc-OriginalRecord.avro/ to $DPLA_DATA/lc/harvest/, then run ./scripts/remap.sh lc.
  • maine: copy s3://dpla-master-dataset/maine/harvest/20220725_111048-maine-OriginalRecord.avro/ to $DPLA_DATA/maine/harvest/, then run ./scripts/remap.sh maine.
  • kentucky: copy s3://dpla-master-dataset/kentucky/harvest/20220720_122521-kentucky-OriginalRecord.avro/ to $DPLA_DATA/kentucky/harvest/, then run ./scripts/remap.sh kentucky. i3.conf only has kentucky.included_in_index = true for this hub. That should be enough because the mapping hardcodes the provider, but if the remap needs another key, add a minimal kentucky block to ingestion3-conf.
  • Verify before syncing:
    • The mapping _SUMMARY shows the same attempted/successful counts as 2022: lc 4,463; maine 63,492; kentucky 141,608.
    • The number of JSONL lines with a non-empty "object" is about 4,463 (lc), about 56,948 (maine) and about 141,563 (kentucky).
    • DPLA IDs are unchanged: the JSONL id set is identical to the 2022 JSONL. The mapping copies each record's existing DPLA ID (_source.id) through, so the IDs shouldn't change.
  • Sync with ./scripts/s3-sync.sh for lc, maine and kentucky. If anomaly detection flags the change in thumbnail coverage, that's expected.
  • After the next monthly index rebuild, confirm the Maine and LC items above show real thumbnails on dp.la, and that the Kentucky item now has object set in the index. Its thumbnail may still render as the placeholder while nyx.uky.edu is unreachable.

Notes

  • Kentucky's legacy thumbnail URLs may be dead, but they're the data the hub provided. This issue makes the index reflect that data correctly. When the source is unreachable the thumbnail API returns an error (502) and dp.la shows the placeholder, so the user-visible result is no worse than today. Repairing stale legacy data (Kentucky thumbnails, and similar legacy issues in Maine and LC) is a separate operation.
  • lc/enrichment/, maine/enrichment/ and kentucky/enrichment/ each contain a 20251229_000000-*-MAP4_0.EnrichRecord.avro directory (written 2025-12-31, no matching JSONL). The re-run supersedes it as the latest enrichment.
  • Removing object from the trait, model and Avro schema, the Parquet export (batch-process-dpla-index) and the docs will be tracked separately.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions