Skip to content

Fix missing thumbnails for retcon hubs: map ingestion1 object to preview; restore Maine and Kentucky profiles - #858

Merged
zspieges merged 1 commit into
mainfrom
829-fix-missing-thumbnails-for-retcon-hubs-maine-lc-kentucky-jsonretconmapping-maps-thumbnail-to-object-instead-of-preview
Nov 19, 2026
Merged

zspieges merged 1 commit into
mainfrom
829-fix-missing-thumbnails-for-retcon-hubs-maine-lc-kentucky-jsonretconmapping-maps-thumbnail-to-object-instead-of-preview

Conversation

@zspieges

@zspieges zspieges commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes the code part of #829. The re-map of lc, maine and kentucky on the ingest EC2 happens after this merges.

Problem

Records mapped by JsonRetconMapping (hubs migrated from ingestion1 dumps) have no thumbnails on dp.la, even though the source data has them. About 203,000 indexed items across Digital Maine, Library of Congress and Kentucky show the placeholder image.

  • The JSONL writer fills the index's "object" field (the thumbnail) from record.preview (model/package.scala, the line marked // in dpla map 3, object is the thumbnail).
  • JsonRetconMapping overrode `object` with ingestion1's _source.object, which was the thumbnail in ingestion1, and never set preview.
  • So the thumbnail went into a field the index never reads. No warning was raised, and JsonRetconMappingTest asserted `object`, so the tests passed.

Changes

  • JsonRetconMapping.scala: override def `object` is now override def preview, with the same body, plus a short comment explaining the mapping.
  • JsonRetconMappingTest.scala: "extract the correct object" is renamed "extract the correct preview", and all five retcon mappings (artstor, kentucky, lc, maine, washington) now check preview. Expected values are unchanged.
  • CHProviderProfiles.scala: restores KyProfile (KentuckyRetconMapping) and MeProfile (MaineRetconMapping), removed in Cleanup dead wood #608, so their legacy harvests can be re-mapped. They have the same shape as LocProfile (JsonProfile, DplaJsonlFileHarvester).
  • CHProviderRegistry.scala: adds "kentucky" -> new KyProfile and "maine" -> new MeProfile.

The registry is only read by name in MappingExecutor and HarvestExecutor, so existing hubs are unaffected.

Not in this PR

Testing

  • sbt "testOnly *JsonRetconMappingTest"
  • sbt test

🤖 Generated with Claude Code

This pull request fixes missing thumbnails for hubs mapped by JsonRetconMapping. It also adds Kentucky and Maine profiles so their legacy harvests can be re-mapped.

Changes

  • JsonRetconMapping.scala: The mapping body moves from override def \object`tooverride def preview. It still reads _source.objectand converts the values to URI-only web resources. Comments explain that ingestion1's thumbnail maps topreview, which the JSONL writer emits as the index's "object"` field.
  • CHProviderProfiles.scala: Adds KyProfile and MeProfile. Each extends JsonProfile, uses DplaJsonlFileHarvester, and returns its retcon mapping (KentuckyRetconMapping and MaineRetconMapping). Comments note that both are legacy harvests kept for re-mapping.
  • CHProviderRegistry.scala: Registers "kentucky" to KyProfile and "maine" to MeProfile.
  • JsonRetconMappingTest.scala: Per the summary, renames the test and checks mapping.preview for the same five provider records. I did not inspect this file's hunks, so the test details are not verified.

Diff size: 4 files, 37 lines added, 7 removed. This matches the stated counts.

Operational notes

  • The PR does not change environment variables, AWS Secrets Manager keys, database schema, CI/CD or infrastructure configuration, or public API shape or endpoints.
  • No security-relevant changes are visible in the diff.
  • Re-mapping the lc, maine, and kentucky harvests is outside this PR. It follows the merge and an sbt assembly run. Syncing results to S3 also follows the merge.
  • The PR description lists sbt "testOnly *JsonRetconMappingTest" and sbt test as unchecked. No test results are reported. Test status: not available.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change renames the mapping method from object to preview and updates its tests. It also adds Kentucky and Digital Maine profiles and registers both providers.

Changes

Preview mapping

Layer / File(s) Summary
Rename the preview mapping method
src/main/scala/dpla/ingestion3/mappers/providers/JsonRetconMapping.scala, src/test/scala/dpla/ingestion3/mappers/providers/JsonRetconMappingTest.scala
The mapping method and its tests use preview. Extraction and expected thumbnail values remain unchanged.

Kentucky and Digital Maine profiles

Layer / File(s) Summary
Add and register provider profiles
src/main/scala/dpla/ingestion3/profiles/CHProviderProfiles.scala, src/main/scala/dpla/ingestion3/utils/CHProviderRegistry.scala
The new profiles use their corresponding retcon mappings and the JSONL harvester. The registry maps kentucky and maine to those profiles.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: mdellabitta

Merge Risk: Minimal · up to 4a972

The change sets the preview field so hubs mapped by the retcon mapping get their thumbnails. It also restores Kentucky and Digital Maine profiles so their legacy harvests can be re-mapped. No concrete defects were found. Run the listed tests before merging, and check the remap results after merge as planned.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: the retcon mapping now writes ingestion1 object values to preview, which fixes missing thumbnails. It also names the restored Maine and Kentucky profiles. The PR objec…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: the retcon mapping now writes ingestion1 object values to preview, which fixes missing thumbnails. It also names the restored Maine and Kentucky profiles. The PR objectives describe the same change. The title is specific and matches the changeset, and it is a little long but still clear.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8af10013-873c-4e45-88d2-d45b6f9cfe50
Commits

Reviewing files that changed from the base of the PR and between b1f36b4 and 4a972f4.

Files selected for processing: 4

  • src/main/scala/dpla/ingestion3/mappers/providers/JsonRetconMapping.scala
  • src/main/scala/dpla/ingestion3/profiles/CHProviderProfiles.scala
  • src/main/scala/dpla/ingestion3/utils/CHProviderRegistry.scala
  • src/test/scala/dpla/ingestion3/mappers/providers/JsonRetconMappingTest.scala

Limit details: You’ve used the included review currently available.

@zspieges
zspieges merged commit 46534f5 into main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix missing thumbnails for retcon hubs (Maine, LC, Kentucky): JsonRetconMapping maps thumbnail to object instead of preview

1 participant