Repository navigation
Conversation
10 tasks
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the code part of #829. The re-map of lc, maine and kentucky on the ingest EC2 happens after this merges.
Problem
Records mapped by
JsonRetconMapping(hubs migrated from ingestion1 dumps) have no thumbnails on dp.la, even though the source data has them. About 203,000 indexed items across Digital Maine, Library of Congress and Kentucky show the placeholder image."object"field (the thumbnail) fromrecord.preview(model/package.scala, the line marked// in dpla map 3, object is the thumbnail).JsonRetconMappingoverrode`object`with ingestion1's_source.object, which was the thumbnail in ingestion1, and never setpreview.JsonRetconMappingTestasserted`object`, so the tests passed.Changes
JsonRetconMapping.scala:override def `object`is nowoverride def preview, with the same body, plus a short comment explaining the mapping.JsonRetconMappingTest.scala: "extract the correct object" is renamed "extract the correct preview", and all five retcon mappings (artstor, kentucky, lc, maine, washington) now checkpreview. Expected values are unchanged.CHProviderProfiles.scala: restoresKyProfile(KentuckyRetconMapping) andMeProfile(MaineRetconMapping), removed in Cleanup dead wood #608, so their legacy harvests can be re-mapped. They have the same shape asLocProfile(JsonProfile,DplaJsonlFileHarvester).CHProviderRegistry.scala: adds"kentucky" -> new KyProfileand"maine" -> new MeProfile.The registry is only read by name in
MappingExecutorandHarvestExecutor, so existing hubs are unaffected.Not in this PR
JsonRetconMappingmaps thumbnail toobjectinstead ofpreview#829 and follow after merge andsbt assembly.objectfield from the trait, model, Avro schema, Parquet export and docs (tracked separately; see Remove unusedobjectmappings from NARA, Smithsonian, and Tennessee mappers #828 for the NARA, Smithsonian and Tennessee mappings).Testing
sbt "testOnly *JsonRetconMappingTest"sbt test🤖 Generated with Claude Code
This pull request fixes missing thumbnails for hubs mapped by
JsonRetconMapping. It also adds Kentucky and Maine profiles so their legacy harvests can be re-mapped.Changes
JsonRetconMapping.scala: The mapping body moves fromoverride def \object`tooverride def preview. It still reads_source.objectand converts the values to URI-only web resources. Comments explain that ingestion1's thumbnail maps topreview, which the JSONL writer emits as the index's"object"` field.CHProviderProfiles.scala: AddsKyProfileandMeProfile. Each extendsJsonProfile, usesDplaJsonlFileHarvester, and returns its retcon mapping (KentuckyRetconMappingandMaineRetconMapping). Comments note that both are legacy harvests kept for re-mapping.CHProviderRegistry.scala: Registers"kentucky"toKyProfileand"maine"toMeProfile.JsonRetconMappingTest.scala: Per the summary, renames the test and checksmapping.previewfor the same five provider records. I did not inspect this file's hunks, so the test details are not verified.Diff size: 4 files, 37 lines added, 7 removed. This matches the stated counts.
Operational notes
sbt assemblyrun. Syncing results to S3 also follows the merge.sbt "testOnly *JsonRetconMappingTest"andsbt testas unchecked. No test results are reported. Test status: not available.