Skip to content

Add elasticsearch-migration skill - #8

Merged
thomasht86 merged 3 commits into
mainfrom
thomasht86/add-es-migration-skill
May 18, 2026
Merged

thomasht86 merged 3 commits into
mainfrom
thomasht86/add-es-migration-skill

Conversation

@thomasht86

Copy link
Copy Markdown
Contributor

Summary

Adds a new top-level skill, elasticsearch-migration/, to the vespa-skills plugin. It is the coordinator skill for ES→Vespa migrations and delegates schema, query, feed, and deploy details to the existing sibling skills.

The skill builds on two upstream resources from the Vespa project:

  • Vespa migration guide — docs.vespa.ai/en/learn/migrating-from-elastic-search.html — the canonical 5-phase migration flow (export → convert → app package → deploy → feed) is taken from this doc.
  • ES_Vespa_parser.py — the starter mapping-to-schema converter from vespa-engine/sample-apps. docs/parser-and-tooling.md walks through its invocation alongside elasticdump, and documents what the parser does not handle (rank profiles, nested fields, analyzer-specific behavior, embeddings, doc IDs, indexing directives) so users know what to hand-edit.

Files

  • elasticsearch-migration/SKILL.md — overview, 5-phase flow, concept-mapping cheat sheet, indexing-directives callout, ranking-divergence warning, gotchas, LLM-friendly doc-fetching section.
  • elasticsearch-migration/docs/concept-mapping.md — field-by-field ES → Vespa mapping with before/after snippets.
  • elasticsearch-migration/docs/query-translation.md — ES Query DSL → YQL side-by-sides (match, bool, range, function_score, knn, nested, aggs, sort).
  • elasticsearch-migration/docs/parser-and-tooling.md — ElasticDump + ES_Vespa_parser.py end-to-end with the parser's limitations.
  • elasticsearch-migration/docs/fetching-docs.md — how to retrieve fresh Vespa and Elasticsearch docs on demand (Vespa .html.md per-page suffix, llms.txt, llms-full.txt; Elastic has no llms.txt, fall back to elastic/docs-content on GitHub or elastic.co/docs/).
  • .claude-plugin/marketplace.json, AGENTS.md, README.md — regenerated by vespaskills generate.

Design constraint

No Vespa or Elasticsearch doc content is embedded inline — the skill teaches Claude how to fetch fresh docs so the content cannot go stale. Every technical claim (YQL operators, schema directives, ES query DSL shape, ElasticDump flags) is grounded in a verbatim quote from the live upstream docs.

Test plan

  • uv run python -m vespaskills.cli generate --check passes
  • All Vespa claims audited against docs.vespa.ai/en/reference/{schemas,querying,ranking}/* and corrected (bool-query rank() form, index { enable-bm25 } block, struct inside document, prenormalized-angular semantics, input.query(name) request param, current canonical URLs)
  • All Elasticsearch / ElasticDump claims audited against www.elastic.co/docs/* and the elasticsearch-dump README (NDJSON output format, full --type enum, current /docs/ URL pattern, docs-content GitHub repo)
  • Live smoke-test: open a session in this repo, ask "I want to migrate my Elasticsearch index to Vespa, where do I start?", confirm the skill is surfaced
  • Follow-up PR: add data/evals_elasticsearch-migration.json modeled on data/evals_app-package.json

🤖 Generated with Claude Code

Coordinator skill for ES→Vespa migrations: 5-phase flow, concept
mapping, query translation, parser tooling, and LLM-friendly doc-
fetching patterns. Defers schema/query/feed/deploy specifics to
existing sibling skills.

Every Vespa, Elasticsearch, and ElasticDump claim is grounded in
verbatim quotes from the live docs (Vespa /reference/{schemas,
querying,ranking}/* via the .html.md form; Elastic /docs/reference/*;
elasticsearch-dump README). Doc URLs use current canonical paths,
not the redirect stubs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@thomasht86
thomasht86 requested a review from radu-gheorghe May 12, 2026 12:42
radu-gheorghe
radu-gheorghe previously approved these changes May 15, 2026

@radu-gheorghe radu-gheorghe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left a few comments; most of them are small. Only significant one: I'd use Logstash instead of Elasticdump.

Great stuff, though! It captures a lot of the mechanics and the general flow.

}

rank-profile default {
first-phase { expression: nativeRank }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consistency nitpick: I'd decide whether we want to steer agent to BM25 or nativeRank. Right now it says nativeRank here and bm25 in the upcoming schema snippet.

Since analysis won't be equivalent until somebody uses Lucene linguistics, I would stick to nativeRank and remove enable-bm25 for the simplest migration.

Comment thread elasticsearch-migration/SKILL.md Resolved
Comment thread elasticsearch-migration/SKILL.md Resolved
Comment thread elasticsearch-migration/SKILL.md Resolved
Comment thread elasticsearch-migration/SKILL.md Resolved
Comment thread elasticsearch-migration/SKILL.md Resolved
@thomasht86
thomasht86 merged commit 5f04f51 into main May 18, 2026
2 checks passed
@abhishekkrthakur
abhishekkrthakur deleted the thomasht86/add-es-migration-skill branch May 18, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants