Kerisse search engine

Information for developers

• Add custom css in /src/assets/main.scss

• The references in manifest.json point to files in the final distribution: /dist

• /dist is generated from source via npm run build

• Start local webserver: npm start, this will start a server on http://localhost:8080/

Static search (Orama)

Search runs entirely in the browser. At build time, scraped JSONL entries are compiled into a gzipped Orama index (output/search-index.orama.json.gz, copied to dist/).

• Rebuild only the index: npm run build:search-index
• Full site build (index + webpack): npm run build
• Requires scraped entries in search-index-entries/ (.jsonl and .json), or a committed output/search-index.orama.json.gz fallback for CI

GitHub Actions needs one of these in the repo:

1. Preferred: commit search-index-entries/*.jsonl after scraping (.not-split backups stay gitignored)
2. Alternative: commit output/search-index.orama.json.gz (CI reuses it when entries are absent)

Scraping (search index)

Run a full scrape with npm run scrape (or sh scraper/main.sh). Output goes to search-index-entries/.

Browser: Puppeteer needs Chrome/Chromium. On macOS it uses Google Chrome by default; override with PUPPETEER_EXECUTABLE_PATH in .env, or run node node_modules/puppeteer/install.js.

Current inventory (generated): config/scrape-sources.md — refresh with npm run diagram:scrape.

---

Pick your goal

I want to…	Sitemap?	Where to edit
Index a GitHub repository	Automatic — built during scrape	config/github-repos.json
Index a website that already has sitemap.xml	Use theirs — point at the URL	config/generic-sites.mjs
Index a website with no sitemap	You create one	config/manual-sitemaps/ and config/generic-sites.mjs
Index one page only	None	config/single-urls/urls.json
Skip some URLs from a site	(uses that site’s sitemap)	config/exclude-urls/ and config/generic-sites.mjs
Add content without scraping	None	config/manual-entries/

---

I want to… index a GitHub repository

Indexes source files + issues via the GitHub API. You do not add a sitemap file — scrape generates it.

1. Open config/github-repos.json.
2. Append an object:

{ "owner": "WebOfTrust", "repo": "keripy", "branch": "main", "category": "Code" }

Field	Required	Meaning
owner	yes	GitHub org/user
repo	yes	Repository name
branch	yes	Branch to index
category	yes	Search facet (e.g. Code)
skipCrawl	no	true = keep existing JSONL; skip sitemap + scrape

3. Run npm run scrape (builds the sitemap, then scrapes).
4. Optional: npm run diagram:scrape.

Output: search-index-entries/{owner}-{repo}-{branch}.jsonl.

Not for GitHub Pages (*.github.io). Those are normal websites — use the website recipes below.

---

I want to… index a website that already has sitemap.xml

Typical for Docusaurus, ReadTheDocs, many docs sites. You point at their sitemap — you do not create one.

1. Confirm the site publishes a sitemap (try https://example.com/sitemap.xml).
2. Open config/generic-sites.mjs.
3. Copy a similar block (e.g. eSSIF-Lab for Docusaurus, keripy for ReadTheDocs, keri.foundation for WordPress).
4. Set at least:
   • sourceType: 'remoteXMLsitemap'
   • sourcePath: 'https://…/sitemap.xml'
   • siteName, source, category, destinationFile
   • domQueryForContent: DOM_QUERY.docusaurus (or .readTheDocs / .wordpressEntryContent)
5. In the export at the bottom, add:

scrape(configYourSite, scrapeSimple.docusaurus); // or readTheDocs / wordpress

Only lines in that export actually run.

6. Optional: skip unwanted URLs — see Skip some URLs.
7. npm run diagram:scrape, then npm run scrape.

---

I want to… index a website with no sitemap

You must create a sitemap and wire it — dropping an XML file alone does nothing.

1. Create an XML sitemap, e.g. config/manual-sitemaps/my-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/page-a</loc></url>
  <url><loc>https://example.com/page-b</loc></url>
</urlset>

(At scrape start this is copied to scraper/sitemaps/.)

2. Open config/generic-sites.mjs and add a config like Slack archive:

const configMySite = {
    sitemap: await createInput({
        sourceType: 'localXMLsitemap',
        sourcePath: 'scraper/sitemaps/my-site.xml',
    }),
    siteName: 'My Site',
    source: 'My Site',
    category: 'Blogs',
    author: '',
    destinationFile: 'search-index-entries/my-site.jsonl',
    domQueryForContent: DOM_QUERY.body, // or another DOM_QUERY.* that fits
};

3. Add to the export: scrape(configMySite, scrapeSimple.body);
4. npm run diagram:scrape, then npm run scrape.

---

I want to… index one page only

No sitemap. One JSON object per page.

1. Open config/single-urls/urls.json.
2. Append:

{
  "url": "https://example.com/post",
  "siteName": "Example Blog",
  "pageTitle": "My Post Title",
  "source": "Blogposts",
  "author": "Ada",
  "category": "Blogs",
  "querySelector": "article p, article h1, article h2, article li"
}

Field	Required	Meaning
url	yes	Page to scrape
querySelector	yes	CSS selector(s) for main content
pageTitle	yes	Title in index + output filename
siteName / source / category	recommended	Facets in search
author / type	optional

3. npm run scrape.

Output: search-index-entries/site-{index}-{pageTitle-slug}.jsonl
({index} = position in the array — inserting in the middle renumbers later files.)

---

I want to… skip some URLs from a site

Only applies to sites configured in generic-sites.mjs.

1. Create config/exclude-urls/my-site.json:

[
  "newsroom",
  "https://example.com/exact-page/"
]

• Fragments like "newsroom" = substring match (drops any URL containing it).
• Full http(s)://… URLs = exact match (trailing slash ignored).

2. In that site’s createInput({…}) in generic-sites.mjs, set:

excludeURLs: 'config/exclude-urls/my-site.json',

3. npm run scrape.

If the file is missing, the scraper logs an error and continues with no excludes.

---

I want to… add content without scraping

1. Put a .json or .jsonl file in config/manual-entries/.
2. Match the usual entry shape (url, content, pageTitle, category, …).
3. Run npm run scrape (or at least the copy step) — files are copied into search-index-entries/.

Examples already there: handmade.json, gleifPDF.jsonl.

---

Shared helpers

DOM presets live in scraper/modules/genericSiteScrape.mjs: DOM_QUERY.docusaurus, .readTheDocs, .wordpressEntryContent, .gleif, .body, plus makeScraper / scrapeSimple.*.