Skip to content

test: add 183 integration tests to close coverage gaps - #5

Merged
teng-lin merged 2 commits into
mainfrom
feature/test-coverage
Feb 8, 2026
Merged

teng-lin merged 2 commits into
mainfrom
feature/test-coverage

Conversation

@teng-lin

@teng-lin teng-lin commented Feb 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds 183 new tests across 8 test files + shared test-helpers.ts (683 → 866 tests)
  • Coverage improved: statements 80%→85%, branches 72%→77%, functions 89%→93%, lines 81%→86%
  • Targets the three biggest coverage gaps: wp-rest-api.ts (56%→93%), http-fetch.ts (80%→93%), text-density-extractor.ts (48%→69%)

New test files

File Tests Focus
wp-rest-api.test.ts 36 PMC list batching, envelope formats, teaser rejection, metadata enrichment
http-fetch-fallbacks.integration.test.ts 16 WP AJAX, Prism API, Next.js fallback chains
security-robustness.test.ts 33 Adversarial HTML, SSRF, prototype pollution, malformed inputs
text-density-extractor.test.ts 15 Success path, metadata extraction, error handling
extraction-edge-cases.test.ts 9 Nav-only HTML, multiple articles, hidden elements, tables
cookie-proxy-integration.test.ts 15 Cookie/proxy propagation through all 5 fallback paths
crawler-integration.test.ts 19 Sliding window concurrency, sitemap+robots, URL frontier
http-client-integration.test.ts 40 SSRF boundaries (IPv4-mapped IPv6, 172.16/12 edges), session management
test-helpers.ts — Shared fixtures: makeResponse, buildArticleHtml, mockExtraction

Test plan

  • npm run lint — clean (only pre-existing warning)
  • npm run format:check — all files pass
  • npm run test — 866 tests passing
  • npm run build — compiles successfully

🤖 Generated with Claude Code

Coverage improved from 80%→85% statements, 72%→77% branches, 89%→93% functions.

Key areas covered:
- wp-rest-api.ts: 56%→93% (PMC lists, envelope formats, teaser rejection)
- http-fetch.ts: 80%→93% (WP AJAX, Prism API, Next.js fallback chains)
- text-density-extractor.ts: 48%→69% (success path, metadata, error handling)
- crawler.ts: →97% (sliding window, sitemap+robots, URL frontier)
- http-client.ts: SSRF boundary tests (IPv4-mapped IPv6, 172.16/12 edges)
- Cookie/proxy propagation through all 5 fallback paths
- Security: adversarial HTML, prototype pollution, malformed inputs
- Shared test-helpers.ts to reduce fixture duplication

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello @teng-lin, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive suite of new integration tests designed to enhance the robustness and reliability of the codebase. The primary goal is to address existing code coverage deficiencies, particularly in critical areas related to HTTP fetching, content extraction, and API interactions. By adding a substantial number of tests, the changes aim to ensure that various edge cases, fallback mechanisms, and security considerations are thoroughly validated, leading to a more stable and predictable system behavior.

Highlights

  • Extensive Test Coverage: Added 183 new integration tests across 8 new test files and a shared test helper file, significantly expanding the test suite.
  • Improved Code Coverage Metrics: Code coverage saw substantial improvements: statements increased from 80% to 85%, branches from 72% to 77%, functions from 89% to 93%, and lines from 81% to 86%.
  • Targeted Coverage Gaps Addressed: Major coverage gaps in wp-rest-api.ts, http-fetch.ts, and text-density-extractor.ts were specifically targeted and improved, leading to more robust functionality in these critical areas.
Changelog
  • src/tests/cookie-proxy-integration.test.ts
    • Added tests for cookie propagation to initial page fetches and various fallback requests (WP REST API, Prism API, WP AJAX, Next.js data routes, PDF fetch).
    • Added tests for proxy propagation to initial page fetches and various fallback requests, including environment variable configuration.
    • Included tests for combined cookie and proxy propagation.
  • src/tests/crawler-integration.test.ts
    • Added tests for sliding-window concurrency in the crawler, ensuring efficient URL processing.
    • Included tests for sitemap and robots.txt interaction, verifying filtering of disallowed URLs and handling of nested sitemaps.
    • Added tests for URL frontier edge cases, such as link deduplication, cross-origin filtering, and URL normalization.
    • Included tests for crawler error resilience, ensuring continued operation despite fetch failures or robots.txt/sitemap issues.
    • Added tests for link extraction from various HTML structures (nav, footer, sidebar) and correct resolution of relative URLs.
    • Verified filtering of javascript:, mailto:, and tel: links.
  • src/tests/extraction-edge-cases.test.ts
    • Added tests for content extraction from HTML with only navigation/footer, multiple article tags, and hidden elements.
    • Included tests to ensure table content is preserved during extraction.
    • Added tests for extraction from minimal valid HTML and handling of insufficient content.
    • Included tests for deeply nested content structures, large amounts of boilerplate, and HTML entities.
  • src/tests/http-client-integration.test.ts
    • Added comprehensive tests for Server-Side Request Forgery (SSRF) protection, including IPv4-mapped IPv6 addresses, unspecified addresses, public IPs, and IPv4 private range boundaries.
    • Included tests for IP literal hostnames bypassing DNS resolution.
    • Added tests for proxy URL validation, including rejection of invalid schemes and DNS-resolved private proxies.
    • Included tests for redactProxyUrl functionality, ensuring sensitive information is masked.
    • Added tests for session management, verifying session cache keying by preset and proxy, and session reuse.
    • Included tests for httpPost content-type and body passing, and custom header merging.
    • Added tests for DNS edge cases, including handling both IPv4 and IPv6, and graceful failure on DNS resolution errors.
  • src/tests/http-fetch-fallbacks.integration.test.ts
    • Added tests for WP AJAX content fallback, verifying recovery from insufficient content and correct parsing of AJAX responses.
    • Included tests for Prism content API fallback, ensuring content recovery from __NEXT_DATA__ and metadata parsing.
    • Added tests for Next.js __NEXT_DATA__ fallback, verifying content extraction when siteUseNextData is enabled.
    • Included tests for the cascading fallback chain when initial content is insufficient.
    • Added tests for WP REST API enrichment, ensuring correct metadata merging when DOM content is preferred.
    • Verified WP AJAX handling of JSON-wrapped responses.
  • src/tests/security-robustness.test.ts
    • Added tests for SSRF and URL handling, ensuring SSRF protection errors are not retried and malformed URLs are handled gracefully.
    • Included tests for malicious HTML extraction, verifying script tag removal, handling of deeply nested elements, large text nodes, null bytes, data URIs, and unclosed tags.
    • Added tests for JSON parsing robustness, including truncated JSON and prototype pollution attempts.
    • Included tests for error handling paths, ensuring synchronous throws, non-Error rejections, and null rejections are caught gracefully.
    • Added tests for content validation edge cases, such as wrong content-type and 5xx status codes.
    • Included tests for HTML entity and encoding edge cases.
    • Added tests for retry logic, verifying correct retry behavior for network errors and non-retry for HTTP 4xx errors.
  • src/tests/test-helpers.ts
    • Added makeResponse, makeFailedResponse, and makeJsonResponse for consistent HTTP response mocking.
    • Added loremText and buildLongContent for generating test content.
    • Added buildArticleHtml and buildInsufficientContentHtml for creating realistic HTML page structures.
    • Added mockExtraction and mockFetchResult for creating mock extraction and fetch results.
  • src/tests/text-density-extractor.test.ts
    • Added tests for successful text density extraction, including content population and byline handling.
    • Included tests for metadata extraction (title, siteName, publishedTime, lang).
    • Added tests for excerpt generation.
    • Included tests for handling insufficient, empty, or pre-parsed HTML.
    • Added tests for error handling during extraction.
  • src/tests/wp-rest-api.test.ts
    • Added tests for tryWpRestApiExtraction, covering extraction from various WP API response formats (array, single object, custom envelope).
    • Included tests for PMC list detection and batch fetching, including truncation for large lists.
    • Added tests for rejecting content with WP API teasers and handling insufficient content length.
    • Included tests for API request failures and malformed JSON responses.
    • Added tests for extracting byline from _embedded.author and published time from date_gmt.
    • Included tests for handling string fields instead of rendered objects.
    • Added tests for appending _embed parameter correctly.
    • Included tests for inheriting siteName and lang from original results.
    • Added tests for enrichWpMetadata, extractSlugFromUrl, and resolveWpApiUrl functions.
Activity
  • The pull request was generated using Claude Code, indicating an AI-assisted development process.
  • The author has performed initial checks including linting, formatting, running all 866 tests, and building the project, all of which passed.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This is an excellent pull request that significantly improves test coverage with a large number of high-quality integration tests. The tests are well-structured, cover a wide range of scenarios including complex fallback logic, security robustness, and various edge cases. The use of shared test helpers is also a great practice. I've found only a couple of very minor issues in the test code, which I've detailed in the comments.

Comment on lines +78 to +79
let article_id = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
jQuery.ajax({ action: 'fetch_article_content', data: {id: articleId} });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There's a potential reference error in this JavaScript snippet. The variable is declared as article_id, but then used as articleId inside the jQuery.ajax call. To make the script valid, you should use a consistent variable name. Renaming article_id to articleId would resolve this.

Suggested change
let article_id = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
jQuery.ajax({ action: 'fetch_article_content', data: {id: articleId} });
let articleId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
jQuery.ajax({ action: 'fetch_article_content', data: {id: articleId} });

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in commit 09f22ac: Fixed variable name from article_id to articleId for consistency.

// Fragment-only link (#section-1) resolves to the same page → deduped as already visited
// /page#anchor and /page normalize to the same URL → fetched only once
const pageFetches = fetchedUrls.filter((u) => u === 'https://example.com/page');
expect(pageFetches.length).toBeLessThanOrEqual(1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This assertion is a bit weak. Given the test setup, the /page URL should be fetched exactly once. Using toBeLessThanOrEqual(1) would allow the test to pass even if the URL is not fetched at all (length 0), which would hide a potential bug in the URL normalization or frontier logic. A stricter assertion toBe(1) would be more appropriate here.

Suggested change
expect(pageFetches.length).toBeLessThanOrEqual(1);
expect(pageFetches.length).toBe(1);

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in commit 09f22ac: Strengthened assertion from toBeLessThanOrEqual(1) to toBe(1) — the URL should always be fetched exactly once.

- Fix variable name inconsistency (article_id → articleId) in WP AJAX fixture
- Strengthen assertion from toBeLessThanOrEqual(1) to toBe(1) for URL dedup test

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@teng-lin
teng-lin merged commit ad5fdb0 into main Feb 8, 2026
5 checks passed
@teng-lin
teng-lin deleted the feature/test-coverage branch February 21, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant