diff --git a/README.md b/README.md index 3ffae36..22ad9bb 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ npx skills add firecrawl/skills | [`firecrawl-build-scrape`](./skills/firecrawl-build-scrape) | Integrate `/scrape` for single-page extraction | Authored here | | [`firecrawl-build-search`](./skills/firecrawl-build-search) | Integrate `/search` for discovery-first workflows | Authored here | | [`firecrawl-build-interact`](./skills/firecrawl-build-interact) | Integrate `/interact` for clicks, forms, and dynamic flows after scrape | Authored here | -| [`firecrawl-research-index`](./skills/firecrawl-research-index) | Find papers in the research paper index — biomedical and life-science literature (PubMed, bioRxiv, medRxiv) plus arXiv preprints | Authored here | -| [`firecrawl-developer-index`](./skills/firecrawl-developer-index) | Answer developer questions from issues, pull requests, READMEs, and documentation pages | Authored here | +| [`firecrawl-research-index`](./skills/firecrawl-research-index) | Find papers (PubMed, bioRxiv, medRxiv, arXiv); `search --categories research` is a website filter, not this index | Authored here | +| [`firecrawl-developer-index`](./skills/firecrawl-developer-index) | Search issues, merged PRs, READMEs, and documentation for library/API/error/bug questions | Authored here | ## MCP Server diff --git a/skills/firecrawl-developer-index/SKILL.md b/skills/firecrawl-developer-index/SKILL.md index 3e302fe..87bec55 100644 --- a/skills/firecrawl-developer-index/SKILL.md +++ b/skills/firecrawl-developer-index/SKILL.md @@ -1,59 +1,41 @@ --- name: firecrawl-developer-index -description: Search issues, merged pull requests, READMEs, and documentation. Use when the question is how a library or API behaves, what an error means, or whether a bug was fixed; prefer this over a general web page. +description: | + Search issues, merged pull requests, READMEs, and documentation. Use when the question is how a library or API behaves, what an error means, or whether a bug was fixed; prefer this over a general web page. +allowed-tools: + - Bash(firecrawl *) + - Bash(npx firecrawl-cli *) --- -# Firecrawl Developer Index +# firecrawl developer -Answer a developer question from the primary source: the issue where the bug was reported, the merged pull request that fixed it, the README or documentation page that states the contract. A blog post that describes a behaviour is a weaker answer than the passage that defines it, so reach for the index first and the open web second. +Answer a developer question from the primary source: the issue, the merged pull request that fixed it, or the README/docs passage that states the contract. -There is **no fixed recipe**. Read the question, decide what kind it is, and choose the approach below. A literal error string wants a different move than "how do I do X". Don't run machinery a question doesn't call for. +## Quick start -## The tools, and what each is uniquely good at +```bash +mkdir -p .firecrawl +firecrawl developer "how do I configure retries" --limit 10 -o .firecrawl/developer.json --json +jq -r '.results[] | .id, .url, .passages[].text' .firecrawl/developer.json +``` -- HTTP: **`GET|POST https://api.firecrawl.dev/v2/search/developer`** - MCP: **`firecrawl_developer_search(query, k?, skills?)`** - CLI: **`firecrawl developer [--limit ] [--skills-only]`** - Ranked results over the whole index. Each carries `id` (`issue:owner/repo#123`), `url`, and the **matched passages in markdown**, so tables and code blocks survive. The artifact kind is the `id` prefix: `doc:`, `issue:`, `pull_request:`, or `readme:`. - The default first move for a developer question. It is the only surface that returns the passages, which is what lets you answer instead of pointing at a page. - `k` / `--limit` is 1–100 and defaults to 10. `skills="only"` / `--skills-only` restricts the search to agent-skill files. - Keyless; send `Authorization: Bearer $FIRECRAWL_API_KEY` for higher rate limits. +Run `firecrawl developer --help` for the full option list. -- MCP: **`firecrawl_search(query, categories: ["developer"])`** - CLI: **`firecrawl search --categories developer`** - Developer hits in a `developer` group beside `web`, each with `url`, `title`, `description` (the matched passage), `position`, and `category: "developer"` — web results carry no `category`, so that is the field to key on when merging. - Use this when you are **already** running a web search and want developer sources weighed in the same call. It exposes none of the filters and no passage control. +HTTP: `GET|POST https://api.firecrawl.dev/v2/search/developer`. MCP: `firecrawl_developer_search`. Each hit carries `id`, `url`, `passages`. Kind is the `id` prefix (`doc:`, `issue:`, `pull_request:`, `readme:`). Hits do not carry a `type` field. -- MCP: **`firecrawl_scrape(url)` / `firecrawl_search(query)`** - CLI: **`firecrawl scrape ` / `firecrawl search `** - General web fetch and search, for what no primary source states: a comparison between two libraries, an outage, a migration write-up, a project with no public repository or indexed docs. - Also the follow-through when a hit is the right page but you need all of it — `scrape` the result's `url`. +**Done when:** the answer quotes a matched passage and cites its `url` (fall back to `url` when `title` is absent), or you have moved to the open web because the index had nothing to say. -## Filters, and what each one costs you +## Tips -Only the HTTP surface takes these. On `GET`, pass `types=issue,pull_request` or repeat the parameter; on `POST`, pass arrays. All are optional. +- Default first move is `firecrawl developer`. Use `search --categories developer` only when you are already running a web search and want developer hits in the same call (no passage control, no index filters). +- Literal error or stack trace: search the string plus the library name. On HTTP, `types=["issue","pull_request"]`. Strip paths, line numbers, and ids, then retry. +- API contract: `readme` and `doc` are authoritative. A merged PR supersedes an issue report. Never answer from an opening report alone. +- Scope last: search the whole index, then narrow with HTTP `types`, `repos`, or `sources`. If a scoped search is empty, read the echoed `indexed` flag before concluding the repo is missing. +- Repository filters (`language`, `topic`, `license`, `min_stars`, …) drop `doc` results unless you also pass `sources`. `types`, `repos`, `sources`, `passages`, and those repository filters are HTTP-only. +- Comparison, opinion, news, or an unindexed project: `firecrawl search`, then `firecrawl scrape`. -- `types` — which of `doc`, `issue`, `pull_request`, `readme` to search. Defaults to all four. Narrowing here is the cheapest way to sharpen a query. -- `repos` (`owner/name`) scopes the repository half, meaning `issue`, `pull_request`, and `readme`; `sources` (documentation source ids, at most 20) scopes the documentation half, meaning `doc`. Passing both **unions** the halves rather than intersecting them. Both echo back in the response with `indexed: true|false` — that is how you tell "not in the index" from "found nothing". -- A filter that cannot match any requested `type` is a `400`, not an empty list: `repos` with no repository type in `types`, or `sources` without `doc`. -- `passages` (1–5, default 1) is the *maximum* passages per result, not a guarantee. Raise it when one page is clearly the right page but the first passage is the wrong part of it. -- `language`, `topic`, `license`, `min_stars`, `max_stars`, `archived`, `fork` describe a **repository**. Most documentation pages in the index have no repository behind them, so no repository fact can admit or exclude one. Send any of these without a `sources` scope and the response holds repository evidence only — `issue`, `pull_request`, `readme`. That is the design, not an index fault: do not retry it and do not report the index broken. To keep documentation, drop the repository filters, or scope the documentation half with `sources` and read the `sources` echo to confirm the id is indexed. +## See also -## Match the approach to the question - -- **Literal error message or stack-trace string** → search the string itself plus the library name, with `types=["issue","pull_request"]`. Whoever hit it filed it. If nothing matches, strip the volatile parts (paths, line numbers, ids, addresses) and retry — the invariant middle of the message is what is indexed. -- **Conceptual "how do I do X"** → the full question in natural language, all four types. The answer is usually a `doc` or a `readme`; raise `passages` before raising `k`. -- **Known bug** → the issue reports it, the merged pull request *fixes* it, and the fix is what you want. Search `types=["issue","pull_request"]`, then re-query the issue's own terms scoped to its repo with `types=["pull_request"]`. A merged PR's passages tell you what changed and in which direction. -- **API contract** ("what does X return", "is Y required", "what is the default") → `readme` and `doc` are authoritative and a blog post is not. Use `types=["readme","doc"]`. If the contract looks like it moved, follow up with `pull_request` for the change that moved it. -- **Version-specific behaviour** → an issue's opening report describes the broken version; its resolution supersedes it. Raise `passages` to see further into the thread, and read the resolution and the linked pull request before answering. Never answer from an opening report alone. -- **Scoped to one library** → `repos=["owner/name"]` when you know the slug, plus `sources` if you want its docs in the same call. If a scoped search comes back empty, read the echoed `indexed` flag first: `false` means nothing from that repo or source can ever match and no rephrasing will help — drop the scope and search the whole index, or go to the web. -- **Ecosystem-wide** ("which libraries do X", "who else hit this") → no scope. Use `language` / `topic` / `min_stars` to keep to maintained repositories, accepting that this gives up all `doc` results. -- **Agent skills and tooling conventions** → `skills="only"` / `--skills-only`. -- **Comparison, opinion, news, or an unindexed project** → the open web. `firecrawl_search`, then `firecrawl_scrape` whatever deserves a full read. Combining is often right: take the contract from the index and the trade-off from the web. - -## Principles - -- **Quote the passage, cite the `url`.** The passages are the evidence; hand them over rather than paraphrasing them into a claim the reader can't check. `title` is frequently absent on `doc` results — fall back to `url`. -- **A merge supersedes a report.** When an issue and a pull request disagree, the merged pull request is the current behaviour. Say which one you read. -- **Scope last, not first.** Search the whole index, then narrow with `types`, `repos`, or `sources` once you know what the hits look like. Scoping first hides the result that would have told you where to look. -- **Go to the web when the index has nothing to say.** Trade-offs, ecosystem opinion, and anything about an unindexed project are web questions. Don't force them through the index, and don't dress a general web page up as a primary source. +- [Developer Index docs](https://docs.firecrawl.dev/features/developer) +- [firecrawl-research-index](../firecrawl-research-index/SKILL.md) — papers, not this index +- [firecrawl-build-search](../firecrawl-build-search/SKILL.md) — product-code `/search` diff --git a/skills/firecrawl-research-index/SKILL.md b/skills/firecrawl-research-index/SKILL.md index 5fbc7fa..fa7cca3 100644 --- a/skills/firecrawl-research-index/SKILL.md +++ b/skills/firecrawl-research-index/SKILL.md @@ -1,69 +1,43 @@ --- name: firecrawl-research-index -description: Find the papers that answer a research query in Firecrawl's research paper index — a corpus of paper abstracts whose largest share is biomedical and life-science literature (PubMed, bioRxiv, medRxiv), alongside arXiv preprints in CS, physics, and math — using semantic search, semantic and structural expansion, and in-body verification. Use this skill for literature-finding and paper-retrieval tasks of any kind, including clinical, biomedical, drug, gene, disease, and other life-science questions, whether the answer is a single paper or a full multi-paper set. The index is reached only through the `firecrawl_research_*` MCP tools or the `firecrawl research` CLI subcommands. Calling `firecrawl_search` with its `categories` option set to `["research"]` is a different feature — it filters ordinary web search to research-affiliated websites (the list includes PubMed, bioRxiv, medRxiv, arXiv, and publisher sites) and returns page results from them, without querying the paper records in this index. +description: | + Find papers in Firecrawl's research paper index (PubMed, bioRxiv, medRxiv, arXiv). Use for literature-finding of any kind, including clinical and biomedical questions; `search --categories research` is a website filter, not this index. +allowed-tools: + - Bash(firecrawl *) + - Bash(npx firecrawl-cli *) --- -# Firecrawl Research Index +# firecrawl research -Find the research papers that answer a research query. Some questions have a single answer; many have several — and when in doubt, lean toward returning the fuller relevant set (most relevant first) rather than narrowing to one. A reader is better served seeing the neighboring methods and papers than having them silently dropped. +Find the papers that answer a research query. When in doubt, return the relevant set (most relevant first) rather than one hit. -## What is in the index +## Quick start -Paper abstracts, with full text reachable per paper. The largest share of the corpus is **biomedical and life-science** literature — **PubMed** journal articles plus **bioRxiv** and **medRxiv** preprints — so clinical, drug, gene, disease, epidemiology, and public-health questions are in scope. **arXiv** preprints cover computer science, physics, and mathematics. Coverage outside those sources is thinner: a paper that exists only behind a publisher paywall or in a niche venue may not be indexed, and the general web tools below are the fallback when it isn't. +```bash +mkdir -p .firecrawl +firecrawl research search-papers "CRISPR base editing off-target effects" \ + --limit 20 -o .firecrawl/papers.json --json +jq -r '.results[] | .primaryId, .title' .firecrawl/papers.json +``` -There is **no fixed recipe**. Read the query, decide what kind it is, and choose the approach below. Some queries need a single search; others need heavy sturctural/semantic expansion. Don't run machinery a query doesn't call for. +Run `firecrawl research --help` for flags. MCP arguments use `paperId`, not `id`. -## The tools, and what each is uniquely good at +A successful `search-papers` response is `{success, results}`. Each hit carries `paperId`, `primaryId` (`pmid:`, `pmcid:`, `doi:`, or `arxiv:`), `ids`, `title`, `abstract`, and `score`. -- MCP: **`firecrawl_research_search_papers(query, k?)`** - CLI: **`firecrawl research search-papers [--k ]`** - Semantic (HyDE) search over **abstracts**. The natural first move for almost any query. - If results look thin or all-alike, re-run with a different framing (sibling domain, rival method, dataset/benchmark name) rather than giving up. +**Done when:** the answer is a cited paper set (or the one named paper), each kept or dropped against a verified constraint, with `search-papers` as the first move unless the query already named an id. -- MCP: **`firecrawl_research_related_papers(seed_ids, intent, mode?, k?)`** - CLI: **`firecrawl research related-papers --intent [--mode ] [--k ]`** - Semantic and structural expansion, ranked to your `intent`. - This reaches papers semantic search *cannot*, and it's how you turn one good hit into the rest of a set. - `mode=similar` → niche siblings; `citers` → who uses/builds on the seeds; `references` → what they build on / compare against. +## Tips -- MCP: **`firecrawl_research_inspect_paper(id)`** - CLI: **`firecrawl research inspect-paper `** - Canonical metadata for **one** paper: title, abstract, authors, categories, source ids, and dates. - Use it after `search_papers` or `related_papers` when you need the complete citation/metadata for a candidate, or when you have an id from elsewhere and need to confirm what paper it resolves to. - This does **not** read the paper body; use `read_paper` for specific full-text questions. +- `search-papers` is the first move. If results look thin or all-alike, re-run with a different framing (sibling domain, rival method, dataset/benchmark name). +- `related-papers` needs `--intent`. `mode=similar` for siblings, `citers` for who builds on the seeds, `references` for what they build on. +- `inspect-paper` is metadata for one id. `read-paper` is in-body passages for one constraint (sample size, method, affiliation). Use it to rule a paper out, not to gatekeep. +- `search --categories research` is a website filter. It returns pages from academic domains, not paper records in this index. +- Named paper ("the Qwen3 report") → one `search-papers`. Method / family / "papers that do X" → expand with `related-papers` and keep neighbors. +- Superlative / leaderboard questions live on the web: `firecrawl search` / `firecrawl scrape`, then `search-papers` each top entry. +- PubMed, bioRxiv, and medRxiv are the largest part of the corpus. Do not send a biomedical query to the open web on the assumption the index is arXiv-only. -- MCP: **`firecrawl_research_read_paper(id, question)`** - CLI: **`firecrawl research read-paper --question `** - In-body passages of **one** paper, to verify a load-bearing constraint (a method actually used, a score actually reported, an affiliation, what a paper compares to). - Use it to settle a specific doubt, not on everything. +## See also -- MCP: **`firecrawl_search(query, categories: ["research"])`** - CLI: **`firecrawl search --categories research`** - **Not this index.** This is a *website* filter: it restricts a normal web search to a short list of research-affiliated domains — the list does include `pubmed.ncbi.nlm.nih.gov`, `biorxiv.org`, `medrxiv.org`, and `arxiv.org` alongside publisher sites — and returns page results in a `research` group beside `web`, each with `url`, `title`, `description` (the matched passage), `position`, and `category: "research"` — web results carry no `category`, so that is the field to key on when merging. - So it reaches those sites' **web pages**; what it does not do is query their **paper records** in this index — no semantic search over abstracts, no citation-graph or related-paper expansion, no canonical paper metadata, and no in-body passages. The results are ordinary web results. - Use it when you are **already** running a web search and want those sites weighed in the same call. For anything that is actually a paper-finding task, use `firecrawl_research_search_papers` and its siblings above. - -- MCP: **`firecrawl_search(query)` / `firecrawl_scrape(url)`** - CLI: **`firecrawl search ` / `firecrawl scrape `** - General **web** search and page fetch, for facts that don't live in paper abstracts: benchmark **leaderboards**, rankings, "who scores best / is largest / is most used." - Find the ranking on the web, then map the top entries back to papers with `search_papers`. - Reach for these only when the corpus can't answer the question on its own. - -## Match the approach to the query - -- **Single *named* paper** ("the Qwen3 report") → one `search_papers`, done. This is the only case that truly wants exactly one paper. -- **Paper by description / by method or technique** ("the paper that introduced X", "training-free N-gram detection of AI text") → find the best match, then assume there's a *family*: expand with `related_papers` and **include the closely-related methods/papers too**. Even when one paper is the exact literal match, surface and keep its neighbors — don't narrow to the single best hit and reason the rest out. Only treat it as one-answer if the query names a specific paper. -- **Enumeration / method-family** ("papers that do X", "alternatives to Adam", "benchmarks for Y") → the answer is a *set*, and this is where `related_papers` earns its keep: expand several strong anchors with `mode=similar`, re-seed from new strong hits. One search is never enough here. -- **Exhibiting** ("papers that *use* / exhibit property P") → the relevant papers apply P but their abstracts may not describe it. Go from P's defining paper outward via `citers`/`references`, and use `read_paper` to confirm a candidate actually uses P. -- **Superlative / leaderboard** ("best on benchmark X", "largest", "most popular") → the ranking lives on **leaderboards / the web**, not in any single abstract. Use `firecrawl_search` / `firecrawl_scrape` to find the benchmark's leaderboard or rankings, read off the top models/papers, then `search_papers` each to get its paper. As a fallback, search the benchmark and `read_paper` candidates for reported numbers. The hardest kind — cast wide. -- **Org / author filtered** ("from \", "by \") → topical match isn't enough; verify the affiliation/authorship (metadata or `read_paper`) before keeping a paper. -- **Compare-against** ("what does paper X benchmark against / build on") → the answer is *inside* paper X: `read_paper(X, ...)` or `related_papers([X], ..., mode="references")`. - -## Principles - -- **Two different features share the word "research."** The paper index is `firecrawl_research_*` / `firecrawl research`. The `categories: ["research"]` option on `firecrawl_search` is a website filter — it does point web search at PubMed, bioRxiv, medRxiv, arXiv, and publisher sites, but what comes back is their web pages, not paper records. If a task is about finding papers, the tools in this skill are the ones that read the corpus; reaching for `categories: ["research"]` will quietly answer a different question. -- **Query shape and subject field are separate.** A clinical-trial question and a machine-learning question take the same shapes above; what differs is only which source the hits come from. Don't send a biomedical or life-science query to the open web on the assumption the corpus is arXiv-only — PubMed, bioRxiv, and medRxiv are the largest part of what `search_papers` reads. -- **When in doubt, include.** For any topic / method / comparison question, return the relevant *family*, not just the single best match — err toward keeping a plausibly-relevant paper rather than dropping it. The neighboring methods are part of a good answer; don't reason close work out just because one paper is the most exact match. -- **Follow the literature, and keep what you find.** The seminal source, the competing methods, the close neighbors are usually a hop away — use `related_papers`, and *include* them, not just the first hit. Stopping at one good result is the most common way to leave the reader with half an answer. -- **Verify to exclude, not to gatekeep.** Use `read_paper` to rule a paper *out* when a hard constraint clearly fails (wrong org/author, doesn't actually report the score). When a paper is plausibly relevant, lean toward keeping it rather than demanding proof. -- **Only drop the clearly off-topic.** Don't pad with papers you're confident are unrelated — but that's a high bar; most plausibly-relevant work should make the cut. +- [Research Index docs](https://docs.firecrawl.dev/features/research) +- [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) — issues, PRs, READMEs, and docs +- [firecrawl-build-search](../firecrawl-build-search/SKILL.md) — product-code `/search`