This project is an extension of the book Can We Be Wrong? The Problem of Textual Evidence in a Time of Data (Cambridge 2020). It asks how we can make textual interpretation more credible and align humanistic methods with cross-disciplinary frameworks of evidentiary evaluation. Rather than consider two different evaluation frameworks for the humanities and the sciences -- or the humanities and computational humanities -- the book argues that both should be subject to the same criteria, which I call the Open Generalization Framework.
Initial results can be found here: https://piperandrew.github.io/Inspector_General/
- The Telos of Generalization. Both the objects and concepts about which the article generalizes should be clearly stated. Generalization goals should not be implicit.
- Evidence Transparency. A) The number and kind of units that will be analyzed should be expressly stated. B) Their representativeness of a more general population should be discussed. What is known about the population they come from and how do the units discussed represent that population?
- Analytic Transparency. The methods that will be used to arrive at generalized claims need to be made explicit. What dimensions of source texts will be analyzed? How will they be selected? And how will they be interpreted?
- Limitations Transparency. Have the limitations of the method, evidence, and claims been explicitly reflected upon?
To facilitate the self-assessment of humanities research, Inspector General prompts an LLM to read humanities research articles and produce a compact diagnostic report for each PDF along with a summary table for further analysis. For each article, its primary outputs are:
- the central generalizable claim cited verbatim
- the number of pieces of evidence used to arrive at that claim
- a summary of the claim plus evidence
- an Open Generalization Score
- a set of subsidiary information including: geographic region of the evidence, timeframe covered, identification of the causal/caused units (i.e. central concepts/entities of the claim), and individual transparency scores for each aspect of the final open generalization score.
Current package version: 1.7.0.
The current default model is OpenAI gpt-5.6-sol through the Responses API, using low reasoning effort and schema-validated output. All prompts are stored as inspectable text files:
src/inspector_general/prompts/article_diagnostic.txtsrc/inspector_general/prompts/article_input_template.txtsrc/inspector_general/prompts/taxonomy_classification.txtsrc/inspector_general/prompts/taxonomy_input_template.txt
Python 3.10 or later is required.
python -m venv .venv
source .venv/bin/activate
python -m pip install .
cp .env.example .envAdd your OpenAI API key to the local .env file. It is ignored by Git. You can instead export OPENAI_API_KEY or use the masked --prompt-for-key option; never put a key directly on the command line.
Place PDFs directly in input/ or in any number of nested subdirectories:
input/
├── article.pdf
└── Journal Title/
├── second-article.pdf
└── issue/
└── third-article.pdf
Process every PDF:
inspector-general inspect input --reports allOr process an exact number in deterministic relative-path order:
inspector-general inspect input --reports 25The defaults are input for the input directory, reports.csv for the combined CSV, and reports/ for Markdown reports, so inspector-general inspect is equivalent to processing every PDF below input/. Override the destinations when needed:
inspector-general inspect input \
--reports all \
--output results/diagnostics.csv \
--report-directory results/reportsThe CSV contains one row per PDF. It includes the source filename, a directory column containing the relative parent directory (blank for top-level PDFs), every diagnostic value, separate score and rationale columns, and the computed Open Generalization Score.
Markdown output mirrors the input directory tree to avoid collisions. For example, input/Journal Title/article.pdf becomes reports/Journal Title/article_report.md.
Long analyses run sequentially in OpenAI's background mode. The CLI prints per-file progress, cancels the active remote response when you press Control-C, and applies a 30-minute timeout to each PDF by default. Set a different per-PDF limit with --timeout SECONDS.
Inspection is resumable by default. Before making another API request, the CLI
restores current Markdown reports into the combined CSV and skips their source
PDFs. It checkpoints the CSV after every newly completed report. Existing reports
older than their source PDFs, or reports that cannot be parsed, are regenerated.
Use --force only when you intentionally want to reprocess every selected PDF.
If a completed response violates a local schema constraint, such as the
one-sentence evidence requirement, the CLI retries it once by default. Set the
total number of validation attempts with --validation-attempts N. A PDF that
still fails is reported at the end without stopping later articles; rerunning the
same command retries only those unfinished PDFs.
Exclude input-relative paths with repeatable glob patterns. Quote patterns so the shell does not expand them before the CLI receives them:
inspector-general inspect input \
--exclude 'Test/**' \
--exclude 'Religion_wrong_MDPI/**' \
--exclude '**/Cover-*.pdf' \
--exclude '**/FrontCover-*.pdf' \
--exclude '**/Notices-*.pdf'Every Markdown report contains these fields:
- Metadata: Article title and author.
- Region: The most specific geographic scope explicitly covered, such as a city, nation, group of nations, continent, or the world. It is limited to 20 words and uses
Not specifiedwhen the article supplies no geographic scope. - Time Frame: Explicit dates, years, decades, centuries, or a named period covered by the article. It is limited to 20 words and uses
Not specifiedwhen the article supplies no time frame. - Central Claim: One exact quotation expressing the article's central generalizable claim.
- Central Causal Unit: The central class or concept being discussed, expressed as a noun phrase of at most 12 words.
- Central Causal Unit Class: The causal unit's entity type, such as person, place, institution, technology, practice, concept, or emotion. It is limited to 6 words.
- Central Caused Unit: Whom or what the causal unit affects, or what it does, expressed as a noun phrase of at most 12 words.
- Central Caused Unit Class: The caused unit's entity type, following the same classification principle and 6-word limit.
- Evidence: One sentence of at most 60 words naming the evidence used to support the central claim.
- N: The explicit or estimated non-duplicated count of identifiable evidence units.
- Summary: Exactly two or three complete sentences of at most 40 words each, restating the claim, evidence, and estimated
N. - Evidence Transparency: A 0-5 score and one rationale sentence of at most 40 words.
- Analytical Transparency: A 0-5 score and one rationale sentence of at most 40 words, assessing how clearly the article states its methods and analytical approach.
- Limitations Transparency: A 0-5 score and one rationale sentence of at most 40 words.
- Open Generalization Score: The deterministic sum of the three component scores, presented as
<score>/15 (<label>).
The final score labels are:
- 0-4:
poor - 5-8:
low - 9-11:
medium - 12-15:
strong
To check one PDF's extraction before spending API credits:
inspector-general extract input/article.pdf --output article-extracted.txtScanned PDFs need OCR before use. Extraction is limited to 800,000 characters by default as a cost and context safeguard. The CLI explains how to raise the limit when needed.
After producing reports.csv, run the secondary workflow:
inspector-general classify reports.csvThis classifies every report row and atomically updates the same CSV. To preserve the original and write a separate table:
inspector-general classify reports.csv --output reports_classified.csvThe workflow preserves every existing column and appends:
region_class
time_frame_class
causal_unit_class
caused_unit_class
region_class is constrained to neighborhood, city, state/province, country, multi-country, or global. time_frame_class is constrained to year or less, years, decade, decades, century, centuries, or millennia. If the original report does not specify a region or time frame, the corresponding normalized field is blank.
The causal and caused unit taxonomy is hierarchical. The family headings organize the controlled leaf classes; only the leaf class is written to reports.csv.
PersonCognitive processEmotionIdentity / demographic attributeBehavior
Social groupSocial relationshipsOrganizationGovernmentLawPolicySocial system
Economic SystemResource
Text / cultural artifactLanguageGenre / aesthetic formConceptNorm / value / beliefReligion / ritual / symbol
ProcessPlaceBuilt environmentEnvironmentGeophysical system
Organism / speciesBody / physiological systemGeneticsDisease / health
Material substancePhysical processTechnologyAlgorithm / data system
Other / unclear is reserved as a fallback when a unit genuinely cannot be assigned to one of these families after review.
The classifier defaults to low reasoning. It accepts the same --model and --reasoning-effort controls as inspect; for example:
inspector-general classify reports.csv --reasoning-effort lowThe faster default is low for both workflows. Move reasoning effort down or up when needed:
inspector-general inspect input \
--reports all \
--model gpt-5.6-sol \
--reasoning-effort highAccepted reasoning levels are none, low, medium, high, xhigh, and max.
Model availability and API usage charges depend on your OpenAI account. Pin a dated model snapshot when reproducibility matters more than automatically tracking an alias.
Journals_List.md is also an input to an authenticated journal-harvesting
workflow. It discovers the final numbered issue published in 2025 for each
journal, labels each contents entry as article, review, or other, and
downloads articles through a persistent browser session:
Full setup, authentication, output, classification, and troubleshooting details
are in the
journal_harvest workflow README.
python -m pip install .
playwright install chromium
inspector-general harvest-journals Journals_List.md \
--proxy-domain "proxy3.library.mcgill.ca"The browser opens the first selected article through McGill's proxy and pauses
if sign-in is required. Credentials are not read or stored by Inspector
General; Chromium keeps the resulting session in the git-ignored
.journal-browser/ profile. The workflow rewrites publisher hostnames in the
same form as the Project MUSE proxy URL.
PDFs are written beneath input/ in compact journal directories containing no
spaces, for example:
input/
├── AmericanHistoricalReview/
│ ├── ArticleTitle-a1b2c3d4.pdf
│ └── SecondArticle-e5f6a7b8.pdf
└── GLQAJournalOfLesbianAndGayStudies/
└── AnotherArticle-11223344.pdf
journal_harvest.csv records the selected volume and issue, DOI, classification,
download path, status, and failure detail for every contents entry. Reviews and
front matter are recorded but skipped by default; use --include-reviews to
download reviews too. Article download attempts are separated by a random
two-to-five-second delay. To inspect issue selection and classifications before
opening a browser:
inspector-general harvest-journals --discovery-onlyThe workflow selects the chronologically final numbered issue among Crossref
records published from January 1 through December 31, 2025. The manifest makes
the selection auditable. Use --year to target another year. Publisher markup
differs, so authenticated pages that do not expose a recognizable PDF link are
retained as failed rows rather than silently omitted.
filename
directory
title
author
region
time_frame
central_claim
central_causal_unit
central_causal_unit_class
central_caused_unit
central_caused_unit_class
evidence
n
summary
evidence_transparency_score
evidence_transparency_rationale
analytical_transparency_score
analytical_transparency_rationale
limitations_transparency_score
limitations_transparency_rationale
open_generalization_score
open_generalization_label