Replace GNU autogen as the opts/man-page generator (#895 phase 2) - #1030
Merged
Conversation
First stage of replacing EOL GNU autogen: scripts/autoopts/defparser.py parses the AutoOpts .def files (heredocs protected from comment stripping, -D conditionals, -L include resolution, adjacent-string values, repeatable blocks) into an order-preserving JSON IR. scripts/autoopts/validate_ir.py cross-checks the IR for all seven tool configurations (including -DTCPREPLAY_EDIT) against the committed autogen output: option names and order must match the INDEX_OPT_* enums - modeling AutoOpts 'documentation' pseudo-flags, which occupy an index slot without emitting a constant (why tcprewrite's enum starts at 1) - and OPTION_CT must equal doc + real + automatic options. All seven pass. Next stages, oracle-driven via scripts/check-generated-opts.sh: emitters for *_opts.h, then *_opts.c (string pool with byte offsets, descriptor tables, usage reflow), then man pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Status, oracle method, and the structural notes needed to implement the .h/.c/man emitters (license text tables, string pool offsets, reflow, doc pseudo-flag indexing, per-tool generation flags). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
emit_h.py generates all seven tool headers (including the -DTCPREPLAY_EDIT variant) byte-identical to the committed autogen output, from the defparser IR. check_emitters.py is the equivalence gate and runs alongside validate_ir.py. Oracle-derived layout rules are recorded in the README: enum field width max(maxname+1, 21), fixed %-24s value column with overflow, blank-line placement around OPT_VALUE, the shared hex counter feeding save/load-opts, the extended SET_OPT_NAME(a) form for settable options with arguments, doc pseudo-flags supplying the HAVE_OPT example name, and load-opts gating EXIT_NO_CONFIG_INPUT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tcpcapinfo, tcpliveplay and tcpprep _opts.c now match autogen's output exactly. Oracle-derived rules so far: string pool with cumulative offsets; 75-column refill with sentence-spacing normalization and two-space joins after sentence-ending lines; texinfo @file{} -> 'x'; FLAGS defines wrapped greedily at 75; must/cant index arrays with guarded-NULL else branches; save/load-opts pool entries, descriptors and disablement strings; homerc/apzHomeList; string vs numeric arg-defaults; flag-code passthrough preserving heredoc blank lines with def-file line references; unguarded range procs folded into the sorted 3-per-row static proc list; OPTPROC_NO_ARGS vs ARGS_REQ from the argument attribute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e 3) emit_c.py now reproduces every committed *_opts.c exactly, including the -DTCPREPLAY_EDIT variant; check_emitters.py gates all 14 generated files (.h and .c). The xgettext paragraph grouping resisted inference, so rather than guessing it is now a faithful port of libopts' optionPrintParagraphs() from libopts/usage.c - autogen's mk-gettextable delegates directly to it, and since we vendor that same libopts the algorithm is authoritative and cannot drift. Other oracle-derived behavior: deduplicated string pool with cumulative offsets; 75-column refill (72 for descrips) with whitespace collapse and two-space sentence joins; texinfo @file{}/@var{} -> 'x' and @item as a paragraph break; documentation pseudo-flags emitting OPTST_DOCUMENT descriptors, lib-name-prefixed optDesc_p pointers and counting as user options; blind uppercasing of flags-must/cant targets (tcpbridge legitimately references a cachefile option it lacks, inside an ifdef-guarded block); bare 'default' attribute selecting the default-opt index; stack-arg, must-set and equivalence descriptor forms. Verified by negative control: perturbing a .def descrip changes the emitted output, so the comparison is not degenerate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
generate.py regenerates all 14 *_opts.c/h files from the .def sources (--check verifies without writing). End-to-end verification with autogen physically removed from the system: deleting every src/*_opts.c/h and regenerating reproduces them byte-identically (git diff empty), the tree configures and builds, the full 'sudo make test' suite passes, and both regressions that sank PR #991 are confirmed working - --load-opts (via the prep_config test case that #991 broke) and --more-help. README records why man pages are still autogen-only: unlike the parser emitters, which port an algorithm from the libopts we vendor, the man pipeline is ~1900 lines of external Scheme and Perl with no in-tree source of truth. Three options with costs are written up for a maintainer decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generates AsciiDoc man-page source directly from the .def IR (descrip/ doc/explain/detail - the same content already driving --help and the option tables), rendered to roff with `asciidoctor -b manpage`. This replaces GNU autogen's man-page pipeline without hand-duplicating any documentation, avoiding the mistake in the reverted PR #991 (which hand-wrote separate .adoc files that could drift from --help). Unlike emit_h.py/emit_c.py this is not byte-oracle-verified: autogen's mdoc pipeline (cmd-doc.tlib + mdoc2man + Mdoc.pm, ~1900 lines of Scheme and Perl) has no in-tree source of truth to port the way optionPrintParagraphs() was ported from the vendored libopts. Rationale is in the emitter's module docstring. check_adoc.py is the validation gate instead: every real option from the .def must appear in the rendered page, asciidoctor must render it, and groff -man -ww must parse the result with zero warnings. All seven tool configurations (including -DTCPREPLAY_EDIT) pass. texi_to_adoc() handles the texinfo subset actually used in the .def corpus: @file/@var -> italic, @samp/@code -> monospace, @table/ @enumerate + @item (both the inline-label and bare-@item-then-@var-on- next-line forms) -> nested AsciiDoc description lists with proper +/ open-block continuation syntax, @example and mid-paragraph indented- line runs -> literal blocks, and texinfo's "- " item-body marker convention stripped rather than rendered as a spurious bullet. Content parity spot-checked against the current autogen-rendered tcpcapinfo.1: same NAME/options/exit-codes/authors/copyright/bugs content, reformatted into proper subsections instead of run-on paragraphs. Known gap: man-doc attribute content (SIGNALS/custom BUGS/SEE ALSO) is not rendered - matching current behavior, since that content turns out absent from the committed autogen-generated pages too (verified by grep). Noted in README.md as a possible follow-up. generate.py now also writes src/*.adoc alongside *_opts.c/h. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tcpedit_stub.def spells the required prologue line "AutoGen definitions options;" (capitalized) while every other .def uses lowercase "autogen definitions options;" - both are accepted by real autogen. Also documents tcpedit_stub.h as a deliberate scope boundary: it uses a different AutoOpts "library" template mode (no copyright/program identity; points at another program's exported optDesc_p) than the seven emit_h.py targets, and stays on phase 1's safe committed + conditional-regen pattern rather than getting a bespoke emitter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pts/man (#895 stage 5, autotools half) src/Makefile.am: every src/*_opts.c/h rule now invokes emit_c.py/emit_h.py directly (one file, one rule, matching how the scripts already work as CLI tools) instead of GNU autogen. New rules generate src/*.adoc via emit_adoc.py and render src/*.1 from it via `asciidoctor -b manpage`, replacing the agman-cmd.tpl pipeline entirely for these seven tools. tcpreplay_edit's OBJECTS dependency is corrected to depend on its own tcpreplay_edit_opts.h (previously named tcpreplay_opts.h, a latent typo masked by BUILT_SOURCES ordering). configure.ac: AC_PATH_PROG for PYTHON3 and ASCIIDOCTOR, each substituted with a scripts/missing-* stub (mirroring the existing AUTOGEN pattern) when absent and something is stale, so a firing rule fails with clear instructions instead of silently using stale output. The pre-existing autogen-version-matching check, which used to gate on src/tcpreplay_opts.c's presence as a proxy for "not a release tarball", now correctly gates on src/tcpedit/tcpedit_stub.h - the one file that still genuinely needs autogen (see scripts/autoopts/README.md). src/tcpreplay.1/tcpprep.1/tcprewrite.1/tcpbridge.1/tcpliveplay.1/ tcpcapinfo.1/tcpreplay-edit.1 are regenerated from the committed .adoc via asciidoctor, replacing the autogen/mdoc2man roff (content parity already verified in the emit_adoc.py commit). Verified end to end with both autogen and asciidoctor physically removed from the system: configure succeeds using the committed files (GNU autogen not found - using the committed tcpedit_stub.h / asciidoctor not found - using the committed pre-generated man pages), `make` reports nothing to rebuild, and the full `sudo make test` suite passes. Also verified with both tools present: a full from-scratch build and test run succeeds, generating tcpedit_stub.h via autogen (unchanged rule) and everything else via python3/asciidoctor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…man (#895 stage 5, cmake half) CMakeLists.txt: find_program(PYTHON3_EXECUTABLE) and find_program(ASCIIDOCTOR_EXECUTABLE) alongside the existing autogen detection, each erroring only if BOTH the tool is missing AND the relevant committed file is absent (mirroring AUTOGEN_EXECUTABLE's existing fallback pattern). The autogen "missing and no pre-generated file" check now correctly tests for src/tcpedit/tcpedit_stub.h - the one file that still needs it - instead of src/tcpreplay_opts.c, which python3 now produces. src/CMakeLists.txt: tcpr_autogen_opts() regenerates *_opts.c/h AND the AsciiDoc man-page source (*.adoc) via emit_h.py/emit_c.py/emit_adoc.py when stale, instead of a single autogen invocation; MANBASE lets a caller give the .adoc/man basename explicitly (tcpreplay_edit_opts -> tcpreplay-edit) since it no longer matches the opts basename 1:1. The optional `manpages` target now renders src/*.1 from the (already current) src/*.adoc via `asciidoctor -b manpage` instead of `autogen -T agman-cmd.tpl`. scripts/check-generated-opts.sh: narrowed to what autogen still produces (tcpedit_stub.h only) - *_opts.c/h and man-page equivalence are already covered by scripts/autoopts/check_emitters.py (byte- identical oracle) and check_adoc.py (content validation), and comparing the now-asciidoctor-rendered *.1 against autogen's own rendering was comparing against the wrong tool. Verified end to end with both autogen and asciidoctor removed: a fresh `cmake -B build` configures using the committed tcpedit_stub.h and man pages, `cmake --build build` succeeds with zero tool invocations, and the resulting tcpreplay binary runs correctly (--version, --more-help). With both tools present: full build, the `manpages` target renders all seven pages via asciidoctor with zero groff warnings, and all four scripts/autoopts gates plus the narrowed check-generated-opts.sh pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update CLAUDE.md, README.md, docs/INSTALL and docs/CHANGELOG to describe the new python3 (scripts/autoopts) + asciidoctor pipeline that replaces GNU autogen for *_opts.c/h and man-page generation, narrowing autogen's remaining role to src/tcpedit/tcpedit_stub.h only. CI now installs asciidoctor and runs the scripts/autoopts equivalence/content checks (validate_ir.py, check_emitters.py, check_adoc.py) alongside the existing tcpedit_stub.h drift check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…vert)
Restore this project's original convention for generated files: nothing
generated is committed, only .def files are source-controlled, and
release tarballs (make dist/dist-xz) ship the already-built output.
Phase 1 briefly deviated from this for src/*_opts.c/h specifically
because GNU autogen was becoming unobtainable (Debian Bug#1076243) - a
git checkout could not build at all without it. python3 and asciidoctor,
which scripts/autoopts (phase 2) uses instead, don't have that problem,
so there's no reason to keep deviating from the original convention for
what they produce. Committing generated output was only causing diff
noise unrelated to the actual change whenever a shared .def (e.g.
tcpedit_opts.def, included by tcprewrite/tcpbridge/tcpreplay-edit) was
edited, touching many generated files in one commit.
- git rm --cached the 28 generated files (src/*_opts.{c,h}, src/*.adoc,
src/*.1 for the 7 scripts/autoopts-covered tools) and add matching
.gitignore entries. src/tcpedit/tcpedit_stub.h and its .1 are
deliberately NOT touched - they're produced by a distinct AutoOpts
template mode still requiring real GNU autogen (never ported to
scripts/autoopts), and autogen genuinely is EOL, which is still the
right justification for keeping that one file's committed pattern.
- src/Makefile.am's existing regeneration rules, BUILT_SOURCES,
EXTRA_DIST and MAINTAINERCLEANFILES already treat "target missing" as
"stale, regenerate" - this is exactly the same logic a fresh checkout
with these files absent needs, so no rule changes were required, only
comments claiming the files were committed. Confirmed against the
pre-#895 Makefile.am (git show e55c058^:src/Makefile.am) that man
pages were always built by plain `make` via automake's default
man_MANS handling (not deferred to install/dist), matching CI's
historical `make; make dist` sequence - restored that behavior here
too, just with python3/asciidoctor instead of autogen.
- CMakeLists.txt/src/CMakeLists.txt: same staleness logic already
handled "missing" correctly; updated wording, and downgraded the
top-level asciidoctor check from FATAL_ERROR to STATUS-only, since
CMake's own manpages target design (unlike autotools) already makes
man-page rendering opt-in - a plain `cmake --build` never needs
asciidoctor, so failing configure over its absence was stricter than
the build itself requires.
- scripts/autoopts/validate_ir.py and check_emitters.py no longer diff
against a committed golden file (there isn't one any more); they now
cross-check IR parsing against emit_h.py's own fresh output and smoke-
test that all emitters run cleanly with plausible output, respectively.
check_adoc.py and scripts/check-generated-opts.sh needed no changes -
neither ever diffed against a committed *_opts.c/h/.adoc/.1 file.
- Updated CLAUDE.md, README.md, docs/INSTALL, docs/CHANGELOG, and
scripts/autoopts/README.md to describe the restored convention: python3
(and asciidoctor, for man pages) are required for any build from a git
checkout, but release tarballs need neither.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…vert configure.ac and src/tcpedit/CMakeLists.txt still described *_opts.c/h and man pages as committed after fdcd3d5 stopped committing them - missed in that commit. scripts/missing-python3 and scripts/missing-asciidoctor had the same issue in their error text, including the exact line a user reported: "src/*.1 man pages are rendered from the committed src/*.adoc". While fixing configure.ac, replacing "#895" with a bare '#' inside an AC_MSG_ERROR([...]) argument broke autoconf regeneration - M4 treats unescaped '#' as a start-of-comment marker, which silently ate the rest of the quoted string including the closing bracket, corrupting the generated configure script ("possibly undefined macro: AS_MESSAGE_LOG_FD"). Reworded to "issue 895" to avoid the bare '#'; verified with a full autogen.sh + configure sanity run afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The tests job's macOS branch never installed asciidoctor, since man pages were committed until the full revert (fdcd3d5) - now that they're ordinary build products, macOS's configure step hard-fails without it: "src/*.1 man pages are not committed to git (issue 895) and asciidoctor is not installed to generate them." This also cascaded into canceling the ubuntu-latest job via the matrix's default fail-fast, masking that job's real (passing) result. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 20, 2026
fklassen
added a commit
that referenced
this pull request
Jul 20, 2026
…ball rebuilds (#1033) (#1034) build: write generated *_opts.c/h/.adoc/.1 atomically to fix dist-tarball rebuilds. Since #1030 stopped committing src/*_opts.c/h/.adoc/.1, they're regenerated at build time by writing directly into the existing target path. All three code paths (autotools' shell redirection, CMake's execute_process OUTPUT_FILE, and scripts/autoopts/generate.py's Path.write_text) required the target file, if it already existed, to be writable - which a file extracted from a make dist tarball isn't guaranteed to be. Fixed by writing to a sibling .tmp path and renaming over the real target instead, which only requires the containing directory to be writable, not the target file itself. Fixes #1033.
5 tasks
fklassen
added a commit
that referenced
this pull request
Jul 20, 2026
) build: stop distributing src/*.adoc in make dist tarballs. Before #895, make dist tarballs shipped pre-built man pages (src/*.1) generated directly from .def via autogen - no intermediate source format was shipped or needed to rebuild from the tarball. After #1030 switched man pages to a two-stage .def -> .adoc -> .1 pipeline, both ended up in EXTRA_DIST with .1: .adoc as the dependency, which only avoided needing asciidoctor to build from a tarball by luck of tar preserving relative timestamps. Fixed by dropping *.adoc from EXTRA_DIST (it remains a git-checkout-only build artifact) and making each *.1 target depend on its .def directly, like *_opts.c/h already do, instead of on *.adoc - so a shipped, already-fresh .1 is up to date structurally, with no missing prerequisite to force a rebuild. Fixes #1035.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 of #895 (GNU autogen is EOL — Debian Bug#1076243). Phase 1 (#1028) made autogen optional by committing its generated output; this phase replaces autogen itself as the generator for that output, while leaving the vendored
libopts/runtime (--load-opts,--save-opts,--more-help, config file parsing) completely untouched — the exact thing the earlier, reverted #991 attempt broke.scripts/autoopts— a from-scratch python3 reimplementation of autogen's.def→*_opts.c/hpipeline for all 7 tools (tcpreplay,tcpreplay-edit,tcprewrite,tcpprep,tcpbridge,tcpliveplay,tcpcapinfo), verified byte-identical against real autogen output during development (scripts/autoopts/README.mddocuments that one-time proof), including autogen's xgettext paragraph-splitting logic, ported from libopts' own in-treeoptionPrintParagraphs()..defdescrip/doc/explain/detailfields that already drive--help(never hand-duplicated, so it can't drift), rendered withasciidoctor -b manpage. Content-validated viascripts/autoopts/check_adoc.py: every option covered, zerogroff -man -wwwarnings.src/tcpedit/tcpedit_stub.h— a distinct AutoOpts "library" template mode, out of scope for this phase (documented inscripts/autoopts/README.md). That file (and its.1) stays committed to git, since autogen genuinely is EOL.Design change mid-review: generated files are no longer committed
The PR originally committed
*_opts.c/h/*.adoc/*.1to git, matching phase 1's approach for*_opts.c/h. On review, that was reconsidered: phase 1 committed generated output specifically because GNU autogen was becoming unobtainable — without committing, a git checkout couldn't build at all.python3andasciidoctordon't have that problem, so there's no reason to keep deviating from this project's original convention (nothing generated was ever committed;make dist/make dist-xzshipped the already-built output) for what they produce. Committing was also causing diff noise unrelated to the actual change whenever a shared.def(e.g.tcpedit_opts.def, included bytcprewrite/tcpbridge/tcpreplay-edit) was edited, touching many generated files in one commit.The latest commit reverts that:
*_opts.c/h,*.adoc,*.1(28 files, for the 7 scripts/autoopts-covered tools) aregit rm --cachedand gitignored. They're regenerated at build time from the committed.deffiles — the existing staleness-check logic insrc/Makefile.am/src/CMakeLists.txtalready treated "file missing" as "stale, regenerate" correctly, so no rule changes were needed there beyond wording.make dist/make dist-xzstill bundle the already-built output, so release tarballs need neither python3 nor asciidoctor — only a git-checkout build does.src/tcpedit/tcpedit_stub.h(autogen-only) is untouched and stays committed, per its own still-valid justification.scripts/autoopts/validate_ir.pyandcheck_emitters.pyno longer diff against a committed golden file (there isn't one anymore); they now cross-check IR parsing againstemit_h.py's own fresh output and smoke-test that all emitters run cleanly with plausible output shape, respectively.check_adoc.pyandscripts/check-generated-opts.shneeded no changes — neither ever diffed against a committed*_opts.c/h/.adoc/.1.Test plan
scripts/autoopts/validate_ir.py,check_emitters.py,check_adoc.pyall passgit clone, none of the 28 files present): plainmakeregenerates*_opts.c/h(required to compile) and man pages (built by default, matching originalman_MANSbehavior confirmed against the pre-[Enhancement] Build-Depends on GNU autogen #895Makefile.am) with python3/asciidoctor installedmake distfrom that clean checkout produces a tarball containing all 28 generated files; extracted and built completely offline (autogen/python3/asciidoctor all removed fromPATH) — succeeds, confirming release tarballs need none of these toolsmake installfrom the offline build installs all 7 man pages correctlysudo make testpasses in full (with a correctly-configured test NIC)cmake -B buildregenerates*_opts.c/h/.adocat configure time (python3 required); a plaincmake --build builddoes not touch.1man pages at all (CMake'smanpagestarget is deliberately opt-in, unlike autotools — confirmed pre-existing, unrelated to this change);cmake --build build --target manpagesrenders them correctly with asciidoctorscripts/check-generated-opts.sh(autogen oracle for the still-committedtcpedit_stub.h) passes🤖 Generated with Claude Code