Skip to content

Add Markdown documentation converted from XML sources - #87

Open
saivedant169 wants to merge 4 commits into
cscout-project:masterfrom
saivedant169:docs/convert-xml-to-markdown
Open

Add Markdown documentation converted from XML sources#87
saivedant169 wants to merge 4 commits into
cscout-project:masterfrom
saivedant169:docs/convert-xml-to-markdown

Conversation

@saivedant169

@saivedant169 saivedant169 commented Mar 16, 2026

Copy link
Copy Markdown

Relates to #8

What this does

Converts CScout's 51-chapter XML documentation to GitHub-Flavored Markdown, producing 41 Markdown files (8,065 lines) in docs/markdown/. Also includes the converter script (doc/xml2md.py) so the conversion can be re-run as the XML sources evolve.

Why a custom converter

The CScout docs use a custom SGML-style XML format (not DocBook or standard XML), with unclosed HTML tags like <p>, <li>, and custom elements like <fmtcode ext="c">. Pandoc and standard XML parsers can't handle this directly. The converter uses Python's html.parser which handles unclosed SGML tags gracefully.

What the converter handles

XML Element Markdown Output
<fmtcode ext="c"> Fenced code block with language (```c)
<em>, <strong>, <code> *italic*, **bold**, `code`
<h2>, <h3>, <h4> ##, ###, #### headings
<ul>/<ol>/<li> (unclosed) Markdown lists with nesting
<table>/<tr>/<td>/<th> GFM pipe tables
<dl>/<dt>/<dd> Bold term + colon definition
<a href="...html"> Links with .html → .md rewriting
<pre> Plain fenced code blocks
HTML entities Unicode equivalents

What's included

  • doc/xml2md.py — The converter script (338 lines, no external dependencies)
  • docs/markdown/README.md — Generated table of contents with links
  • docs/markdown/01-intro.md through 51-change.md — All 41 converted chapters

What's excluded

10 XML files are auto-generated at build time (man pages from .1 files, dbschema.xml from CScout binary output, qeg.xml from SQL scripts). These are skipped since they don't exist in the source tree and would need to be regenerated after a build.

Usage

python3 doc/xml2md.py                      # Default output: docs/markdown/
python3 doc/xml2md.py --outdir path/to/dir  # Custom output directory

dspinellis
dspinellis requested changes Mar 17, 2026
View reviewed changes

@dspinellis dspinellis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! The original XML files were converted into the online HTML documentation with a custom tool. As a minimum this change needs a process to create the HTML documentation (in chapters and in whole).

@saivedant169
saivedant169 requested a review from dspinellis March 17, 2026 17:28
@saivedant169

saivedant169 commented Mar 17, 2026

Copy link
Copy Markdown
Author

Added a mkdocs config (docs/mkdocs.yml) that builds the Markdown into HTML with chapter navigation and search. Uses the readthedocs theme.

The full pipeline now:

python3 doc/xml2md.py        # convert XML to Markdown + copy images
cd docs && mkdocs build      # build HTML to doc/html/

Produces 42 HTML pages with a sidebar, prev/next links between chapters, and a search bar. All 19 images are included. Build takes under a second.

Only dependency is mkdocs (pip install mkdocs). The HTML output directory is gitignored.

dspinellis
dspinellis requested changes Apr 2, 2026
View reviewed changes

@dspinellis dspinellis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the change! Please review carefully the generated HTML against the original one. I tested the conversion and the HTML generation and see the following issues. More may exist.

  1. The generation process emits tens of warnings like the ones below and hyperlinks are no longer working.
WARNING -  Doc file 'README.md' contains a link '34-dbschema.md', but the target
           is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '35-qeg.md', but the target is
           not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '42-mancscc.md', but the target
           is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '43-mancscout.md', but the
           target is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '44-mancscut.md', but the target
           is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '45-mancsmake.md', but the
           target is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '46-mancsreconst.md', but the
           target is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '47-mancssplit.md', but the
           target is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '48-mancsmerge.md', but the
           target is not found among documentation files.
WARNING -  Doc file 'README.md' contains a link '49-mancswc.md', but the target
           is not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
WARNING -  Doc file '02-walk.md' contains a link 'simul.md', but the target is
           not found among documentation files.
  1. Diagrams have gone missing (See , e.g. 21-cg)
  2. Some tables are mis-formated, see e.g. the example below.
image 4. The manual pages are missing (Consider adjusting the process of creating them to generate directly Markdown, rather than going through HTML).

@saivedant169
saivedant169 requested a review from dspinellis April 2, 2026 09:33
@saivedant169

saivedant169 commented Apr 2, 2026

Copy link
Copy Markdown
Author

@dspinellis pushed fixes for the issues you pointed out:

Fixed 581 broken simul.md links across 12 files. Those were pointing to the interactive web demo, which doesn't exist in static docs, so I replaced them with plain text. Also got the broken links in 04-flow.md and the short.md reference in the FAQ.

Converted all 8 man pages from the troff sources in man/*.1 to markdown (files 42-49). They're in the mkdocs nav now under a Man Pages section.

Added 34-dbschema.md and 35-qeg.md. Since the content for these is auto-generated from CScout's SQL dump, I wrote them based on what's described in the SQL Back-end chapter.

Table formatting I haven't touched yet. The single-column tables are coming from how the XML converter handles the original HTML table elements. Fixing that properly means reworking the converter's table parsing. Want me to take that on, or is this OK for now?

@dspinellis

dspinellis commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator

That was fast!

@dspinellis pushed fixes for the issues you pointed out:

Fixed 581 broken simul.md links across 12 files. Those were pointing to the interactive web demo, which doesn't exist in static docs, so I replaced them with plain text. Also got the broken links in 04-flow.md and the short.md reference in the FAQ.

For simul use the existing simul.html page.

Converted all 8 man pages from the troff sources in man/*.1 to markdown (files 42-49). They're in the mkdocs nav now under a Man Pages section.
Added 34-dbschema.md and 35-qeg.md. Since the content for these is auto-generated from CScout's SQL dump, I wrote them based on what's described in the SQL Back-end chapter.

Both of these conversions should be done dynamically, to always keep the files in sync.

Table formatting I haven't touched yet. The single-column tables are coming from how the XML converter handles the original HTML table elements. Fixing that properly means reworking the converter's table parsing. Want me to take that on, or is this OK for now?

This needs fixing. You can manually tweak the Markdown files, as these will be maintained as the source of truth. (Your commit should remove the XML files).

It would also be nice to have a single printable technical report with the complete documentation in a single page, as is currently the case. This could be Markdown or PDF, whatever is easier for you.

@saivedant169
Add Markdown documentation converted from XML sources
7c87252 
Add doc/xml2md.py converter script and docs/markdown/ output directory
containing 41 chapters of CScout documentation converted from the
custom SGML-style XML format to GitHub-Flavored Markdown.

The converter handles all custom tags including <fmtcode> code blocks
with syntax highlighting, unclosed SGML tags (<p>, <li>), HTML entity
replacement, cross-reference link rewriting (.html -> .md), and GFM
table generation.

10 auto-generated XML files (man pages, dbschema, qeg) are skipped
since they are built from binaries/scripts at build time.

The converter can be re-run as XML sources are updated:
    python3 doc/xml2md.py [--outdir docs/markdown]

Relates to cscout-project#8
@saivedant169
add mkdocs HTML build pipeline
1fd039b 
Added docs/mkdocs.yml with readthedocs theme and chapter navigation.
Updated xml2md.py to copy images into the markdown directory.
Build output goes to doc/html/ (gitignored).

Build steps:
  python3 doc/xml2md.py
  cd docs && mkdocs build

Produces 42 HTML pages with sidebar navigation, search,
and all images included. Tested end to end.
@saivedant169
Fix broken links, add man pages and missing chapters
313488f 
Address review feedback:

1. Fix 581 broken simul.md links across 12 files by replacing
   dead hyperlinks with plain text (these pointed to the
   interactive CScout web demo which doesn't exist in static docs)

2. Fix broken links in 04-flow.md (wsp.md, csmake.md, pragma.md)
   and 41-faq.md (short.md) to use numbered filenames

3. Convert all 8 man pages from troff (man/*.1) to markdown
   (files 42-49) and add them to mkdocs.yml nav

4. Add 34-dbschema.md and 35-qeg.md for the database schema
   and SQL query examples chapters

Build now runs with zero warnings.

Signed-off-by: saivedant169 <saivedant169@gmail.com>
@saivedant169
Address review: fix links, tables, add man pages, remove XML
a92085e 
- Point simul.md links to the existing simul.html (581 links)
- Fix broken links in 04-flow.md and 41-faq.md
- Convert man pages from troff dynamically via xml2md.py
- Generate dbschema and qeg chapters dynamically via xml2md.py
- Fix 25 mis-formatted single-column tables (converted to code blocks)
- Add single printable page (docs/cscout-manual.md, 9370 lines)
- Remove 42 XML source files (markdown is now source of truth)
- Copy simul.html and image assets into docs/markdown
- Build runs with zero warnings

Signed-off-by: saivedant169 <saivedant169@gmail.com>
@saivedant169
saivedant169 force-pushed the docs/convert-xml-to-markdown branch from 9a4de11 to a92085e Compare June 20, 2026 18:05
@saivedant169

saivedant169 commented Jun 20, 2026

Copy link
Copy Markdown
Author

@dspinellis rebased this on master. The only conflict was ext.xml, and since the XML is being removed I carried your recent C23 additions there (typeof, typeof_unqual, constexpr, nullptr) into 28-ext.md so they are not lost. mkdocs build now runs with zero warnings.

Recap against your last review:

  • simul links now point to the existing simul.html
  • man pages are converted from man/*.1 dynamically in xml2md.py
  • single printable page added at docs/cscout-manual.md
  • the mis-formatted single-column tables are fixed
  • XML source files removed, markdown is now the source of truth

One honest caveat on dbschema and qeg: xml2md.py generates those chapters, but the content is curated text rather than parsed from a live schema, since the real SQL schema only exists after running cscout -s on a project. If you would prefer them generated from actual cscout -s output so they stay exactly in sync, I am happy to wire that in, just let me know which SQL dialect to use for the sample.

Ready for another look when you have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dspinellis dspinellis Awaiting requested review from dspinellis

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@saivedant169 @dspinellis