levels.rpf is 76,155,065,056 bytes. Its table of contents is encrypted, so the archive cannot be
enumerated by name. tools/world_census.py measures the data area directly: it inflates every zlib
stream, validates each 0DRP / version 0x7a resource, joins continuation streams, and counts the
map resources and placement entities they contain.
| Measure | Value |
|---|---|
| Data area scanned | 76,154,954,352 bytes (start 0x1B070) |
Map resources (kind=map) |
23,535 |
| Placement entities | 1,670,442 |
| Complete / validated map rows | 23,535 / 23,535 |
| Exact entity counts | 1,670,442 / 1,670,442 |
| Shards complete | 36 / 36 |
| Zlib streams inflated | 1,117,857 |
| Stored stream bytes | 74,877,281,159 |
| Inflated bytes (all streams) | 174,941,755,029 |
| Rejected zlib candidates | 840 |
| Metadata streams | 164,914 |
| Duplicate content groups | 0 |
| Scan time | 290.8 s |
A map row is a resource whose schema includes CMapData. maps_exact means every entity block
counted against the declared entity_size and no entity block was left in a graphics page.
entities_exact is the same check summed over all rows. Six entities of the 23,535 maps were split
across a continuation. The stream and byte totals don't depend on the shard size: each shard scans an
8 MiB warm-up window before its start only to find a stream boundary, and the streams there are counted by
the shard they belong to. The stored total therefore equals the accepted stream payload in Coverage below.
The 23,535 rows above are placement maps: a CMapData root and at least one countable
CEntityDef block. The same pass also validates 138,839 CMapData-only metadata resources with
no CEntityDef block, plus 26,075 other 0x7a metadata resources. Counting every CMapData
root regardless of entity schema gives 162,374 resources, but only the 23,535 with entities are
placement maps in this census. Keeping the two classes separate avoids inflating the map count with
map-definition metadata.
| Detail | Count |
|---|---|
Placement maps (CMapData + CEntityDef) |
23,535 |
CMapData-only metadata (no CEntityDef) |
138,839 |
Other 0x7a metadata |
26,075 |
| Placement maps joining a continuation | 2,868 |
| Placement maps using a raw continuation page | 3 |
| Metadata resources using a raw continuation page | 1 |
| Byte-identical duplicate map payloads | 0 |
| Duplicate 32-bit map-name hashes | 0 |
| Entity blocks in graphics pages | 0 |
| Unresolved or truncated map resources | 0 |
| Byte class | Bytes | Share of data area |
|---|---|---|
| Accepted stream payload | 74,877,281,159 | 98.32% |
Raw graphics pages (0x6) |
2,762,232 | 0.004% |
| Raw system continuation | 44,304 | 0.0001% |
| Zero padding | 1,058,806,784 | 1.39% |
| Unread non-zero bytes | 216,059,873 | 0.28% |
| Accounted | 76,154,954,352 | 100% |
The unread bytes are mostly small alignment and metadata gaps between streams. The gap audit found
23 0DRP signatures in unread ranges, but none resolved to a map or metadata resource
(raw_maps = 0, raw_meta = 0), so the census is not missing a map class. No unaligned zlib stream
that inflates to a 0DRP header was found in a gap (missed_zlib_meta = 0). The 906 rejected
candidates are zlib streams that do not form a complete 0x7a resource; they are ordinary archive
payload, not counted maps.
The same walk was run over the ten other non-radio RPF9 archives in the package, in 1 GiB shards
(32 shards total) with 12 jobs; --merge-only then rebuilt each summary from its shard files. All
shards completed and every byte of the scanned data areas is accounted for. A CMapData-only
metadata row has a CMapData root but no CEntityDef table; the Other 0x7a meta column is every
other non-map 0x7a resource.
| Archive | Size (bytes) | Data span (bytes) | Maps | Entities | CMapData-only meta |
Other 0x7a meta rows |
Shards | Stream coverage | Accounted |
|---|---|---|---|---|---|---|---|---|---|
data.rpf |
481,544,304 | 481,474,576 | 0 | 0 | 0 | 15,768 | 1/1 | 87.9% | 100% |
packs.rpf |
4,798,213,264 | 4,798,195,520 | 0 | 0 | 0 | 4,678 | 5/5 | 95.5% | 100% |
common.rpf |
6,642,192 | 6,603,920 | 0 | 0 | 0 | 0 | 1/1 | 85.5% | 100% |
textures.rpf |
192,585,024 | 192,583,088 | 0 | 0 | 0 | 0 | 1/1 | 97.8% | 100% |
anim.rpf |
4,407,626,384 | 4,407,608,368 | 0 | 0 | 0 | 22,823 | 5/5 | 95.4% | 100% |
navmeshes.rpf |
6,707,512,496 | 6,707,510,576 | 0 | 0 | 0 | 9,603 | 7/7 | 97.5% | 100% |
shaders.rpf |
411,695,600 | 411,661,232 | 0 | 0 | 0 | 0 | 1/1 | 33.4% | 100% |
movies.rpf |
8,832,481,776 | 8,832,454,448 | 0 | 0 | 0 | 0 | 9/9 | 0.0% | 100% |
script.rpf |
4,358,960 | 4,358,352 | 0 | 0 | 0 | 0 | 1/1 | 59.2% | 100% |
appdata0.rpf |
8,880 | 8,016 | 0 | 0 | 0 | 0 | 1/1 | 0.0% | 100% |
| Total | 25,842,668,880 | 25,842,458,096 | 0 | 0 | 0 | 52,872 | 32/32 | 62.2% | 100% |
Outside levels.rpf, no 0x7a resource has a CMapData root: 0 placement maps / 0
entities. The 52,872 metadata rows describe other 0x7a resources. tools/world_index.py
independently parsed the ten archives and also returned 0 maps / 0 entities. audio.rpf (radio) was
not scanned.
movies.rpf and appdata0.rpf have 0% stream coverage because their data areas contain no complete
zlib streams; movies.rpf holds Bink video (KB2 at file offset 0x7000) and appdata0.rpf is an
8,016-byte data area. Their negative result rests on the whole-gap audit, which looked for a raw
0DRP + 0x7a header and for an unaligned zlib stream hiding one. shaders.rpf (33.4%) and the
unread remainders of data, packs, anim and navmeshes are non-zlib payload. Across the ten
archives the gap audit found 63 0DRP + 0x7a signatures in unread ranges, but none resolved to
a map or metadata resource (gap_0drp_other = 63), and no unaligned zlib stream inflated to a
0DRP header (missed_zlib_meta = 0). All ten are valid 9FPR archives with complete shards.
movies.rpf and appdata0.rpf are not supported by the zlib stream walk, so their negative result
is a signature audit of the data area rather than a full stream census; neither archive is malformed.
The other eight have stream censuses with 33–98% accepted payload and audited gaps.
Adding these archives does not change the measured total: 23,535 maps / 1,670,442 entities, all
in levels.rpf. SixNG publishes 30,785 map files / 2,249,490 entities, leaving 7,250 maps / 579,048
entities. The difference is not explained by maps outside levels.rpf (zero found here) or by
duplicates (the levels census reports zero byte-identical map payloads and zero duplicate map-name
hashes). The census counts top-level 0x7a resources and does not enumerate nested payloads, so it
cannot rule out a nested-resource counting rule. levels.rpf also contains 138,839 CMapData-only
metadata resources and 26,075 other 0x7a metadata resources. The CMapData-only class has no
CEntityDef table and cannot account for the 579,048 entities; the other class has no CMapData root
and is not a placement map. Nothing shows SixNG counts either class, so the remaining difference is
unexplained.
summary.json sets census_exact only when all of these hold:
completeis true and every expected shard was written;maps_exact == mapsandentities_exact == entities;entity_gfx == 0andentity_unresolved == 0;incomplete == 0andinvalid == 0;raw_maps == 0andmissed_zlib_meta == 0.
The run above satisfies every condition. accounted_ratio is 1.0 because every byte is assigned to
accepted payload, a raw page, zero padding or counted unread data.
The scan starts at the data-area offset from the 9FPR header. It walks 16-byte-aligned zlib
candidates in independent shards and:
- inflates each stream, keeping the first bytes needed to recognize
0DRPz; - validates the little-endian
0x7aheader, struct table, block table and member offsets; - follows a contiguous split resource, either another zlib page or a raw system page whose length the resource's pointers require;
- counts
CEntityDefblocks against the declared entity size, and records0x6graphics extents; - merges shard rows, checks duplicate names and content hashes, and audits every gap for a missed
0DRPsignature or unaligned zlib stream.
Each shard is written to shard_<index>_<start>_<end>.json; summary.json and maps.json are
rebuilt from the shards. Keep the input stable during a run and use one census per output directory.
Each shard records a local archive fingerprint based on file size, timestamps, filesystem identifiers
and up to three 4 KiB samples. If filesystem identifiers are unavailable, the resolved path also
contributes to the hash. Verifying every byte would require reading the whole archive. Copies or metadata changes can require a rescan. --resume rescans missing,
older-format and mismatched shards. --merge-only reports their ranges as missing and exits 1.
Use --resume to refresh format 4 shards in place, since their stream totals can include warm-up streams twice.
python -I tools\world_census.py <levels.rpf> --out work\world\census --shard-size 2147483648 --jobs 12
python -I tools\world_census.py <levels.rpf> --out work\world\census --merge-only
python -I tests\test_world_census.pyThe default shard size is 2 GiB, the default job count is half the logical CPUs, and the scan is
bounded by --start / --limit when a partial pass is wanted. --jobs 1 runs the shards in-process
instead of a process pool, which is useful for small synthetic archives and tests.
- The census counts resources, not names. It does not decrypt the RPF9 TOC, so it cannot say which
archive entry owns a map;
docs/world.mdcorrelates the measured bands with the placement files it could carve. - Only
0x7aresources are validated. Other resource types contribute to stream and byte counts but are not classified. - A map row proves the resource has a
CMapDataroot and a self-consistent entity table. It does not prove the map is reachable, loaded, or complete in the runtime. - Output contains resource offsets, sizes, counts and content hashes only. It is aggregate metadata; keep it out of git along with the archive and carved streams.