perf(lookups): pack the code tables as base 36 differences

.all-contributorsrc

@@ -221,6 +221,13 @@
 			"avatar_url": "https://github.com/JanderSilv.png",
 			"profile": "https://github.com/JanderSilv",
 			"contributions": ["code"]
+		},
+		{
+			"login": "kwy404",
+			"name": "kwy404",
+			"avatar_url": "https://github.com/kwy404.png",
+			"profile": "https://github.com/kwy404",
+			"contributions": ["code", "test"]
 		}
 	],
 	"repoType": "github",

README.md

@@ -143,7 +143,7 @@ Our "thank you" goes to these wonderful people ([emoji key](https://github.com/k
     <td align="center"><a href="https://github.com/vicentevendramin"><img src="https://github.com/vicentevendramin.png?size=100" width="100px;" alt=""/><br /><sub><b>Vicente Vendramin</b></sub></a><br /><a href="https://github.com/brazilian-utils/javascript/commits?author=vicentevendramin" title="Code">💻</a></td>
     <td align="center"><a href="https://github.com/joaopedroassad"><img src="https://github.com/joaopedroassad.png?size=100" width="100px;" alt=""/><br /><sub><b>Joao Assad</b></sub></a><br /><a href="https://github.com/brazilian-utils/javascript/commits?author=joaopedroassad" title="Code">💻</a> <a href="https://github.com/brazilian-utils/javascript/commits?author=joaopedroassad" title="Tests">⚠️</a></td>
     <td align="center"><a href="https://github.com/JanderSilv"><img src="https://github.com/JanderSilv.png?size=100" width="100px;" alt=""/><br /><sub><b>Jander Silva</b></sub></a><br /><a href="https://github.com/brazilian-utils/javascript/commits?author=JanderSilv" title="Code">💻</a></td>
-    <td></td>
+    <td align="center"><a href="https://github.com/kwy404"><img src="https://github.com/kwy404.png?size=100" width="100px;" alt=""/><br /><sub><b>kwy404</b></sub></a><br /><a href="https://github.com/brazilian-utils/javascript/commits?author=kwy404" title="Code">💻</a> <a href="https://github.com/brazilian-utils/javascript/commits?author=kwy404" title="Tests">⚠️</a></td>
     <td></td>
     <td></td>
     <td></td>

docs/getting-started.md

@@ -72,21 +72,21 @@ A few utils embed an official dataset and weigh far more than everything else co
 | `getMunicipalitiesByAreaCode` · `getAreaCodeByMunicipalityCode` | 5571 IBGE municipalities, with the DDD of each (Anatel) | 165.0 - 167.8 KB | 52.1 - 52.8 KB |
 | `getMunicipalities` · `getMunicipalityByCode` · `getMunicipality` | 5571 IBGE municipalities, with names and codes | 153.6 - 154.0 KB | 49.4 - 49.7 KB |
 | `getCities` | 5571 IBGE municipality names | 153.4 KB | 49.2 KB |
-| `getCbo` | CBO 2002 occupation titles | 117.3 KB | 29.9 KB |
-| `getCest` | CEST descriptions and segments (Convênio ICMS 142/18) | 115.5 KB | 26.1 KB |
-| `getCnae` | CNAE-Subclasses 2.3 | 91.6 KB | 20.0 KB |
-| `isValidNcm` | NCM (Nomenclatura Comum do Mercosul) codes | 82.6 KB | 22.8 KB |
-| `getNbs` | NBS 2.0 (Nomenclatura Brasileira de Serviços) descriptions | 80.6 KB | 13.1 KB |
-| `getCfop` | CFOP operation descriptions | 67.6 KB | 6.3 KB |
+| `getCest` | CEST descriptions and segments (Convênio ICMS 142/18) | 111.5 KB | 24.0 KB |
+| `getCbo` | CBO 2002 occupation titles | 107.6 KB | 24.7 KB |
+| `getCnae` | CNAE-Subclasses 2.3 | 86.6 KB | 17.9 KB |
+| `getNbs` | NBS 2.0 (Nomenclatura Brasileira de Serviços) descriptions | 75.5 KB | 11.5 KB |
+| `getCfop` | CFOP operation descriptions | 66.7 KB | 5.2 KB |
 | `getClassTrib` | cClassTrib (IBS/CBS) names and descriptions | 50.0 KB | 9.0 KB |
 | `getBanks` · `getBankByCode` · `getBankByIspb` | Banco Central STR participants (COMPE + ISPB) | 37.6 - 37.8 KB | 9.0 - 9.2 KB |
+| `isValidNcm` | NCM (Nomenclatura Comum do Mercosul) codes | 29.6 KB | 8.9 KB |
 | `getIsbnInfo` · `formatIsbn` | ISBN ranges of the International ISBN Agency (RangeMessage) | 27.0 - 27.1 KB | 6.2 KB |
 | `isValidCid10` | CID-10 V2008 category and subcategory codes plus the SIM `U07` codes, without the descriptions | 26.2 KB | 6.8 KB |
-| `getServiceItem` | Service list of the Lei Complementar 116/2003 | 26.1 KB | 8.4 KB |
-| `isValidCbo` | CBO 2002 occupation codes, without the titles | 16.2 KB | 5.5 KB |
-| `isValidCnae` | CNAE-Subclasses 2.3 codes, without the descriptions | 9.6 KB | 3.4 KB |
-| `isValidNbs` | NBS 2.0 codes, without the descriptions | 8.5 KB | 2.3 KB |
-| `isValidCest` | CEST codes, without the descriptions | 7.6 KB | 2.3 KB |
+| `getServiceItem` | Service list of the Lei Complementar 116/2003 | 26.0 KB | 8.0 KB |
+| `isValidCbo` | CBO 2002 occupation codes, without the titles | 6.6 KB | 1.7 KB |
+| `isValidCnae` | CNAE-Subclasses 2.3 codes, without the descriptions | 4.5 KB | 1.9 KB |
+| `isValidCest` | CEST codes, without the descriptions | 3.5 KB | 0.8 KB |
+| `isValidNbs` | NBS 2.0 codes, without the descriptions | 3.4 KB | 1.2 KB |
 
 The root of the package is a single ESM module, so a bundler cannot split one of these datasets out of it: importing a heavy util from the root puts its whole dataset in your main bundle, and a dynamic `import()` of the root does not help. To lazy-load one, import it from its own subpath:
 

docs/pt-br/getting-started.md

@@ -72,21 +72,21 @@ Alguns utilitários embutem uma base de dados oficial e pesam muito mais que tod
 | `getMunicipalitiesByAreaCode` · `getAreaCodeByMunicipalityCode` | 5571 municípios do IBGE, com o DDD de cada um (Anatel) | 165,0 - 167,8 KB | 52,1 - 52,8 KB |
 | `getMunicipalities` · `getMunicipalityByCode` · `getMunicipality` | 5571 municípios do IBGE, com nomes e códigos | 153,6 - 154,0 KB | 49,4 - 49,7 KB |
 | `getCities` | nomes dos 5571 municípios do IBGE | 153,4 KB | 49,2 KB |
-| `getCbo` | títulos das ocupações da CBO 2002 | 117,3 KB | 29,9 KB |
-| `getCest` | descrições e segmentos do CEST (Convênio ICMS 142/18) | 115,5 KB | 26,1 KB |
-| `getCnae` | CNAE-Subclasses 2.3 | 91,6 KB | 20,0 KB |
-| `isValidNcm` | códigos NCM (Nomenclatura Comum do Mercosul) | 82,6 KB | 22,8 KB |
-| `getNbs` | descrições da NBS 2.0 (Nomenclatura Brasileira de Serviços) | 80,6 KB | 13,1 KB |
-| `getCfop` | descrições das operações do CFOP | 67,6 KB | 6,3 KB |
+| `getCest` | descrições e segmentos do CEST (Convênio ICMS 142/18) | 111,5 KB | 24,0 KB |
+| `getCbo` | títulos das ocupações da CBO 2002 | 107,6 KB | 24,7 KB |
+| `getCnae` | CNAE-Subclasses 2.3 | 86,6 KB | 17,9 KB |
+| `getNbs` | descrições da NBS 2.0 (Nomenclatura Brasileira de Serviços) | 75,5 KB | 11,5 KB |
+| `getCfop` | descrições das operações do CFOP | 66,7 KB | 5,2 KB |
 | `getClassTrib` | nomes e descrições do cClassTrib (IBS/CBS) | 50,0 KB | 9,0 KB |
 | `getBanks` · `getBankByCode` · `getBankByIspb` | participantes do STR do Banco Central (COMPE + ISPB) | 37,6 - 37,8 KB | 9,0 - 9,2 KB |
+| `isValidNcm` | códigos NCM (Nomenclatura Comum do Mercosul) | 29,6 KB | 8,9 KB |
 | `getIsbnInfo` · `formatIsbn` | faixas do ISBN da Agência Internacional do ISBN (RangeMessage) | 27,0 - 27,1 KB | 6,2 KB |
 | `isValidCid10` | códigos das categorias e subcategorias da CID-10 V2008 mais os códigos `U07` do SIM, sem as descrições | 26,2 KB | 6,8 KB |
-| `getServiceItem` | lista de serviços da Lei Complementar 116/2003 | 26,1 KB | 8,4 KB |
-| `isValidCbo` | códigos das ocupações da CBO 2002, sem os títulos | 16,2 KB | 5,5 KB |
-| `isValidCnae` | códigos da CNAE-Subclasses 2.3, sem as descrições | 9,6 KB | 3,4 KB |
-| `isValidNbs` | códigos da NBS 2.0, sem as descrições | 8,5 KB | 2,3 KB |
-| `isValidCest` | códigos do CEST, sem as descrições | 7,6 KB | 2,3 KB |
+| `getServiceItem` | lista de serviços da Lei Complementar 116/2003 | 26,0 KB | 8,0 KB |
+| `isValidCbo` | códigos das ocupações da CBO 2002, sem os títulos | 6,6 KB | 1,7 KB |
+| `isValidCnae` | códigos da CNAE-Subclasses 2.3, sem as descrições | 4,5 KB | 1,9 KB |
+| `isValidCest` | códigos do CEST, sem as descrições | 3,5 KB | 0,8 KB |
+| `isValidNbs` | códigos da NBS 2.0, sem as descrições | 3,4 KB | 1,2 KB |
 
 A raiz do pacote é um único módulo ESM, então o bundler não consegue separar uma dessas bases de dados dele: importar um utilitário pesado da raiz coloca a base inteira no seu bundle principal, e um `import()` dinâmico da raiz não ajuda. Para carregar sob demanda, importe do subpath próprio:
 

scripts/banks.ts

@@ -4,6 +4,7 @@ import { readFile, writeFile } from "node:fs/promises";
 import { resolve } from "node:path";
 
 import { fetchWithRetry } from "../src/_internals/fetch-with-retry/fetch-with-retry.ts";
+import { serializeCodes } from "./lookup-table.ts";
 
 const scriptsDirectory = import.meta.dirname;
 
@@ -183,17 +184,19 @@ export type Bank = {
 
 export const BANKS: Bank[] = ${JSON.stringify(sorted)};`;
 
-	const compeCodes = sorted.map((bank) => bank.code).join("");
+	const compeCodes = serializeCodes(sorted.map((bank) => bank.code));
 	const constantsPath = resolve(scriptsDirectory, "..", "./src/is-valid-bank-account/constants.ts");
 	const constants = await readFile(constantsPath, "utf8");
-	const literal = (compeCodes.match(/.{1,90}/g) ?? []).map((chunk) => `\t"${chunk}"`).join(" +\n");
-	const compeCodesPattern = /export const COMPE_CODES =\n(?:\t"\d*" \+\n)*\t"\d*";/;
+	const compeCodesPattern = /export const COMPE_CODES =\n\t"[\d:,a-z\\\n]*";/;
 
 	if (!compeCodesPattern.test(constants)) {
 		throw new Error("COMPE_CODES literal not found in src/is-valid-bank-account/constants.ts");
 	}
 
-	const updated = constants.replace(compeCodesPattern, `export const COMPE_CODES =\n${literal};`);
+	const updated = constants.replace(
+		compeCodesPattern,
+		`export const COMPE_CODES =\n\t${compeCodes};`,
+	);
 
 	console.log(`Generated ${sorted.length} banks from ${source}`);
 

scripts/cbo.ts

@@ -72,9 +72,10 @@ export const renderCbo = (titles: Record<string, string>): Record<string, string
 	return {
 		"./src/_internals/constants/cbo.ts": `/**
  * CBO 2002 (Classificação Brasileira de Ocupações) occupation codes: the raw 6 digits of every
- * code, back to back in ascending order, which \`findCodeIndex\` reads. The title of a code is at
- * the same index of \`CBO_DESCRIPTIONS\`, a module of its own (\`cbo-descriptions.ts\`) so that
- * checking a code does not bundle the titles.
+ * code, in ascending order, packed as base 36 differences by \`packCodes\`
+ * (scripts/lookup-table.ts), which \`findCodeIndex\` reads. The title of a code is at the same
+ * index of \`CBO_DESCRIPTIONS\`, a module of its own (\`cbo-descriptions.ts\`) so that checking a
+ * code does not bundle the titles.
  *
  * Built from \`CBO2002 - Ocupacao.csv\` of the "Estrutura CBO (CSV)" archive the Ministério do
  * Trabalho e Emprego serves on the CBO site (files dated 10/07/2026), a plain \`CODIGO;TITULO\`

scripts/cest.ts

@@ -270,10 +270,11 @@ export const renderCest = (
 
 	return {
 		"./src/_internals/constants/cest.ts": `/**
- * CEST (Código Especificador da Substituição Tributária) codes: the 7 digits of every code, back
- * to back in ascending order, which \`findCodeIndex\` reads. The description of a code is at the
- * same index of \`CEST_DESCRIPTIONS\`, a module of its own (\`cest-descriptions.ts\`, with the
- * segment names) so that checking a code does not bundle the descriptions.
+ * CEST (Código Especificador da Substituição Tributária) codes: the 7 digits of every code, in
+ * ascending order, packed as base 36 differences by \`packCodes\` (scripts/lookup-table.ts), which
+ * \`findCodeIndex\` reads. The description of a code is at the same index of \`CEST_DESCRIPTIONS\`,
+ * a module of its own (\`cest-descriptions.ts\`, with the segment names) so that checking a code
+ * does not bundle the descriptions.
  *
  * Built from Anexos II to XXVI of Convênio ICMS 142/18, the consolidated text in force, which
  * CONFAZ heads with this line:

scripts/cfop.ts

@@ -89,10 +89,11 @@ export const renderCfop = (table: Record<string, string>): Record<string, string
 
 	return {
 		"./src/_internals/constants/cfop.ts": `/**
- * CFOP (Código Fiscal de Operações e Prestações) codes: the 4 digits of every operable code, back
- * to back in ascending order, which \`findCodeIndex\` reads. The official description of a code is
- * at the same index of \`CFOP_DESCRIPTIONS\`, a module of its own (\`cfop-descriptions.ts\`) so
- * that checking a code does not bundle the descriptions.
+ * CFOP (Código Fiscal de Operações e Prestações) codes: the 4 digits of every operable code, in
+ * ascending order, packed as base 36 differences by \`packCodes\` (scripts/lookup-table.ts), which
+ * \`findCodeIndex\` reads. The official description of a code is at the same index of
+ * \`CFOP_DESCRIPTIONS\`, a module of its own (\`cfop-descriptions.ts\`) so that checking a code
+ * does not bundle the descriptions.
  *
  * Built from the consolidated Anexo II of Convênio SINIEF s/nº of 15 December 1970, the text in
  * force (given its current wording by Ajuste SINIEF 03/24 and last amended by Ajuste SINIEF

scripts/cnae.ts

@@ -31,10 +31,11 @@ export const renderCnae = (subclasses: Record<string, string>): Record<string, s
 
 	return {
 		"./src/_internals/constants/cnae.ts": `/**
- * CNAE-Subclasses 2.3 (Classificação Nacional de Atividades Econômicas) subclass codes: the raw
- * 7 digits of every subclass, back to back in ascending order, which \`findCodeIndex\` reads. The
- * official description of a subclass is at the same index of \`CNAE_DESCRIPTIONS\`, a module of
- * its own (\`cnae-descriptions.ts\`) so that checking a code does not bundle the descriptions.
+ * CNAE-Subclasses 2.3 (Classificação Nacional de Atividades Econômicas) subclass codes: the raw 7
+ * digits of every subclass, in ascending order, packed as base 36 differences by \`packCodes\`
+ * (scripts/lookup-table.ts), which \`findCodeIndex\` reads. The official description of a subclass
+ * is at the same index of \`CNAE_DESCRIPTIONS\`, a module of its own (\`cnae-descriptions.ts\`) so
+ * that checking a code does not bundle the descriptions.
  *
  * 2.3 is the current subclass revision of CNAE 2.0: CONCLA's own CNAE browser lists it as
  * "CNAE-Subclasses 2.3" under "CNAE 2.0 (Res 02/2010)" and tells anyone opening an older table

scripts/legal-natures.ts

@@ -254,9 +254,10 @@ ${stringifyEntries(legacyDescriptions)}
 };
 
 /**
- * The four digit code of every entry of \`LEGAL_NATURE\`, the official and the legacy ones, back to
- * back in ascending order, which \`findCodeIndex\` reads: \`isValidLegalNature\` checks a code
- * against it without bundling the descriptions.
+ * The four digit code of every entry of \`LEGAL_NATURE\`, the official and the legacy ones, in
+ * ascending order, packed as base 36 differences by \`packCodes\` (scripts/lookup-table.ts), which
+ * \`findCodeIndex\` reads: \`isValidLegalNature\` checks a code against it without bundling the
+ * descriptions.
  */
 export const LEGAL_NATURE_CODES = ${serializeCodes(Object.keys({ ...current, ...legacyDescriptions }).sort())};
 

scripts/lookup-table.ts

@@ -24,22 +24,49 @@ export const splitRecord = <T>(data: Record<string, T>): LookupTable<T> => {
 };
 
 /**
- * Writes codes of one fixed width as the source of a single string literal that holds them back
- * to back, which `findCodeIndex` reads. Each code sits on a line of its own, ended by a line
- * continuation that adds nothing to the value, so a refresh diffs to the codes that changed.
- * @param {string[]} codes - The codes, in the order they are looked up in.
- * @returns {string} The string literal.
+ * Packs ascending numeric codes of one fixed width into the text `findCodeIndex` reads: the width,
+ * a colon, then each code as its difference from the one before it (the first from zero), in base
+ * 36 and separated by commas. The differences of a dense table are one or two characters where the
+ * codes are six to nine digits, so the table ships about a third of the bytes it would as the codes
+ * written back to back.
+ * @param {string[]} codes - The codes, in ascending order, every one the same number of digits.
+ * @returns {string} The packed table, e.g. `"3:1,2,1"` for `["001", "003", "004"]`.
  */
-export const serializeCodes = (codes: readonly string[]): string => {
+export const packCodes = (codes: readonly string[]): string => {
 	const [first = ""] = codes;
 
-	if (first === "" || codes.some((code) => code.length !== first.length || !/^\w+$/.test(code))) {
-		throw new Error("lookup table codes are not all word characters of one width");
+	if (first === "" || codes.some((code) => code.length !== first.length || !/^\d+$/.test(code))) {
+		throw new Error("lookup table codes are not all digits of one width");
 	}
 
-	return `"${codes.join("\\\n")}"`;
+	let previous = 0;
+	const differences = codes.map((code, index) => {
+		const value = Number(code);
+
+		if (index > 0 && value <= previous) {
+			throw new Error(`lookup table code ${code} is out of order or repeated`);
+		}
+
+		const difference = (value - previous).toString(36);
+		previous = value;
+
+		return difference;
+	});
+
+	return `${first.length}:${differences.join(",")}`;
 };
 
+/**
+ * Writes codes of one fixed width as the source of a single string literal that holds them
+ * packed by `packCodes`, which `findCodeIndex` reads. Each difference sits on a line of its own,
+ * ended by a line continuation that adds nothing to the value, so a refresh diffs to the codes
+ * that changed (and the one after each).
+ * @param {string[]} codes - The codes, in ascending order.
+ * @returns {string} The string literal.
+ */
+export const serializeCodes = (codes: readonly string[]): string =>
+	`"${packCodes(codes).replaceAll(",", ",\\\n")}"`;
+
 /**
  * Writes every generated file of a dataset, relative to the repository root.
  * @param {Record<string, string>} files - The content of each file, by its path.

scripts/nbs.ts

@@ -107,10 +107,10 @@ export const renderNbs = (table: Record<string, string>): Record<string, string>
 	return {
 		"./src/_internals/constants/nbs.ts": `/**
  * NBS 2.0 (Nomenclatura Brasileira de Serviços, Intangíveis e Outras Operações que Produzam
- * Variações no Patrimônio) codes: the raw 9 digits of every complete code, back to back in
- * ascending order, which \`findCodeIndex\` reads. The official description of a code is at the
- * same index of \`NBS_DESCRIPTIONS\`, a module of its own (\`nbs-descriptions.ts\`) so that
- * checking a code does not bundle the descriptions.
+ * Variações no Patrimônio) codes: the raw 9 digits of every complete code, in ascending order,
+ * packed as base 36 differences by \`packCodes\` (scripts/lookup-table.ts), which \`findCodeIndex\`
+ * reads. The official description of a code is at the same index of \`NBS_DESCRIPTIONS\`, a module
+ * of its own (\`nbs-descriptions.ts\`) so that checking a code does not bundle the descriptions.
  *
  * Built from the official \`NBSa_2-0.csv\` the MDIC publishes on gov.br, a plain
  * \`NBS 2.0;DESCRIÇÃO\` CSV in ISO-8859-1, the version 2.0 as amended by the Portaria Conjunta

scripts/ncm.ts

@@ -82,11 +82,12 @@ const isInForce = (entry: NcmEntry, today: Date): boolean =>
  */
 export const renderNcm = (codes: readonly string[]): Record<string, string> => ({
 	"./src/is-valid-ncm/constants.ts": `/**
- * NCM (Nomenclatura Comum do Mercosul) 8 digit codes in force on the generation date, back to
- * back in ascending order (one per source line, joined by line continuations). A code is included
- * when the generation date falls within its Siscomex \`Data_Inicio\`/\`Data_Fim\` range (both
- * inclusive) — this also keeps codes that are valid today but carry a scheduled future end date,
- * not only the ones with no end date (\`Data_Fim: "31/12/9999"\`).
+ * NCM (Nomenclatura Comum do Mercosul) 8 digit codes in force on the generation date, in ascending
+ * order, packed as base 36 differences by \`packCodes\` (scripts/lookup-table.ts) (one per source
+ * line, joined by line continuations). A code is included when the generation date falls within its
+ * Siscomex \`Data_Inicio\`/\`Data_Fim\` range (both inclusive) — this also keeps codes that are
+ * valid today but carry a scheduled future end date, not only the ones with no end date
+ * (\`Data_Fim: "31/12/9999"\`).
  *
  * Generated by \`node ./scripts/ncm.ts\`. Do not edit by hand.
  *