Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
scripts/data/*.csv -text
8 changes: 6 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,12 @@ itself:
- **Generated from an official source** by a script in `scripts/` (`npm run build:data`, run
every Monday by the `Update datasets` workflow): banks (Banco Central, `banks.ts`), CBO
(`cbo.ts`), CFOP (CONFAZ, `cfop.ts`), municipalities and states (IBGE, `cities.ts`,
`states.ts`), CNAE (`cnae.ts`), legal natures (CONCLA, `legal-natures.ts`) and NCM (Siscomex,
`ncm.ts`). When a run changes a file, the workflow opens a pull request whose description, written
`states.ts`), the DDD of every municipality (Anatel, `area-codes.ts`), CNAE (`cnae.ts`), legal
natures (CONCLA, `legal-natures.ts`) and NCM (Siscomex, `ncm.ts`). The CBO is the one table
read from a file kept in the repository, `scripts/data/cbo2002-ocupacao.csv`: the MTE serves
its current release only after a reCAPTCHA, so a new release is downloaded by hand from
https://cbo.mte.gov.br/cbosite/pages/downloads.jsf ("Estrutura CBO (CSV)"), reviewed, copied
over that file and its digest put in `CBO_CSV_SHA256` of `scripts/cbo.ts`. When a run changes a file, the workflow opens a pull request whose description, written
by `scripts/data-summary.ts`, lists per table how many entries were added and removed, with a
sample of each. Never edit these files by hand.
- **Maintained by hand**, because the source is a law or a regulation with no machine-readable
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ A few utils embed an official dataset and weigh far more than everything else co
| `getMunicipalitiesByAreaCode` · `getAreaCodeByMunicipalityCode` | 5571 IBGE municipalities, with the DDD of each (Anatel) | 165.0 - 167.8 KB | 52.1 - 52.8 KB |
| `getMunicipalities` · `getMunicipalityByCode` · `getMunicipality` | 5571 IBGE municipalities, with names and codes | 153.6 - 154.0 KB | 49.4 - 49.7 KB |
| `getCities` | 5571 IBGE municipality names | 153.4 KB | 49.2 KB |
| `getCbo` | CBO 2002 occupation titles | 115.7 KB | 29.5 KB |
| `getCbo` | CBO 2002 occupation titles | 117.3 KB | 29.9 KB |
| `getCest` | CEST descriptions and segments (Convênio ICMS 142/18) | 115.5 KB | 26.1 KB |
| `getCnae` | CNAE-Subclasses 2.3 | 91.6 KB | 20.0 KB |
| `isValidNcm` | NCM (Nomenclatura Comum do Mercosul) codes | 82.6 KB | 22.8 KB |
Expand Down
2 changes: 1 addition & 1 deletion docs/pt-br/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Alguns utilitários embutem uma base de dados oficial e pesam muito mais que tod
| `getMunicipalitiesByAreaCode` · `getAreaCodeByMunicipalityCode` | 5571 municípios do IBGE, com o DDD de cada um (Anatel) | 165,0 - 167,8 KB | 52,1 - 52,8 KB |
| `getMunicipalities` · `getMunicipalityByCode` · `getMunicipality` | 5571 municípios do IBGE, com nomes e códigos | 153,6 - 154,0 KB | 49,4 - 49,7 KB |
| `getCities` | nomes dos 5571 municípios do IBGE | 153,4 KB | 49,2 KB |
| `getCbo` | títulos das ocupações da CBO 2002 | 115,7 KB | 29,5 KB |
| `getCbo` | títulos das ocupações da CBO 2002 | 117,3 KB | 29,9 KB |
| `getCest` | descrições e segmentos do CEST (Convênio ICMS 142/18) | 115,5 KB | 26,1 KB |
| `getCnae` | CNAE-Subclasses 2.3 | 91,6 KB | 20,0 KB |
| `isValidNcm` | códigos NCM (Nomenclatura Comum do Mercosul) | 82,6 KB | 22,8 KB |
Expand Down
4 changes: 2 additions & 2 deletions docs/pt-br/utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -2772,7 +2772,7 @@ isValidCbo('2124abc05'); // false (não é uma forma documentada)
isValidCbo(-212405); // false (não é um inteiro seguro não negativo)
```

Fonte: [tabela de ocupações da CBO 2002 publicada pelo MTE](https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv).
Fonte: [tabelas da CBO 2002 publicadas pelo MTE ("Estrutura CBO (CSV)", arquivos de 10/07/2026, 2.725 ocupações)](https://cbo.mte.gov.br/cbosite/pages/downloads.jsf). Até a 2.4.0 a tabela vinha da versão mais antiga do gov.br (06/06/2025): faltavam 37 ocupações, e 6 que o MTE retirou depois (225142, 322105, 322115, 322120, 322125 e 782820) deixam de ser válidas.

### parseCbo

Expand Down Expand Up @@ -2802,7 +2802,7 @@ getCbo('000000'); // null
getCbo('2124abc05'); // null (não é uma forma documentada)
```

Fonte: [tabela de ocupações da CBO 2002 publicada pelo MTE](https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv).
Fonte: [tabelas da CBO 2002 publicadas pelo MTE ("Estrutura CBO (CSV)", arquivos de 10/07/2026, 2.725 ocupações)](https://cbo.mte.gov.br/cbosite/pages/downloads.jsf). Até a 2.4.0 a tabela vinha da versão mais antiga do gov.br (06/06/2025): faltavam 37 ocupações, e 6 que o MTE retirou depois (225142, 322105, 322115, 322120, 322125 e 782820) deixam de ser válidas.

### isValidCnae

Expand Down
4 changes: 2 additions & 2 deletions docs/utilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -2772,7 +2772,7 @@ isValidCbo('2124abc05'); // false (not a documented form)
isValidCbo(-212405); // false (not a non-negative safe integer)
```

Source: [CBO 2002 occupation table published by the MTE](https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv).
Source: [CBO 2002 tables published by the MTE ("Estrutura CBO (CSV)", files of 10/07/2026, 2,725 occupations)](https://cbo.mte.gov.br/cbosite/pages/downloads.jsf). Up to 2.4.0 the table came from the older gov.br release (06/06/2025): 37 occupations were missing, and 6 the MTE has since dropped (225142, 322105, 322115, 322120, 322125 and 782820) are no longer valid.

### parseCbo

Expand Down Expand Up @@ -2802,7 +2802,7 @@ getCbo('000000'); // null
getCbo('2124abc05'); // null (not a documented form)
```

Source: [CBO 2002 occupation table published by the MTE](https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv).
Source: [CBO 2002 tables published by the MTE ("Estrutura CBO (CSV)", files of 10/07/2026, 2,725 occupations)](https://cbo.mte.gov.br/cbosite/pages/downloads.jsf). Up to 2.4.0 the table came from the older gov.br release (06/06/2025): 37 occupations were missing, and 6 the MTE has since dropped (225142, 322105, 322115, 322120, 322125 and 782820) are no longer valid.

### isValidCnae

Expand Down
62 changes: 40 additions & 22 deletions scripts/cbo.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,28 @@
#!/usr/bin/env node

import { fetchSortedRecord } from "./fetch-sorted-record.ts";
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { resolve } from "node:path";

import {
runAsEntryPoint,
serializeCodes,
splitRecord,
writeGeneratedFiles,
} from "./lookup-table.ts";
import { sortRecord } from "./sort-record.ts";

/**
* The official occupation table, `CBO2002 - Ocupacao.csv` of the "Estrutura CBO (CSV)" archive the
* Ministério do Trabalho e Emprego serves at https://cbo.mte.gov.br/cbosite/pages/downloads.jsf
* (files dated 10/07/2026, 2,725 occupations), kept byte for byte in the repository. That archive
* is only served after a reCAPTCHA, so no script can download it; the CSVs gov.br serves without
* one are an older release (updated 06/06/2025, 2,694 occupations). A new release is downloaded by
* hand, reviewed, and its digest put in `CBO_CSV_SHA256`.
*/
const CBO_CSV_PATH = resolve(import.meta.dirname, "data", "cbo2002-ocupacao.csv");

const CBO_CSV_SHA256 = "ad775f2f0b662cf1b702a9c610ed168a8dbf3a407fc67c81cc3b012e447f9893";

/** The official CSV is published in ISO-8859-1, not UTF-8. */
const CBO_CSV_DECODER = new TextDecoder("iso-8859-1");
Expand Down Expand Up @@ -60,17 +76,20 @@ export const renderCbo = (titles: Record<string, string>): Record<string, string
* the same index of \`CBO_DESCRIPTIONS\`, a module of its own (\`cbo-descriptions.ts\`) so that
* checking a code does not bundle the titles.
*
* Built from the official \`cbo2002-ocupacao.csv\` the Ministério do Trabalho e Emprego publishes
* on gov.br, a plain \`CODIGO;TITULO\` CSV in ISO-8859-1 whose codes are already the 6 digit
* form \`Cbo.code\` accepts, so no normalization is applied to them.
* Built from \`CBO2002 - Ocupacao.csv\` of the "Estrutura CBO (CSV)" archive the Ministério do
* Trabalho e Emprego serves on the CBO site (files dated 10/07/2026), a plain \`CODIGO;TITULO\`
* CSV in ISO-8859-1 whose codes are already the 6 digit form \`Cbo.code\` accepts, so no
* normalization is applied to them. The site serves it only after a reCAPTCHA, so the file is
* kept in the repository (\`scripts/data/cbo2002-ocupacao.csv\`) and checked against a reviewed
* SHA-256. The \`cbo2002-ocupacao.csv\` gov.br serves is an older release (updated 06/06/2025): it
* lacks 37 occupations of this one and still lists 6 it no longer does.
*
* Generated by \`node ./scripts/cbo.ts\`. Do not edit by hand.
*
* @see Official: https://cbo.mte.gov.br/cbosite/pages/downloads.jsf
* "Estrutura CBO (CSV)", the CBO 2002 tables as published by the Ministério do Trabalho e Emprego.
* @see Official: https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv
* The CBO 2002 occupation table, as published by the Ministério do Trabalho e Emprego.
* @see Based on: https://raw.githubusercontent.com/lucaashoff/lista-cbo-json/main/cbos.json
* Community mirror of the same table, the fallback this script was generated from before the
* official CSV was used.
* The older release of the same table on gov.br, used up to 2.4.0.
*/
export const CBO_CODES = ${serializeCodes(codes)};

Expand All @@ -86,30 +105,29 @@ export const CBO_FORMAT_REGEX = /^\\d{4}[\\s.\\-/]?\\d{2}$/;
*
* Generated by \`node ./scripts/cbo.ts\`. Do not edit by hand.
*
* @see Official: https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv
* The CBO 2002 occupation table, as published by the Ministério do Trabalho e Emprego.
* @see Official: https://cbo.mte.gov.br/cbosite/pages/downloads.jsf
* "Estrutura CBO (CSV)", the CBO 2002 tables as published by the Ministério do Trabalho e Emprego.
*/
export const CBO_DESCRIPTIONS: readonly string[] = ${JSON.stringify(descriptions)};
`,
};
};

const main = async (): Promise<void> => {
const sorted = await fetchSortedRecord(
"https://www.gov.br/trabalho-e-emprego/pt-br/assuntos/cbo/servicos/downloads/cbo2002-ocupacao.csv",
"CBO CSV",
async (response) => {
const data = parseCsv(CBO_CSV_DECODER.decode(await response.arrayBuffer()));
const csv = await readFile(CBO_CSV_PATH);
const digest = createHash("sha256").update(csv).digest("hex");

if (digest !== CBO_CSV_SHA256) {
throw new Error(
`${CBO_CSV_PATH} has SHA-256 ${digest}, not the reviewed ${CBO_CSV_SHA256}: review the new file and update CBO_CSV_SHA256`,
);
}

if (Object.keys(data).length === 0) {
throw new Error("CBO CSV holds no occupation");
}
const data = parseCsv(CBO_CSV_DECODER.decode(csv));

return data;
},
);
if (Object.keys(data).length === 0) throw new Error("CBO CSV holds no occupation");

await writeGeneratedFiles(renderCbo(sorted));
await writeGeneratedFiles(renderCbo(sortRecord(data)));
};

await runAsEntryPoint(import.meta.filename, main);
Loading
Loading