Skip to content

source-normalization.md

Latest commit

 

History

History
86 lines (59 loc) · 2.78 KB

source-normalization.md

File metadata and controls

86 lines (59 loc) · 2.78 KB
Raw

Source Normalization

Status: active 0.5.0 contract

elderheim-core owns source normalization before language parsing starts. The normalizer is no_std and writes normalized bytes to a caller-provided sink instead of allocating.

Line Endings

Input line endings are normalized to \n.

Input Output
LF LF
CRLF LF
CR LF

The normalized stream is the input to later lexing and diagnostics.

Byte Policy

The current policy accepts printable ASCII bytes 0x20..=0x7e plus line endings. Other bytes fail with E-CORE-SOURCE-BYTE.

This is a compiler-foundation policy, not a Dartmouth BASIC grammar. Later language profiles may add profile-specific source restrictions after normalization.

Blank Lines

NormalizationPolicy::BASIC_STRICT rejects blank normalized lines with E-CORE-SOURCE-BLANK-LINE. A line containing only spaces is blank.

NormalizationPolicy::PRESERVE_BLANK_LINES keeps blank lines for tests and future non-BASIC source policies.

Limits

The normalizer checks the configured source byte limit before writing to the sink. It also enforces the configured line limit after line-ending normalization.

Sink Error Contract

Any Err returned by normalize_source invalidates all bytes written to the sink during that call. Callers must discard or reset partial sink contents before using the sink again.

Source IDs

SourceId is computed over normalized bytes. Inputs that differ only by LF, CRLF, or CR line endings therefore produce the same source ID.

The ID is intended as a stable compiler-internal identity for reports and fixtures. It is not a cryptographic digest.

SourceId must not be used as a security boundary, cache-integrity boundary, or proof that two source inputs are identical.

At v0.13.5, normalized source additionally receives a cryptographic SourceDigest from the shared typed digest subsystem. SourceId remains the diagnostic identifier and is never promoted or converted into that digest.

Source Construction

Source::from_normalized is the public constructor for diagnostic source views. It accepts only normalized bytes and re-checks the normalized source byte, line-limit, and blank-line policy before returning a Source.

Normalized source validation is intentionally centralized in source/normalize.rs. The normalizer and Source::from_normalized share the same normalized-byte scanner so the byte policy, blank-line policy, line count, and line-limit checks cannot drift between source ingestion and diagnostic source views.

Verification

The 0.5.0 stop requires:

  • LF, CRLF, and CR normalization fixtures;
  • invalid byte fixtures;
  • blank-line policy fixtures;
  • large source rejection tests;
  • source ID stability tests;
  • public source construction rejection tests;
  • sink error propagation tests.