Repository navigation
fix: Match Cesium's oct encoding of vertex normals - #87
Open
kylebarron wants to merge 1 commit into
Open
kylebarron wants to merge 1 commit into
kylebarron wants to merge 1 commit into
Conversation
`oct_encode` mapped each component from [-1, 1] to [0, 255] with `floor((v * 0.5 + 0.5) * 256)`. For a component of exactly 1.0 that is 256, which wraps to 0 when cast to uint8, so a normal along +x encoded as `[0, 128]` and decoded as `[-1, 0, 0]`, the opposite direction. Cesium's `toSNorm`, which the function cites and which quantized-mesh-tile ports, uses `round((v * 0.5 + 0.5) * 255)`, and decoders divide by 255. Use the same formula. Besides fixing the overflow, this removes a small bias in every encoded normal. The new tests check axis-aligned normals round-trip and that output matches quantized-mesh-tile's `octEncode` exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kylebarron
added this pull request to stack #91
October 5, 2026 21:52
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was written by Claude (Claude Code), not by @kylebarron.
Fixes
oct_encodeflipping normals that have a component of exactly 1.0, and matches Cesium's encoding.The bug
oct_encodemapped each component from [-1, 1] to [0, 255] withfloor((v * 0.5 + 0.5) * 256). For a component of exactly 1.0 that gives 256, which wraps to 0 when cast touint8. So a normal along +x encoded as[0, 128]and decoded as[-1, 0, 0], the opposite direction.Fix
Use
round((v * 0.5 + 0.5) * 255). That's Cesium'sCesiumMath.toSNorm, which the function cites and which quantized-mesh-tile ports, and decoders divide by 255 to reverse it.Behavior change
This changes the encoded bytes of most vertex normals by up to one unit. It fixes the overflow and also removes a small bias in every encoded normal. Decoded normals get closer to the true values. It's in the CHANGELOG.
Tests
New tests check that axis-aligned normals round-trip, and that the output exactly matches quantized-mesh-tile's
octEncodeon 1000 random unit vectors. Three tests fail without the fix and pass with it.🤖 Written by Claude Code