Track std.base64 non-ASCII string semantics vs official Jsonnet

[He-Pin]

Motivation

During the stdlib audit, std.base64 / std.base64Decode showed a non-ASCII string semantics difference between sjsonnet and official C++ Jsonnet v0.22.0.

We are not changing this in the current stdlib correctness PR because this behavior may be visible to users and jrsonnet currently makes the same UTF-8 choice as sjsonnet. This issue tracks the discrepancy separately.

Evidence

Official jsonnet v0.22.0:

std.base64("é")                         // "6Q=="
std.base64Decode("w6k=")                 // "é"
std.base64Decode("6Q==")                 // "é"
std.base64DecodeBytes(std.base64("é"))   // [233]
std.base64("Ā")                          // runtime error: Can only base64 encode strings / arrays of single bytes.

Current sjsonnet:

std.base64("é")                         // "w6k="
std.base64Decode("w6k=")                 // "é"
std.base64Decode("6Q==")                 // "�"
std.base64DecodeBytes(std.base64("é"))   // [195, 169]

Local jrsonnet check:

jrsonnet 0.5.0-pre98
commit 80cd36abd868507312e2cc2c78cb0f55a684c620

jrsonnet matches sjsonnet's UTF-8-byte behavior here:

std.base64("é")                         // "w6k="
std.base64Decode("w6k=")                 // "é"
std.base64Decode("6Q==")                 // runtime error: bad utf8
std.base64DecodeBytes(std.base64("é"))   // [195, 169]

Root Cause

Official C++ Jsonnet's stdlib implements string base64 as codepoint/char bytes:

local bytes =
  if std.isString(input) then
    std.map(std.codepoint, input)
  else
    input;

base64Decode(str)::
  local bytes = std.base64DecodeBytes(str);
  std.join('', std.map(std.char, bytes)),

sjsonnet currently encodes string input as UTF-8 bytes and decodes bytes as UTF-8 strings.

Proposed Direction

If sjsonnet decides to align strictly with official Jsonnet:

• Keep byte-array std.base64 and std.base64DecodeBytes behavior unchanged.
• Change string std.base64 to encode each character as a single byte, rejecting codepoints above 255.
• Change std.base64Decode to map decoded bytes directly to std.char(byte) semantics instead of UTF-8 decoding.
• Add directional tests for "é", "w6k=", "6Q==", and a high-codepoint rejection case such as "Ā".

This would intentionally diverge from jrsonnet's current UTF-8 behavior but match official C++ Jsonnet v0.22.0.

[He-Pin]

@CertainLach fyi

[CertainLach]

I would argue that the upstream is wrong

base64 is a function over bytes, it makes no sense to treat passed string as codepoints and truncate them, this isn't the behavior sane people expect.

And for the most usecases like encoding kubernetes secrets it is surprising if it silently truncates non-utf8 strings
Same with the silent lossy handling of base64Decode

Note that this kind of insane behavior

std.base64("é")                         // "6Q=="
std.base64DecodeBytes(std.base64("é"))   // [233]

Is not even being followed in go-jsonnet:

std.base64("é")                         // "w6k="
std.base64DecodeBytes(std.base64("é"))   // [195, 169]

As it requires the interpreter to imply that the strings are byte-strings with the legacy (non-utf8, non-ascii), and not the unicode as you would expect given that the slices are codepoint-based (Meaning é is being encoded as [233] and not as [195, 169] according to utf-8, which is the sane assumption given that we want bytes, we pass unicode string, and the canonical byte representation of unicode string in byte code units is utf-8)

[CertainLach]

Even if the codepoint behavior made any sense, std.base64("Ā") should require range of the codepoints to be 0<=x<=127, and not 0<=x<=255, because otherwise it just corrupts the values during roundtrip in go-jsonnet

[He-Pin]

That's true, I also feel that these behaviors are a bit strange and do not align with user intuition.

[CertainLach]

Even if the codepoint behavior made any sense, std.base64("Ā") should require range of the codepoints to be 0<=x<=127, and not 0<=x<=255, because otherwise it just corrupts the values during roundtrip in go-jsonnet

Same with base64Decode - with codepoint range limitation it should limit bytes to ACSII (0..=127) too

E.g go-jsonnet is unable to handle this: std.base64(std.base64Decode(std.base64([196, 128])))

Just because this codepoint range check is dumb

[CertainLach]

I also think none of the implementations should silently corrupt the byte values, e.g I my implementation
std.base64Decode("6Q==") // runtime error: bad utf8

Is better, and this corruption should be added explicitly if needed:
std.base64Decode("6Q==", lossy = true) // �

[CertainLach]

Cc: @johnbartholomew

[He-Pin]

In any case, in my actual work, sjsonnet serves as a component of the data orchestration for our Java MCP server. Therefore, I believe that the Base64 implementation here should be as compatible as possible with Java's Base64 implementation within my specific context; otherwise, it would be truly counter-intuitive.

[CertainLach]

Not sure if I want this to land in master (not until the agreement is reached), uploaded just as a reference for the implemenetation I believe is correct, and what I believe should be changed in sjsonnet, and in go-jsonnet together with arbitrary codepoint range limitation in std.base64 (Which doesn't make sense, given that it encodes values in utf8 anyway)
deltarocks/jrsonnet@40e0ccd

It adds the lossy parameter for base64Decode:

std.base64Decode("6Q==") // Error
std.base64Decode("6Q==", lossy=true) // �

[He-Pin]

Additional four-implementation check from the stdlib alignment audit:

Environment:

• official C++ Jsonnet: Jsonnet commandline interpreter v0.22.0
• go-jsonnet: Jsonnet commandline interpreter (Go implementation) v0.22.0
• jrsonnet: source-built jrsonnet 0.5.0-pre98, commit 80cd36abd868507312e2cc2c78cb0f55a684c620
• sjsonnet: PR fix: Align stdlib 1:1 edge semantics #800 worktree assembly

Observed behavior:

expression	official Jsonnet 0.22.0	go-jsonnet 0.22.0	source-built jrsonnet	sjsonnet PR #800
std.base64("hello")	"aGVsbG8="	"aGVsbG8="	"aGVsbG8="	"aGVsbG8="
std.base64([104,101,108,108,111])	"aGVsbG8="	"aGVsbG8="	"aGVsbG8="	"aGVsbG8="
std.base64("é")	"6Q=="	"w6k="	"w6k="	"w6k="
std.base64("Ā")	error	error	"xIA="	"xIA="
std.base64([256])	error	error	error	error
std.base64Decode("Zg==")	"f"	"f"	"f"	"f"
std.base64Decode("Zg")	error	error	error	error

Interpretation:

• Official std.jsonnet treats string input as codepoints and requires each codepoint to fit in one byte. That gives "é" -> [233] -> "6Q==" and rejects "Ā" because codepoint 256 is outside the byte range.
• go-jsonnet 0.22.0 is mixed here: "é" is accepted but encoded as UTF-8 bytes, while "Ā" is rejected because it checks the codepoint range before encoding.
• jrsonnet and sjsonnet both use UTF-8-byte behavior for strings and therefore accept "Ā".

This reinforces that there is no cross-community consensus here. Keeping this as a tracking/policy issue, rather than changing sjsonnet inside the current correctness PR, still looks right.

[johnbartholomew]

Thanks for auditing this and exploring the situation so thoroughly.

base64 is a function over bytes, it makes no sense to treat passed string as codepoints and truncate them, this isn't the behavior sane people expect.

Fully agree - C++ Jsonnet behaviour is just bad here.

From the documentation (https://jsonnet.org/ref/stdlib.html#std-base64, and for the decode functions), I will note:

• The documentation advises against using std.base64Decode (which returns a string) and suggests to use std.base64DecodeBytes (which returns an array) in combination with std.decodeUTF8.
• std.base64 takes a string, or an array of numbers (limited to range 0 to 255, i.e., bytes).

I would surmise that the intention is that base64 should not really be used to encode strings (Unicode text) at all, but rather byte arrays, which might be anything (for example you could do local arr = importbin "hello.jpeg"). If you want to base64 encode a string you should prefer to call std.base64(std.encodeUTF8("some string")).

I think the discussion so far has already covered everything. I see these options:

1. Change std.base64 to entirely reject string parameter, with an error message that explains to callers that they should call std.encodeUTF8 to get a byte array if they want to base64-encode some text for some reason. In some sense this is cleanest but it might be too disruptive.

2. (This idea was already mentioned by CertainLach) Change std.base64 to reject string parameter if it contains any non-ASCII characters (i.e., any Unicode codepoint > 127). This is slightly less disruptive than rejecting string inputs entirely. If I've understood the situation correctly, pure ASCII text should round-trip through any of the existing implementations including the strange existing C++ Jsonnet behaviour, so the inputs that are still allowed should be forward and backward compatible. Note that array inputs must still support full byte arrays (values in range 0 to 255), only string inputs should be restricted to ASCII.

3. Define std.base64(x) with string input to behave as std.base64(std.encodeUTF8(x)). If I've understood correctly this matches most of the existing implementations, it just requires a change to C++ Jsonnet.

(1) and (2) have the advantage of breaking loudly rather than silently changing behaviour for anyone currently relying (unintentionally or not) on the existing C++ Jsonnet behaviour.

[CertainLach]

The documentation advises against using std.base64Decode (which returns a string) and suggests to use std.base64DecodeBytes (which returns an array) in combination with std.decodeUTF8.

I believe this was added exactly because of how does base64 function support codepoints over 127

I would surmise that the intention is that base64 should not really be used to encode strings (Unicode text) at all

Yet there is many cases where this is what the software expects for some reason
Also there is a consensus on how should Unicode strings be represented as bytes (utf8), so again should not be a problem

(1) and (2) have the advantage of breaking loudly rather than silently changing behaviour for anyone currently relying (unintentionally or not) on the existing C++ Jsonnet behaviour.

Given that this behavior is already broken for go-jsonnet user (go-jsonnet already behaves like jrsonnet/sjsonnet in that regard, with the exception of base64 limiting codepoints to be <255, which is again not correct), I think this is a time to rip the bandaid and implement the behavior users most likely want (3)

Btw, what is your opinion on replacing implicit invalid utf8 replacement with the explicit one?

std.base64Decode("6Q==") // Error (using lossy=false default)
std.base64Decode("6Q==", lossy=true) // �

Given that the current docs advise against using such string anyway, maybe it is worth adding at the same time with documentation reword?