Skip to content

fix: support binary map keys in unions#239

Merged
tomas-abrahamsson merged 1 commit into
tomas-abrahamsson:masterfrom
thalesmg:20250904-fix-union-binary-map-keys-upstream
Sep 14, 2025
Merged

fix: support binary map keys in unions#239
tomas-abrahamsson merged 1 commit into
tomas-abrahamsson:masterfrom
thalesmg:20250904-fix-union-binary-map-keys-upstream

Conversation

@thalesmg

@thalesmg thalesmg commented Sep 4, 2025

Copy link
Copy Markdown
Contributor

Previously, when using maps with binary keys, flat unions and verification, unions would not work, even with binary map keys as expected.

For example, with the following definitions:

Src = """
syntax = "proto3";

message test {
  map<string, string> args = 1;
}

message union {
  oneof u {
    int32 a = 1;
    string b = 2;
  }
}
""".
{ok, Mod, Bin} = gpb_compile:string(aaa, Src, [binary, strings_as_binaries, {maps, true}, {maps_oneof, flat}, {maps_key_type, binary}, {verify, always}, {maps_unset_optional, omitted}]).
code:load_binary(Mod, "aaa.erl", Bin).

... normal maps work as expected:

7>   Encoded1 = Mod:encode_msg(#{<<"args">> => #{<<"hey">> => <<"hi">>}}, test).
 <<10,9,10,3,104,101,121,18,2,104,105>>
8>   Mod:decode_msg(Encoded1, test).
 #{<<"args">> => #{<<"hey">> => <<"hi">>}}

... but unions break, due to verifier failing to consider binary keys when encoding:

20>   %% Should produce <<8, 1>>.
      Mod:encode_msg(#{<<"a">> => 1}, union).
** exception error: {gpb_type_error,{{multiple_oneof_keys,[],u},
                                     [{value,#{<<"a">> => 1}},{path,"union.u"}]}}
     in function  aaa:mk_type_error/3 (aaa.erl, line 567)
     in call from aaa:encode_msg/3 (aaa.erl, line 79)

... and decoding due to incorrect generated code which produced <<(atom_to_list(SomeAtom))>> for the map key:

21>   Mod:decode_msg(<<8, 1>>, union).
** exception error: {gpb_error,{decoding_failure,{<<8,1>>,
                                                  union,
                                                  {error,badarg,
                                                         [{aaa,dfp_read_field_def_union,6,
                                                               [{file,"aaa.erl"},
                                                                {line,292},
                                                                {error_info,#{cause => {1,integer,type,"a"},
                                                                              function => format_bs_fail,module => erl_erts_errors}}]},
                                                          {aaa,decode_msg_1_catch,3,[{file,"aaa.erl"},{line,227}]},
                                                          {erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},
                                                          {shell,exprs,7,[{file,"shell.erl"},{line,916}]},
                                                          {shell,eval_exprs,7,[{file,"shell.erl"},{line,872}]},
                                                          {shell,restricted_eval_loop,5,
                                                                 [{file,"shell.erl"},{line,866}]}]}}}}
     in function  aaa:decode_msg_1_catch/3 (aaa.erl, line 231)

Previously, when using maps with binary keys, flat unions and verification,
unions would not work, even with binary map keys as expected.

For example, with the following definitions:

```erlang
Src = """
syntax = "proto3";

message test {
  map<string, string> args = 1;
}

message union {
  oneof u {
    int32 a = 1;
    string b = 2;
  }
}
""".
{ok, Mod, Bin} = gpb_compile:string(aaa, Src, [binary, strings_as_binaries, {maps, true}, {maps_oneof, flat}, {maps_key_type, binary}, {verify, always}, {maps_unset_optional, omitted}]).
code:load_binary(Mod, "aaa.erl", Bin).
```

... normal maps work as expected:

```erlang
7>   Encoded1 = Mod:encode_msg(#{<<"args">> => #{<<"hey">> => <<"hi">>}}, test).
 <<10,9,10,3,104,101,121,18,2,104,105>>
8>   Mod:decode_msg(Encoded1, test).
 #{<<"args">> => #{<<"hey">> => <<"hi">>}}
```

... but unions break, due to verifier failing to consider binary keys when encoding:

```erlang
20>   %% Should produce <<8, 1>>.
      Mod:encode_msg(#{<<"a">> => 1}, union).
** exception error: {gpb_type_error,{{multiple_oneof_keys,[],u},
                                     [{value,#{<<"a">> => 1}},{path,"union.u"}]}}
     in function  aaa:mk_type_error/3 (aaa.erl, line 567)
     in call from aaa:encode_msg/3 (aaa.erl, line 79)

```

... and decoding due to incorrect generated code which produced
`<<(atom_to_list(SomeAtom))>>` for the map key:

```erlang
21>   Mod:decode_msg(<<8, 1>>, union).
** exception error: {gpb_error,{decoding_failure,{<<8,1>>,
                                                  union,
                                                  {error,badarg,
                                                         [{aaa,dfp_read_field_def_union,6,
                                                               [{file,"aaa.erl"},
                                                                {line,292},
                                                                {error_info,#{cause => {1,integer,type,"a"},
                                                                              function => format_bs_fail,module => erl_erts_errors}}]},
                                                          {aaa,decode_msg_1_catch,3,[{file,"aaa.erl"},{line,227}]},
                                                          {erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},
                                                          {shell,exprs,7,[{file,"shell.erl"},{line,916}]},
                                                          {shell,eval_exprs,7,[{file,"shell.erl"},{line,872}]},
                                                          {shell,restricted_eval_loop,5,
                                                                 [{file,"shell.erl"},{line,866}]}]}}}}
     in function  aaa:decode_msg_1_catch/3 (aaa.erl, line 231)
```
@tomas-abrahamsson

Copy link
Copy Markdown
Owner

Thanks for a good report and fix. I'll merge it and make a new release.

@tomas-abrahamsson tomas-abrahamsson merged commit f0b2433 into tomas-abrahamsson:master Sep 14, 2025
5 checks passed
@tomas-abrahamsson

Copy link
Copy Markdown
Owner

Merged and included in 4.21.5.

@thalesmg thalesmg deleted the 20250904-fix-union-binary-map-keys-upstream branch September 15, 2025 00:15
@thalesmg

Copy link
Copy Markdown
Contributor Author

Thanks! 🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants