Skip to content

SVStringBreak fails parsing when locale is set - #105

Merged
analytophile merged 2 commits into
CSHS-CWRA:mainfrom
robmtud:robmtud/SVStringBreak_fix
Aug 19, 2026
Merged

analytophile merged 2 commits into
CSHS-CWRA:mainfrom
robmtud:robmtud/SVStringBreak_fix

Conversation

@robmtud

@robmtud robmtud commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Previously the function SVStringBreak used the C function strxfrm to extract the substrings for variable name and layer number. This function considers current locale settings for string processing and it's purpose relates to comparing strings in a given locale. It's usage here is relying on undefined behaviour, because the return value is not checked and that value gives the necessary target buffer size minus one. That target size in turn can be larger than the length of the substrings extracted here, depending on the C implementation and locale.

The function was replaced with a call to strncpy and the platform specific distinctions removed.

The issue appeared for me only when calling the Raven BMI bindings from within an IPython session. Raven itself does not initialize any locales, but in the IPython session the locale gets intialized and set to the environment settings. Than parsing of variables like SOIL[0] would fail and Raven would exit with ERROR : StringToSVtype: Unrecognized State Variable (2).

I was able to reproduce the issue within Raven itself by setting a locale in Raven before the first call to strxfrm, e.g. adding the line

setlocale(LC_ALL, "en_US.UTF-8");

in SVStringBreak.

Because of the undefined behavior being implementation specific I do not know if it will show the same symptoms on other platforms, I tested on a Debian Linux system.

I also noticed there were different code paths for different platforms with __APPLE__ and __CYGWIN__ having their target buffer sizes increased in the strxfrm call. My guess is this might relate to this issue.

robmtud and others added 2 commits August 17, 2026 15:13
Previously the function 'SVStringBreak' used the C function 'strxfrm' to
extract the substrings for variable name and layer number.
This function considers current locale settings for string processing
and it's purpose relates to comparing strings in a given locale.
It's usage here is relying on undefined behaviour, because the return
value is not checked and that value gives the necessary target buffer
size minus one. That target size in turn can be larger than the length
of the substrings extracted here, depending on the C implementation and
locale.

The function was replaced with a call to 'strncpy' and the platform
specific distinctions removed.
@analytophile
analytophile merged commit 7a4ee23 into CSHS-CWRA:main Aug 19, 2026
15 checks passed
@robmtud
robmtud deleted the robmtud/SVStringBreak_fix branch August 20, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants