jpc_dec.c: undefined behavior in jpc_undo_roi from unvalidated RGN SPrgn shift

[huanghuihui0904]

poc_jpc_undo_roi.c
jpc_undo_roi_ubsan.log

Describe the bug

jpc_undo_roi (src/libjasper/jpc/jpc_dec.c) uses an ROI shift value taken from the JPEG-2000 RGN marker segment as a bit-shift count without bounding it against the operand width. The SPrgn field is read straight from the codestream as a byte (jpc_cs.c:875, range 0–255) and copied into ccp->roishift with no validation (jpc_dec.c:1980). A malformed codestream with a large SPrgn makes the background-shift count exceed the operand width, so

// src/libjasper/jpc/jpc_dec.c:2106, inside jpc_undo_roi
mag <<= bgshift;

is evaluated with a shift count far greater than the operand width, which is undefined behavior in C.

I found this with symbolic execution of jpc_dec.c and reproduced it from a real codestream under UndefinedBehaviorSanitizer.

Where the shift count comes from

ccp->roishift is the unvalidated SPrgn byte (0–255). The per-band shift is derived from it (jpc_dec.c:882-883):

band->roishift = (ccp->roishift + band->numbps >= JPC_PREC) ?
  (JPC_PREC - 1 - band->numbps) : ccp->roishift;     // JPC_PREC == 32

and jpc_undo_roi is then called with bgshift = ccp->roishift - band->roishift (jpc_dec.c:1192-1193):

jpc_undo_roi(band->data, band->roishift,
             ccp->roishift - band->roishift,   /* bgshift */
             band->numbps);

When ccp->roishift is large (e.g. 255), the first branch is taken, so band->roishift = 31 - numbps and

bgshift = ccp->roishift - band->roishift = 255 - (31 - numbps) = 224 + numbps

i.e. bgshift is in the low hundreds (far past the operand width) and is used directly as the shift count at line 2106.

The relevant validation only clamps negative ROI shifts

// jpc_dec.c:2078-2084
if (roishift < 0) {
    jas_logwarnf("warning: forcing negative ROI shift to zero ...\n");
    roishift = 0;
}

There is no upper-bound check on roishift, and bgshift is not checked at all before being used as a shift count.

To Reproduce

The PoC (poc_jpc_undo_roi.c) and its full sanitizer log (jpc_undo_roi_ubsan.log) are attached to this issue.

poc_jpc_undo_roi.c embeds a 212-byte 16×16 grayscale JPEG-2000 codestream produced by JasPer's own encoder, into whose main header a single RGN marker segment has been spliced just before the first SOT:

FF 5E 00 05 00 00 FF
| RGN | Lrgn| Cr| Sr| SPrgn = 0xFF (255)

It decodes those bytes via jas_image_decode, so the PoC depends on nothing but libjasper.

The following commands reproduce the issue from a clean checkout (run from the root of the JasPer source tree; the build directory is placed next to the checkout because JasPer rejects in-source build directories):

git clone https://github.com/jasper-software/jasper.git
cd jasper
git checkout 5ce57f67e0011e0d2483992a5d77a092b1b64eb7   # newest master at time of testing

cmake -S . -B ../jas_build -DCMAKE_BUILD_TYPE=Debug \
  -DJAS_ENABLE_SHARED=ON -DJAS_ENABLE_PROGRAMS=OFF -DJAS_ENABLE_DOC=OFF \
  -DCMAKE_C_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g -O0" \
  -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined" \
  -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address,undefined"
cmake --build ../jas_build --target libjasper -j

gcc -g -O0 -fsanitize=address,undefined -fno-omit-frame-pointer -I../jas_build/src/libjasper/include -Isrc/libjasper/include poc_jpc_undo_roi.c -L../jas_build/src/libjasper -ljasper -Wl,-rpath,../jas_build/src/libjasper -o poc_jpc_undo_roi

UBSAN_OPTIONS=print_stacktrace=1 ./poc_jpc_undo_roi 2>&1 | tee jpc_undo_roi_ubsan.log

(UBSAN_OPTIONS=print_stacktrace=1 is required for the call stack below; without it UBSan prints only the one-line runtime error diagnostic.)

Observed output

[poc] decoding 212-byte JPEG-2000 codestream (RGN SPrgn=255)
src/libjasper/jpc/jpc_dec.c:2106:9: runtime error: shift exponent 235 is too large for 64-bit type 'long int'
    #0 0x7f38fecfec79 in jpc_undo_roi src/libjasper/jpc/jpc_dec.c:2106
    #1 0x7f38feceea64 in jpc_dec_tiledecode src/libjasper/jpc/jpc_dec.c:1192
    #2 0x7f38fece2920 in jpc_dec_process_sod src/libjasper/jpc/jpc_dec.c:671
    #3 0x7f38fecdf841 in jpc_dec_decode src/libjasper/jpc/jpc_dec.c:434
    #4 0x7f38fecde439 in jpc_decode src/libjasper/jpc/jpc_dec.c:270
    #5 0x7f38fec79b2c in jas_image_decode src/libjasper/base/jas_image.c:477
    #6 0x55956a01356a in main poc_jpc_undo_roi.c:37
    #7 0x7f38fdeac082 in __libc_start_main ../csu/libc-start.c:308
    #8 0x55956a0132cd in _start (poc_jpc_undo_roi+0x12cd)

[poc] jas_image_decode returned 0x608000000240

The UBSan diagnostic pinpoints the mag <<= bgshift shift at jpc_dec.c:2106, and the stack (jas_image_decode → jpc_decode → jpc_dec_tiledecode → jpc_undo_roi) shows it is reached directly from the public decode entry point on the crafted codestream.

Affected version: upstream master at commit 5ce57f67e0011e0d2483992a5d77a092b1b64eb7.

Impact

On x86 the shift count is typically masked by the hardware, so a stock build may silently produce incorrect sample values; on architectures or builds that trap or with -fsanitize=shift, decoding aborts. It is reachable from any JPEG-2000 decode entry point (jas_image_decode, jpc_decode, the jp2 wrapper). One fix direction is to reject ROI shift values that cannot be represented safely before they are used as shift counts, or to clamp/guard the derived bgshift in jpc_undo_roi.