LJpeg: support square MCU layout (Sony LJpeg, -32%) by LebedevRI · Pull Request #698 · darktable-org/rawspeed

LebedevRI · 2024-03-26T02:05:46Z

This is essentially rebased #483, and partially reverts #484,
now that i've figured out how we can deduce the MCU size:
e9e3f1b + 63f7170 !

This, hopefully, partially solves the roadblock towards different predictor support.

Comparing /home/lebedevri/rawspeed/build-Clang18-release/src/utilities/rsbench/rsbench-old to /home/lebedevri/rawspeed/build-Clang18-release/src/utilities/rsbench/rsbench
Benchmark                                                                                                                               Time             CPU      Time Old      Time New       CPU Old       CPU New
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9394-compressed-lossless.DNG/threads:32/process_time/real_time_pvalue                 0.0081          0.0066      U Test, Repetitions: 27 vs 27
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9394-compressed-lossless.DNG/threads:32/process_time/real_time_mean                  -0.0160         -0.0162            11            11           364           359
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9394-compressed-lossless.DNG/threads:32/process_time/real_time_median                -0.0181         -0.0156            11            11           361           356
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9394-compressed-lossless.DNG/threads:32/process_time/real_time_stddev                +0.1098         +0.1135             0             0             8             9
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9394-compressed-lossless.DNG/threads:32/process_time/real_time_cv                    +0.1279         +0.1318             0             0             0             0
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9395-compressed-lossless.DNG/threads:32/process_time/real_time_pvalue                 0.3683          0.4781      U Test, Repetitions: 27 vs 27
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9395-compressed-lossless.DNG/threads:32/process_time/real_time_mean                  +0.0089         +0.0082            15            15           464           468
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9395-compressed-lossless.DNG/threads:32/process_time/real_time_median                +0.0145         +0.0152            15            15           461           468
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9395-compressed-lossless.DNG/threads:32/process_time/real_time_stddev                +0.2341         +0.2787             0             0            11            14
./Adobe DNG Converter/Canon EOS 5D Mark III/5G4A9395-compressed-lossless.DNG/threads:32/process_time/real_time_cv                    +0.2232         +0.2683             0             0             0             0
./Adobe DNG Converter/Canon EOS 5D Mark IV/B13A0729.dng/threads:32/process_time/real_time_pvalue                                      0.2913          0.2326      U Test, Repetitions: 27 vs 27
./Adobe DNG Converter/Canon EOS 5D Mark IV/B13A0729.dng/threads:32/process_time/real_time_mean                                       -0.0098         -0.0102            30            29           939           930
./Adobe DNG Converter/Canon EOS 5D Mark IV/B13A0729.dng/threads:32/process_time/real_time_median                                     -0.0193         -0.0181            30            29           936           920
./Adobe DNG Converter/Canon EOS 5D Mark IV/B13A0729.dng/threads:32/process_time/real_time_stddev                                     -0.1583         -0.1647             1             1            28            23
./Adobe DNG Converter/Canon EOS 5D Mark IV/B13A0729.dng/threads:32/process_time/real_time_cv                                         -0.1500         -0.1561             0             0             0             0
./Fujifilm/X100S/fujifilm-x100s-daylight-DSCF9505.dng/threads:32/process_time/real_time_pvalue                                        0.0001          0.0000      U Test, Repetitions: 27 vs 27
./Fujifilm/X100S/fujifilm-x100s-daylight-DSCF9505.dng/threads:32/process_time/real_time_mean                                         -0.0312         -0.0321             4             4           129           125
./Fujifilm/X100S/fujifilm-x100s-daylight-DSCF9505.dng/threads:32/process_time/real_time_median                                       -0.0479         -0.0480             4             4           128           122
./Fujifilm/X100S/fujifilm-x100s-daylight-DSCF9505.dng/threads:32/process_time/real_time_stddev                                       +0.8491         +0.7719             0             0             4             8
./Fujifilm/X100S/fujifilm-x100s-daylight-DSCF9505.dng/threads:32/process_time/real_time_cv                                           +0.9087         +0.8306             0             0             0             0
./Samsung/Galaxy S21 Ultra/20230712_115041.dng/threads:32/process_time/real_time_pvalue                                               0.0000          0.0000      U Test, Repetitions: 27 vs 27
./Samsung/Galaxy S21 Ultra/20230712_115041.dng/threads:32/process_time/real_time_mean                                                +0.0128         +0.0128           149           151           149           151
./Samsung/Galaxy S21 Ultra/20230712_115041.dng/threads:32/process_time/real_time_median                                              +0.0128         +0.0128           149           151           149           151
./Samsung/Galaxy S21 Ultra/20230712_115041.dng/threads:32/process_time/real_time_stddev                                              -0.0387         -0.0437             0             0             0             0
./Samsung/Galaxy S21 Ultra/20230712_115041.dng/threads:32/process_time/real_time_cv                                                  -0.0509         -0.0558             0             0             0             0
./Samsung/Galaxy S23 Ultra/20231214_130645.dng/threads:32/process_time/real_time_pvalue                                               0.0066          0.7687      U Test, Repetitions: 27 vs 27
./Samsung/Galaxy S23 Ultra/20231214_130645.dng/threads:32/process_time/real_time_mean                                                +0.0055         -0.0000           580           584          7001          7001
./Samsung/Galaxy S23 Ultra/20231214_130645.dng/threads:32/process_time/real_time_median                                              +0.0076         -0.0107           579           583          7224          7147
./Samsung/Galaxy S23 Ultra/20231214_130645.dng/threads:32/process_time/real_time_stddev                                              +0.5071         -0.0321             4             6          1181          1143
./Samsung/Galaxy S23 Ultra/20231214_130645.dng/threads:32/process_time/real_time_cv                                                  +0.4988         -0.0320             0             0             0             0
./Sony/ILCE-7RM5/7RM5-S35-LosslessCompressedMedium.ARW/threads:32/process_time/real_time_pvalue                                       0.0000          0.0000      U Test, Repetitions: 27 vs 27
./Sony/ILCE-7RM5/7RM5-S35-LosslessCompressedMedium.ARW/threads:32/process_time/real_time_mean                                        -0.3269         -0.3256            29            19           910           614
./Sony/ILCE-7RM5/7RM5-S35-LosslessCompressedMedium.ARW/threads:32/process_time/real_time_median                                      -0.3205         -0.3224            29            19           908           615
./Sony/ILCE-7RM5/7RM5-S35-LosslessCompressedMedium.ARW/threads:32/process_time/real_time_stddev                                      -0.0947         -0.0594             1             1            21            20
./Sony/ILCE-7RM5/7RM5-S35-LosslessCompressedMedium.ARW/threads:32/process_time/real_time_cv                                          +0.3449         +0.3946             0             0             0             0
OVERALL_GEOMEAN                                                                                                                      -0.0593         -0.0594             0             0             1             1

…frame size

…tInterval/`

codecov · 2024-03-26T03:14:37Z

Codecov Report

Attention: Patch coverage is 60.00000% with 50 lines in your changes are missing coverage. Please review.

Project coverage is 60.84%. Comparing base (8297c38) to head (7d97477).

Files	Patch %	Lines
...rc/librawspeed/decompressors/LJpegDecompressor.cpp	62.13%	39 Missing ⚠️
src/librawspeed/decompressors/LJpegDecoder.cpp	60.00%	4 Missing and 2 partials ⚠️
...zz/librawspeed/decompressors/LJpegDecompressor.cpp	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #698      +/-   ##
===========================================
- Coverage    60.87%   60.84%   -0.03%     
===========================================
  Files          272      272              
  Lines        16464    16490      +26     
  Branches      2084     2090       +6     
===========================================
+ Hits         10022    10033      +11     
- Misses        6314     6328      +14     
- Partials       128      129       +1

Flag	Coverage Δ
benchmarks	`11.78% <0.80%> (-0.02%)`	⬇️
integration	`44.95% <68.18%> (-0.01%)`	⬇️
linux	`57.05% <63.02%> (-0.06%)`	⬇️
macOS	`25.03% <0.00%> (-0.04%)`	⬇️
rpu_u	`44.95% <68.18%> (-0.01%)`	⬇️
unittests	`21.46% <0.80%> (-0.04%)`	⬇️
windows	`∅ <ø> (∅)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kmilos · 2024-03-26T09:51:58Z

This, hopefully, partially solves the roadblock towards different predictor support.

Yay! 🚀

LebedevRI added 25 commits March 26, 2024 02:41

LJpegDecoder: check that frame/tile areas match, not dimensions

3ee4910

LJpegDecoder: Maximal output tile size should be a multiple of LJpeg …

63f7170

…frame size

LJpegDecoder: compute MCU layout from tile size and ljpeg frame size

97c82e3

LJpegDecoder: extract jpegFrameDim helper variable

01470cc

LJpegDecompressor: extract cps variable from frame

bce1bee

LJpegDecoder: plumb MCUSize into LJpegDecompressor

00ef6d6

LJpegDecompressor: rewrite checks in ctor for MCU size

debd42d

LJpegDecompressor: we do not expect to support row-partial MCU's

dfdbb30

LJpegDecompressor: template decodeN() on MCUSize

087d5c6

LJpegDecompressor: template decodeRowN() on MCUSize

c64abc5

LJpegDecompressor: `s/numRowsPerRestartInterval/numLJpegRowsPerRestar…

f6214c3

…tInterval/`

LJpegDecompressor: s/rowOfRestartInterval/ljpegRowOfRestartInterval/

627ae6b

LJpegDecompressor: s/row/ljpegRow/

08270f7

LJpegDecompressor: correctly calculate numRestartIntervals

a5392fc

LJpegDecompressor: correctly calculate output row

16599fb

LJpegDecompressor: s/fullBlocks/numFullMCUs/

a87d59c

LJpegDecompressor::decodeRowN(): iterate over mcu index, not col

43dea6c

LJpegDecompressor: s/block/mcu/

acac68f

LJpegDecompressor: take outStripe, not outRow

af901e7

LJpegDecompressor: actually produce a stripe

ad03fff

LJpegDecompressor: rewrite predictor reloading

b6da886

LJpegDecompressor: extract outTile in the non-edge-pixel code

dfff2ae

LJpegDecompressor: drop incorrect invariant

9f89219

LJpegDecompressor: support {2,2} MCU's

e2ff718

ArwDecoder: LJpeg decompressor supports {2,2} MCU's, drop workaround

7d97477

LebedevRI force-pushed the ljpeg-mcu branch from b2498f6 to 7d97477 Compare March 26, 2024 02:41

LJpegDecompressor: don't inline decodeN() into decode()

53dbe15

LebedevRI changed the title ~~LJpeg: support square MCU layout (as used by Sony LJpeg)~~ LJpeg: support square MCU layout (Sony LJpeg, -32%) Mar 26, 2024

LebedevRI merged commit 1ef967a into darktable-org:develop Mar 26, 2024

LebedevRI deleted the ljpeg-mcu branch March 26, 2024 03:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LJpeg: support square MCU layout (Sony LJpeg, -32%)#698

LJpeg: support square MCU layout (Sony LJpeg, -32%)#698
LebedevRI merged 26 commits intodarktable-org:developfrom
LebedevRI:ljpeg-mcu

LebedevRI commented Mar 26, 2024 •

edited

Loading

Uh oh!

codecov bot commented Mar 26, 2024 •

edited

Loading

Uh oh!

kmilos commented Mar 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LebedevRI commented Mar 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kmilos commented Mar 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LebedevRI commented Mar 26, 2024 •

edited

Loading

codecov bot commented Mar 26, 2024 •

edited

Loading