After porting Fast-FoundationStereo to QNN HTP on QCS8550 / Snapdragon 8 Gen 2, the model can run correctly on the board, but the performance is much lower than expected. The current FP32 latency is about 1.71 s/frame.
From the profiling results, the bottleneck is not simply whether the operators are offloaded to HTP. The main issue seems to be that the graph structure is not HTP-friendly. In particular, _Mul_1 + ReduceSum, which comes from the stereo correlation / einsum computation, is expanded into a huge 6D intermediate tensor. The single _Mul_1 operator alone accounts for roughly one third of the total cycles.
At the same time, operators such as GridSample, Transpose / layout transform, Add, and Resize also introduce significant DDR traffic and VTCM pressure. I have tried several optimization approaches, including layout reordering, splitting Mul / ReduceSum, rewriting GridSample, using a CorrDot custom op, and delaying transpose operations. Most of these versions can run successfully, but they do not significantly reduce the overall latency. Some custom op attempts even make the model slower.
I would like to ask: for this kind of stereo cost volume / correlation large-tensor computation on QNN HTP v73, are there more effective optimization approaches? For example:
fused correlation / dot-product custom op
HVX / HTP vectorized implementation
mixed INT8 quantization
reducing layout transforms
reducing DDR data movement
other graph rewrites that are more suitable for QCS8550
The goal is to reduce the single-frame inference latency on the board as much as possible.
After porting Fast-FoundationStereo to QNN HTP on QCS8550 / Snapdragon 8 Gen 2, the model can run correctly on the board, but the performance is much lower than expected. The current FP32 latency is about 1.71 s/frame.
From the profiling results, the bottleneck is not simply whether the operators are offloaded to HTP. The main issue seems to be that the graph structure is not HTP-friendly. In particular, _Mul_1 + ReduceSum, which comes from the stereo correlation / einsum computation, is expanded into a huge 6D intermediate tensor. The single _Mul_1 operator alone accounts for roughly one third of the total cycles.
At the same time, operators such as GridSample, Transpose / layout transform, Add, and Resize also introduce significant DDR traffic and VTCM pressure. I have tried several optimization approaches, including layout reordering, splitting Mul / ReduceSum, rewriting GridSample, using a CorrDot custom op, and delaying transpose operations. Most of these versions can run successfully, but they do not significantly reduce the overall latency. Some custom op attempts even make the model slower.
I would like to ask: for this kind of stereo cost volume / correlation large-tensor computation on QNN HTP v73, are there more effective optimization approaches? For example:
fused correlation / dot-product custom op
HVX / HTP vectorized implementation
mixed INT8 quantization
reducing layout transforms
reducing DDR data movement
other graph rewrites that are more suitable for QCS8550
The goal is to reduce the single-frame inference latency on the board as much as possible.