FPTQuant end-to-end training

[phiahr]

I'm trying to recreate the results of the FPTQuant paper but the results I get using my own end-to-end training are different (a lot worse).

So my questions are:

1. Is there an official or reference implementation of FPTQuant (or AIMET configuration) end-to-end training (not only the local optimization) that reproduces the results reported in the paper?
2. Since MultiHeadValueTransformOp is not explicitly made to be invertible in AIMET’s implementation, but should be invertible, is it expected that a proper end-to-end training will never result in a singular transform or can that be caused through specific models or hyperparameters?
   (in my end-to-end training the MultiHeadValue transform becomes non-invertible during training with the current implementation)
3. Could the degradation and non-invertible transform I’m seeing be due to applying this method to a smaller model (e.g., LLaMA 160M), or should the approach still work in this regime?

Any guidance on these points would be very helpful for understanding whether this is a usage issue or a limitation in the current implementation.

[quic-kyunggeu]

Hi @phiahr, thanks for raising this issue. Unfortunately, we don't have an official example of FPTQuant end-to-end training that reproduces the results in the paper. FPTQuant implementation in AIMET is an experimental feature that was added for internal experiment and has not been publicly released. We might release it as a public feature later if we find a crucial use case, but otherwise it will remain internal/experimental.