EFQ-Softmax Method Aims to Remove Exponentials from Attention Quantization
A new arXiv paper proposes EFQ-Softmax, a technique for quantizing the softmax step in Transformer attention without computing exponentials. Low-bit attention already speeds up inference by shifting matrix multiplications to FP8 or FP4 engines, but the softmax path often still needs higher precision for shifted-score exponentials and temporary probability storage. The work targets that remaining bottleneck in the attention pipeline.