Paper proposes setting inference depth per deployment from its expected questions

A new arXiv paper argues that although a transformer model is trained to handle any prompt, each real-world deployment only receives a narrow slice of queries, such as delivery complaints for a support assistant or Python for a coding tool. Yet every deployment currently pays the same inference compute cost. The authors propose choosing inference depth based on the questions a given deployment actually asks, rather than applying one uniform setting across all uses.