Paper argues compliance data is often misused as evaluation data for AI systems

A new arXiv paper claims that a common mistake in assessing deployed AI systems is treating data gathered for operational monitoring or regulatory compliance as though it were collected for comparative evaluation. Using automated driving as its main example, the work calls for clearer measurement validity standards so that compliance-oriented datasets are not used to make comparative performance claims. The authors frame this as a recurring evaluation failure rather than an isolated incident.