Agent-based models (ABMs) often rely on psychometric constructs such as `opinions¿, `stubbornness¿, `happiness¿, etc. The measurement process for these constructs is quite different from the one used in physics as there is no standardized unit of measurement for opinion or happiness. Consequently, measurements are usually affected by `psychometric distortions,¿ which can substantially impact models¿ predictions. Even if distortions are well known in psychometrics, their existence and nature is obscure to many researchers outside this field. In this paper, we introduce distortions to the ABM community. Initially, we show where distortions come from and how to observe them in real-world data. We then show how they can strongly impact predictions, qualitative comparison with data and the problem they pose for validation of models. We conclude our analysis by discussing how researchers may mitigate this problem and highlight possible future modelling trends that will address this problem.