Behavioural lateralization, the asymmetric expression of cognitive functions, is reported to enhance key fitness-relevant traits such as group coordination, multitasking and predator escape. Therefore, studies reporting negative effects on lateralization in fish due to environmental stressors such as ocean acidification, hypoxia and pollutants are worrisome. However, such studies tend to use a detour test and focus on population level measures, without validating whether lateralization is consistent within individuals across time. We conducted a multispecies, international assessment of the repeatability (R) of lateralization in four previously studied fish species using a detour test (T-maze), a common method for testing lateralization. We also reanalysed a published data set on a fifth species using new statistical methods. We expected the three shoaling species to exhibit greater within-individual consistency in lateralization than their nonshoaling counterparts given previous reports of stronger lateralization in group-living fishes. Absolute and relative lateralization scores were highly nonrepeatable in all five species (0.01<R<0.08), irrespective of their shoaling status. We carefully reviewed 31 published studies in which the detour test was employed to examine lateralization in fish and identified statistical issues in all of them. We develop and propose new statistical analyses to test for population and individual level lateralization. The commonly used detour test does not appear to be appropriate for quantifying behavioural lateralization in fishes, calling into question functional inferences drawn by many published studies, including our own. Potential fitness benefits of lateralization and anthropogenic effects on lateralization as a proxy for adaptive brain functioning need to be assessed with alternative paradigms.