We investigated how operationalizing bilingualism affects the results on a Simon task in a population of monolingual and bilingual native English speakers (N = 166). Bilingualism was measured in different ways within participants, and the measurements were used both as dichotomous and continuous variables. Our results show that the statistical significance and effect size varied across operationalizations. Specifically, the Composite Factor Score (the Language and Social Background Questionnaire’s general score), showed a bilingual disadvantage on reaction times regardless of how it was used (dichotomously or continuously). When dividing participants into monolinguals and bilinguals based on the Nonnative Language Social Use score (a Language and Social Background Questionnaire subscore), differences in accuracy and reaction times were found between the groups, but the Nonnative Language Social Use score did not predict accuracy when used as a continuous variable (only reaction times). Finally, earlier age of acquisition predicted faster reaction times, but only when used on a continuum. Effect sizes were between the small and medium range. No differences on the Simon effect were found. Our results call for cautiousness when comparing studies using different types of measurements, highlight the need for clarity and transparency when describing samples, and stresses the need for more research on the operationalization of bilingualism.