We study the migration behavior of the native Swedish population following refugee immigration, with a particular focus on examining whether there is support for an ethnically based migration response. Using rich geo-coded Swedish data, we account for possible endogeneity problems by combining policy-induced initial immigrant settlements with exogenous contemporaneous immigration as captured by refugee shocks. We find the same flight among all natives, irrespective of their parental foreign background. This suggests that ethnic distance to the new immigrants is not the dominant channel causing natives' flight behavior. Instead, refugee immigration seems to lead to more socio-economically segregated neighborhoods.