Co-occurring psychiatric comorbidity is high among problem gamblers, and anxiety disorders has repeatedly been linked to problem gambling. Less conclusive, however, is the association between problem gambling and specific anxiety disorders. The aim of this study is to examine the association between problem gambling and specific anxiety disorders in subgroups of gender, age and socio-economic status (SES) in the general Swedish population. A case-control design was employed - nested in the Swedish longitudinal gambling study cohort. All anxiety disorders studied - Panic Disorder, Social Phobia, Generalized Anxiety Disorder (GAD) and Post-Traumatic Stress Disorder (PTSD), were significantly associated with problem gambling, however the pattern differed across subgroups. Social Phobia was the anxiety disorder most commonly associated with problem gambling across subgroups. The strongest associations between problem gambling and various anxiety disorders were found in participants under the age of 25, among females, and in the group with middle SES. In those groups three of the four anxiety disorders studied were significantly associated with problem gambling, with different patterns. Quite remarkably, participants under the age of 25 had three times higher risk of having had GAD compared to their controls. Efforts to prevent an escalation of either gambling or anxiety could target the presented vulnerable groups specifically.