Aims. The fine-structure of the magnetic field in a sunspot penumbra in the upper chromosphere is to be explored and compared to that in the photosphere. Methods. Spectropolarimetric observations with high spatial resolution were recorded with the 1.5-m GREGOR telescope using the GREGOR Infrared Spectrograph (GRIS). The observed spectral domain includes the upper chromospheric Hei triplet at 10 830 angstrom and the photospheric Si I 10 827.1 angstrom and Ca I 10 833.4 angstrom spectral lines. The upper chromospheric magnetic field is obtained by inverting the Hei triplet assuming a Milne-Eddington-type model atmosphere. A height-dependent inversion was applied to the Si I 10 827.1 angstrom and Ca I 10 833.4 angstrom lines to obtain the photospheric magnetic field. Results. We find that the inclination of the magnetic field varies in the azimuthal direction in the photosphere and in the upper chromosphere. The chromospheric variations coincide remarkably well with the variations in the inclination of the photospheric field and resemble the well-known spine and interspine structure in the photospheric layers of penumbrae. The typical peak-to-peak variations in the inclination of the magnetic field in the upper chromosphere are found to be 10 degrees-15 degrees, which is roughly half the variation in the photosphere. In contrast, the magnetic field strength of the observed penumbra does not vary on small spatial scales in the upper chromosphere. Conclusions. Thanks to the high spatial resolution of the observations that is possible with the GREGOR telescope at 1.08 microns, we find that the prominent small-scale fluctuations in the magnetic field inclination, which are a salient part of the property of sunspot penumbral photospheres, also persist in the chromosphere, although at somewhat reduced amplitudes. Such a complex magnetic configuration may facilitate penumbral chromospheric dynamic phenomena, such as penumbral micro-jets or transient bright dots.