Charged-particle fragmentation functions for jets azimuthally balanced by a high-transverse-momentum, prompt, isolated photon are measured in 25 pb(-1) of pp and 0.49 nb(-1) of Pb + Pb collision data at 5.02 TeV per nucleon pair recorded with the ATLAS detector at the Large Hadron Collider. The measurements are compared to predictions of Monte Carlo generators and to measurements of inclusively selected jets. In pp collisions, a different jet fragmentation function in photon-tagged events from that in inclusive jet events arises from the difference in fragmentation between light quarks and gluons. The ratios of the fragmentation functions in Pb + Pb events to that in pp events are used to explore the parton color-charge dependence of jet quenching in the hot medium. In relatively peripheral collisions, fragmentation functions exhibit a similar modification pattern for photon-tagged and inclusive jets. However, photon-tagged jets are observed to have larger modifications than inclusive jets in central Pb + Pb events.