Coherent engineering of landscape potential in crystalline materials is a rapidly evolving research field. Ultrafast optical pulses can manipulate low-frequency shear phonons in van der Waals layered materials through the dynamical dressing of electronic structure and photoexcited carrier density. In this work, we provide a diagrammatic formalism for nonlinear Raman force and implement it to shear phonon dynamics in bilayer graphene. We predict a controllable splitting of double degenerate shear phonon modes due to light-induced phonon mixing and renormalization according to a coherent nonlinear Raman force mechanism. Intriguingly, we obtain a light-induced shear phonon softening that facilitates structural instability at a critical field amplitude for which the shear phonon frequency vanishes. The phonon splitting and instability strongly depend on the laser intensity, frequency, chemical potential, and temperature of photoexcited electrons. This study motivates future experimental investigation of the optical fine tuning and regulation of shear phonons and layer stacking order in layered van der Waals materials.