Research using ultracold atomic ensembles in high-finesse cavities enabled early demonstrations of optomechanical dynamics in the quantum regime. The emerging field of cavity optomagnonics has demonstrated coherent interactions between optical polarization and excitations of macroscopic spin ensembles, analogous to optomechanical systems, introducing novel spin dynamics and sensitivity to magnetic fields. Optical coupling to our atomic ensemble’s collective spin provides an ideal system to explore this new area of cavity optodynamics, and can be realized simultaneously with optomechanical interactions. In this talk, I will describe our recent work demonstrating a negative-mass instability, arising from a cavity-mediated, collective spin-orbit coupling in an atomic ensemble. The instability arises due to coherent coupling between the spin precession of a high-energy spin ensemble, which behaves like an effective negative-mass oscillator, and the atomic center-of-mass motion, resulting in a resonant pair-creation interaction which drives amplification and correlation of both modes. This interaction should facilitate quantum-limited amplification, driving the system into a two-mode squeezed state, but is obscured by technical noise in our apparatus.

I will further describe measurement and analysis techniques to infer squeezing in such a two-mode system, as well as ongoing work within spin optodynamics.