It should be possible to do. The first question is whether your headphones can handle the increased voltage of a LiPo,
Common alkaline type AAA cells start off at around 1.6v when fresh and the voltage declines from there. So with 2 fresh cells in series your headphones will be seeing at most about 3.2v. And somewhat less than 3v for most of the battery life.
A freshly charged LiPo will be at 4.2v and will decline to about 3.7v for most of the discharge cycle.
Assuming that your phones could handle up to 4.2v, you could just connect the '-' wire to the GND pin and the '+' wire to the BAT pin.
If you want to limit the voltage to something closer to the alkaline cells, you could wire a diode in series. A 1n4001 diode will have about a 0.8v drop. That will limit you to about 3.4v max and around 2.9v for most of the discharge cycle.
https://www.adafruit.com/product/755