Using 3W Stereo Speaker Bonnet PID 3346 with Headphones

[MichaelB123]

I'd like to use the 3W Stereo Speaker Bonnet (PID 3346) with both speakers or headphones, controlling the desired option with a toggle switch to go from one to the other and adding in attenuators for headphone use as needed.

For speakers, the pair of outputs work fine. Adding series 500 ohm resistors to left (plus) and right (plus) outputs drops the level to a suitable and safe volume for my 24 ohm headphones. However, I ran into a problem due to the fact that headphones (unlike a pair of speakers) have a common ground for left and right drivers.

Tying the left (minus) and right (minus) outputs together and using that for the common ground to the phones makes a huge hissing sound. I thought that would work, but it doesn't. But if I use, for example, just the left (minus) output for the common ground, and leave the right (minus) output unconnected, the headphones work reasonably well, but there is still a faint hiss out of the side with the unconnected output.

What is going on here? Is there a solution that will allow me to use this bonnet with headphones properly?