It's been done before. Can't post the link to it cuz I'm still a new member, but I can come back to post it lol.
The guy used Nintendo Switch speakers and the PAM8403 stereo amp board. Looks like he didn't wire a shutdown for the amp though so the speaker would still run with headphones plugged in.
I know if you wire A- of the PAM8302A mono amp to pin 5 of the headphone jack it effectively acts like a shutdown for headphones (don't know the exact details why that works), so maybe something similar could be done here?