It's been done before. Can't post the link to it cuz I'm still a new member, but I can come back to post it lol.
The guy used Nintendo Switch speakers and the PAM8403 stereo amp board. Looks like he didn't wire a shutdown for the amp though so the speaker would still run with headphones plugged...