As far as I know, both gamepad and console shells should be physically identical. So, yeah, I believe that you can swap both cases in order to your 32GB variant in white. I remember seeing people from reddit buying refurbished white 32GB Wii Us.
Japan had a white 32gb, and I think they're cheaper than buying the black one elsewhere. That's why you see a lot of people trying to change the language from Japanese to English.
I own the AU model in both 8gb and 32gb, and while I haven't done a full case swap, I have mixed and matched their parts to make "panda"-styled Wii Us; some black and some white. All the parts I tried fit together just fine and I've had no issues.
I may end up doing a full swap though just because the white plastic seems so much more durable than the black gloss.