Yes, you need to open the case (unless you're happy sanding down the buttons from outside the case, which would make sure they're reduced to the right level…). It's not too bad on the n3DS XL - the ABXY buttons are on a separate board that is quite straightforward to remove.
Replacing the Circle Pad with something larger would mean you couldn't close the system. Since the rubber cover came off the Circle Pad on my o3DS some years ago, I use plastic covers like this -
https://www.amazon.co.uk/Cyber-・Premium-Slide-Cover-Clear/dp/B00UURTXGK/ref=sr_1_29?keywords=3ds+circle+pad&qid=1581072079&s=videogames&sr=1-29 - better grip, gets less sweaty.