The 3DS doesn't have any of fixed-function hardware the GBA/DS has (well, except the GBA sound FIFO + timer --> DS SND --> 3DS CSND), so your options are sadly limited and resource-intensive.
I can only list a few options from the top of my head:
- You literally draw over the entire scene with a transparent rectangle
(I do this in a GPU-composited program of mine)
- You can update the
PDC LUT (0x80 and 0x84) in VBlank region to dim all pixels, but unless you're doing this in baremetal, and/or working on a techdemo/demoscene, DO NOT DO THIS IN THINGS INTENDED TO BE USED BY EVERYONE, as it will break a LOT of stuff.
- You can allocate a dimming buffer, and use the ARM11 CPU's data- and media-processing instructions to do color multiply faster (needs 0xRRGGBBAA mode to work)
- You could dim the backlight perhaps? This method is not recommended either.
- You can turn off the LCD panels themselves via an MCU command, and they will slowly fade to black
(while this one is a joke, it is actually valid, but also not recommended either)
You can accelerate the "brute-force" methods by using Core1 (syscore) for low-impact tasks while Core0 (appcore) hammers over the pixels furiously. And let's not forget that the new3DS has two more cores, one dedicated to QTM, and the other is completely free for programmers to use. Although keep in mind the limited FCRAM bandwidth! Utilize your caches well!