Neo Geo Pocket and GBC battery mod?

Rechargeable battery is basically the same in concept whatever device you are doing. Voltage is voltage and as long as it delivers the amps (which if you are comparing to anything previously powered by AA disposable batteries is no big ask) then you are good. Too much voltage and you blow stuff up, too little and it won't power.
The device might have rated power limits, though for batteries then the simple start is take 1.5V for each AA cell and multiply it up to how many there are. Now AA cells don't stay at 1.5V throughout their runtime and drop quite quickly ( https://data.energizer.com/PDFs/E91.pdf ) so the device will then have a cutoff voltage below which I won't work. This is your range you want to hit, though you might be able to go a bit higher at the risk of maybe cooking the device in the longer term.
If you have an external power supply (preferably a nice bench supply) then you can figure out the voltage.
You might also want to know how much current it draws (naturally high action game/flash cart, max volume, bright screen...) while it does it so if you have an ammeter/current sensor on your supply or a multimeter the find that out too; when batteries on the seller have nice little mAh rating that means how many milliamps it will supply for an hour, simple division to then get the upper limit on runtime as practically it will be a bit lower. You then get to choose how much to spend, what to sacrifice to make it (and whatever management it needs) fit, what you want for a lifetime (having it last a week of solid play is great and all but if you are rarely more than a few metres from a wall socket then... yeah) and other such things. Management is necessary for lithium based batteries, though some might have their own onboard, but they do provide the most power for the size.
Hopefully the battery itself provides the voltage you want but if not then you get to figure out how to make it the right one.
Stepping down is usually no big deal -- resistors, diodes, zener diodes, boost convertors, regulators... all potential options here with varying sizes, costs, efficiencies and the like.
Stepping up with DC voltage is harder. Boost convertors however have got pretty good in recent years so many will opt for that. You can also series wind batteries (makes management harder but still done all the time -- if you have ever used a battery powered hand tool then it will likely be this, also 95% of the time when the battery pack dies it is only one or two cells within it that have died because management hard) and get to the required or step down again if you want to go that path.

Ultimately then you fit rechargeable battery plus any necessary management in along with charger and any voltage management if you need to step it up or down to get to a suitable voltage. Fitting it and getting the charger port in there then tends to be the hard part and limits the size of battery you can wedge in there, or dictates how much of the case you slice out to get that bigger battery in there.
If the device has external power source that might charge an original battery then either put a diode in there to stop poorly handled reverse charging, figure out how to embrace it (free power and probably in a place that is nice to plug in on the original device), delete it (if the wires going to the charge port are cut...) or otherwise make sure some user you made promise up, down, sideways and on their pets that they would not do it do not in turn burn their house down when "I just thought".
In some newer devices you can get particularly unlucky and there will be a nice secret handshake that happens when you put the battery in and the management chip has a chat with the device proper. If so replicate this, disable this or otherwise cannibalise an existing chip (and hope it does not have a built in "I have been charged 1000 times and now I am declaring myself dead despite being nowhere close because marketing needs a "user has a too fat wallet and we want some of that'" deal, or sort that too).

 

Power draw is a thing you can consider but most times basic handheld computer plus screen ends up similar enough regardless of decade* and not going to be outrageous to the point where modern lithium struggles -- not like you are trying to heat up anything, run a motor or anything like that which can be truly different and properly suck down the juice. Or if you prefer the basic AA battery that we used throughout most of that and still used up through the xbox 360 is going to be the same zinc and carbon that your grandparents likely used and still have about the same ability to push out power and people are still as likely to get upset when their device did not last a day/week/whatever/compare it to the competition so designers of devices played to that.

*I am sure today if you gave me a few million I could probably have a something that is functionally identical to a gameboy but running modern versions of the same chips and have it run for multiple times longer on a coin cell but eh.

To that end it really is

Figure out voltage it needs or at least get one that works and match it. Matching it might see you need to drop voltage or boost it up but that is usually easy enough and you buy a device that outputs what you want. video series if you like videos about things
Get battery (can be standalone, can be replacement from something else)
Battery needs to have sweet nothings whispered to it during charging** so make sure you also have a management controller somewhere in the mix.

Cram it all into whatever space there is. More advanced types might extend the shell out or cut away excess plastic to allow it to fit a bigger battery in there.

**if you are bored



If the thing already has a rechargeable battery and you want to do a modern replacement (either because higher capacity or because good replacements are rarer than shit from a rocking horse) then the "has a special chip that speaks to the device" and "might use a different charging method on the onboard charger you then get to work around" can kick in.