Somewhat curious that removing the battery makes it work again, though it would not be the first thing to have the dead battery gobble up all the power and stop everything else from working (had a laptop the other year actually do this, annoyingly the battery be glued to the case and needing massive disassembly). Hopefully it is a snip and go and no need to fake out the battery.
If the original company is not around and selling viable replacements (be wary of things that sat on shelves for 10 years), and you can't recover the battery (if this is actual death and not some deep discharge or detected consumer's wallet too fat error* you can claw it back from) then you have a few options, none of which are ideal.
As this is an unknown controller company then bets are not good on finding a replacement, but I have certainly been surprised before so if you can throw an email at their customer service or engineering department then you might get something. You might also be lucky and find a newer model with a compatible battery (if you can reuse a design then you tend to) that is still sold. If you have not got the controller apart and battery out, any numbers you can find on it or similar already then do that before going too much further.
*many laptops and fancy tools will have a little EEPROM on it to note how many times it has been charged, various things then seen to query this chip or stop working after so many charges regardless of what state the thing is in. The PSP also notably used this to activate recovery mode.
1) You find the chemistry out and have a custom replacement made.
2) You find the chemistry out and a replacement that will fit from some other device.
3) You adapt the device by various means to work with a larger battery, newer chemistry or something that is an improvement. Can be quite difficult.
1) Sadly seems to be slow in the coming. Was hoping it would have been here by now, hopefully in a few years as the principle is not that hard and should be as doable as making your own custom circuit board (as in send file to factory along with funds, it slices and wraps one up with pins in appropriate place, tests/charges and sends back to you) but oh well.
2) There are million devices with common batteries. Get something that fits in the gap (or a gap you make larger) of the same/compatible chemistry and throw wires around as necessary -- positive, negative, any sense setup (the third pin can be things like information on the battery or a simple thermistor is uses to help charging) can probably be bypassed or even catered for.
3) Rechargeable battery charging is a field unto itself
This is why you might struggle, or at least find your shiny new battery dying in short order, if you don't handle the charging aspect and they vary a bit.
You can get adapters/standalone charge controllers and bypass things though (maybe at the cost of any internal sensing) -- if the thing takes USB to charge then tapping the USB in and skipping all/deleting the internal charging might be an option. Said adapters/controllers take space though which means less battery capacity, or finding space for it in the shell which is probably easier said than done.
If you are really fancy, or maybe the thing is really primitive, then you can upgrade to a more modern chemistry as well but if you know how to do that then you know everything I already mentioned and much more besides.