If it flashed orange for a second or two then that is one thing and usually speaks to an error in the charging system, if it happens for minutes then the fault is more likely with the replacement battery. In this case one of the cells in series to make up the battery likely failed so it shows as charged in short order but does not have the power to give out for more than a few seconds. There is no reconditioning or replacement or individual cells here like there is for some tools and laptops using 18650s and such. Time for a new battery, I don't know what we are suggesting for SP replacements these days -- anything official has likely sat on the shelf for more than a decade and is long since dead/so far into deep discharge and that leaves either recovered ones from second hand devices (gamble if ever there was one) or third party (even larger gamble most of the time), not much in the way of custom or adaptation I have seen yet either.
In some cases it was found that the GBA charge circuit detection is not so hot and you can do things like charge, run it down (in your case probably with a suitably high value resistor across some contacts) and charge again to have it work better next charge but this does not seem like one of those.