When it starts flashing red, usually. Sometimes I set it to charge earlier (for example, if I'm going to be going somewhere with it later and it's on one bar), but I just as frequently bring my charger with me.
It's how I've been doing things for five years with my aqua blue o3DS and I haven't noticed a significant decrease in battery life in all that time.