Unless you're swapping out SD cards frequently, you shouldn't need to worry about durability. There's no reason to take it out of the system otherwise.
My Photofast adapter (which is probably a knock off) still works fine after 11 years. It has a crack in the top, and fell apart once a few years ago when I did need to re-seat the SD cards, but it still works once clicked into the PSP.
Basically, dont take it out unless you NEED to and it wont matter much. Since you can do all your transfers through USB, you shouldn't need to.