I've done this before, where I broke FCEUX GX (dont mess with video output options, PSA)
I'd do a fresh install of MAME again to be safe, and make sure to delete any config files it has, not just /apps/MAME and /MAME/. Even though it has these directories, it's possible it's still making config files on the root of the SD.
Also try running it off of another SD/USB if you can, that way you can narrow down if it's a problem with MAME or something else.