I'm assuming then that you didn't create a directory in c: called ctrtool [i.e. 'c:\ctrtool'] I just gave that as an example because it doesn't involve much typing, where did you extract ctrtool to? that is where you need to direct it.
At a guess you are trying to save 'romfs.bin' to a system directory, save it somewhere else like 'c:\' and see if it works, I just gave 3dsExplorer a shot and it worked fine for me.
Is that error a RAM or a 4gb limit thing? not sure, alternatively you could download this version of braindump and use it instead. This just straight dumps the exefs.bin and romfs.bin rather than creating a cxi.
http://www.mediafire.com/download/pd6dwq9lg6dg26o/Braindump.rar