This error most likely means it can't find one of the accessory programs it needs. Did you extract the whole zip including the progfiles folder? If you haven't installed a zip program, Windows likes to open zip files as if they're folders. But then if you run an exe that's inside the zip, it extracts just the exe to a temp folder.
Make sure you have the progfiles folder from the zip in the same folder as agb_edit.exe. Progfiles should contain 3dstool.exe, ctrtool.exe and makerom.exe.
If that doesn't fix it for you, please paste the couple of lines above the error, since those will say what it can't find.