With a fresh start.
Make sure to have any extisting retroarch .cia's uninstalled. And remove the retroarch folder from your sd.
- Download the latest nightly .cia build,
here
Inside you'll find 2 folders; cia and retroarch
- Copy the Retroarch_3ds.cia, from the cia folder, to your SD. Anywhere.
- Copy the complete 'retroarch' folder to the root your SD. 'sdmc:/retroarch/'
- Install the Retroarch_3ds.cia.
- Try to load it.
Inside 'sdmc:/retroarch/cores/' you'll find all available cores. On first boot, the retroarch_3ds.cia, will try to load the first found core in the core folder. This usually would be 81_libretro.cia. If not installed, it will try to install it, and then load it.
If this doesn't work, try to install one of the .cia cores found in the cores folder, and launch it directly. It should generate the .cfg files, and then you can quit through the retroarch menu.
After which you should be able to start from the Retroarch_3ds.cia.
The .3dsx and .cia builds can be used together, BUT, once you've loaded a rom it's set to that *core*.3dsx or .cia.
If you switch and try to load the same rom again, you have to 'Reset Core Association', found in the menu after selecting the rom. Or it will give an error about the core not found.
It will lookup the available cores in the cores folder for the format you've launched. 3dsx. or .cia
edit: Also, the Retroarch_3ds.cia will try to load the last used core. If this core is a .3dsx or an outdated .cia, retroarch will crash. You'll have to load a .cia core directly before using the retroarch_3ds.cia again.
If this still doesn't work, you'll have to give us more info about the crash. And it's probably also not retroarch related it that case. Hope this will help.