From what I heard you can update it, just not past a certain firmware (9.0+ I believe, so any n3DS-aware versions will not work)
It might be possible to transfer from a n3DS on a <=8.x o3DS firmware and an o3DS, but that's not useful for OP since their o3DS is on 11.4, and system transfer cannot work between two different versions of the system transfer app.
Edit: Also, o3DS emuNAND on a n3DS was never figured out, was it? That 2.1 ctrtransfer only works on sysNAND.
It would be a useful feature to have though.