I tested yours again on my REBUG 4.86.1 LITE PS3. It still boots back to the XMB.
On the rpcs3 that I am using it has OFW 4.86 so maybe there is something different. Then again things can sometimes work in emulators where they would freeze on a real system due to un-aligned code or something similar.
Part of the rpcs3 log (I removed my weak PC specs from the log) with your .SELF file:
RPCS3 v0.0.5-6860-4576a31f4 Alpha | HEAD
S {PPU[0x1000000] Thread (main_thread)} SPU: SPU Recompiler Runtime (ASMJIT) initialized...
S LDR: Boot successful.
U {PPU[0x1000000] Thread (main_thread) [0x0061fe64]} sys_prx TODO: _sys_prx_register_module()
U {PPU[0x1000000] Thread (main_thread) [0x006200bc]} sys_prx TODO: _sys_prx_register_library(library=*0x6226a0)
S {PPU[0x1000000] Thread (main_thread) [0x006209ac]} sys_prx: Loaded module: /dev_flash/sys/external/libsysmodule.sprx
U {PPU[0x1000000] Thread (main_thread) [0x00663d8c]} HLE TODO: Unimplemented syscall sys_ss_access_control_engine -> CELL_OK
S {PPU[0x1000000] Thread (main_thread) [0x006214d0]} sys_prx: Loaded module: /dev_flash/sys/external/libgcm_sys.sprx
S {PPU[0x1000000] Thread (main_thread) [0x006214d0]} sys_prx: Loaded module: /dev_flash/sys/external/libsre.sprx
S {PPU[0x1000000] Thread (main_thread) [0x006214d0]} sys_prx: Loaded module: /dev_flash/sys/external/libfs.sprx
U {PPU[0x1000000] Thread (main_thread) [0x0062061c]} sys_prx TODO: _sys_prx_register_module()
E {PPU[0x1000000] Thread (main_thread) [0x004eb0ec]} sys_mmapper: sys_mmapper_allocate_address(size=0x10000000, flags=0x40f, alignment=0x0, alloc_addr=*0x116d3bc0)
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data.rsdk” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644) x2
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/420701D3A57A564031790EE4228C7BDE.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/AD8ECA1EB8CDCB6CEAA5CE2E361EF48E.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/38021520621EDBEBB1E1B2FE4423535F.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/B5238FBBD2653E8506337C21FACF1A88.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644) x70
E {rsx::thread} RSX: OpenGL: initialization of glGetTextureImage failed.
E {rsx::thread} RSX: OpenGL: initialization of glTextureSubImage1D failed.
E {rsx::thread} RSX: OpenGL: initialization of glTextureSubImage2D failed.
E {rsx::thread} RSX: OpenGL: initialization of glTextureSubImage3D failed.
E {rsx::thread} RSX: OpenGL: initialization of glTextureBufferRange failed.
E {rsx::thread} RSX: OpenGL: initialization of glTextureBarrier failed.
U {PPU[0x1000000] Thread (main_thread) [0x005f0830]} cellSysutil TODO: cellVideoOutConfigure(videoOut=0, config=*0x54cbac00, option=*0x0, waitForEvent=1)
U {PPU[0x1000000] Thread (main_thread) [0x005f0830]} cellSysutil TODO: cellVideoOutConfigure(videoOut=0, config=*0x100367a0, option=*0x0, waitForEvent=0)
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {rsx::thread} RSX: NV3089_IMAGE_IN_SIZE: Invalid blit dimensions passed
E {PPU[0x1000000] Thread (main_thread) [0x004ebc54]} 'sys_fs_fstat' failed with 0x8001002a : CELL_EBADF [1]
E {PPU[0x1000000] Thread (main_thread) [0x005f0a48]} Failed to bind device Joystick #0 to handler MMJoystick
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644) x3
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/870F0B5491E4D7EE7EA03F7F7A3D93CC.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644)
E {PPU[0x1000000] Thread (main_thread) [0x004eb9fc]} 'sys_fs_open' failed with 0x80010006 : CELL_ENOENT, “/dev_usb000/SonicMania/Data/Objects/Static/C626F7AC50103D2F03D73ADC4EA5EC0B.bin” [1]
U {PPU[0x1000000] Thread (main_thread) [0x004eba4c]} sys_fs TODO: sys_fs_chmod(path=“/dev_usb000/SonicMania/log.txt”, mode=0644) x6