Ok, took some time today to dig into it.
The XMB and games are rendered all the time at 1080p.
Tried changing all the settings within retroarch.cfg.
What I wanted to do is have native 720p within RetroArch, I'll explain why in a moment.
Tests I did:
Xbox set to 720p
Set fullscreen true, set fullscreen resolution: 1280x720.
Result 720p output and rendering at 1080p.
Set fullscreen to false, have windowed fullscreen to true and windowed resolution to 1280x720.
Results in 720p output and rendering at 1080p.
Tried with window scaled to x1, same results.
Tried with both fullscreen and windowed set to false, same results.
I guess this must be set before the compiling process and RetroArch devs choose the safe path, 1080p can be upscaled to 4k and downscaled to 720p.
Now the reason I want native 720p is that 720p is a multiple of 240p. That means 240p scaled to 720p by 3x will result in a frame taking the entire screen.
That means 320x240p games will fill the entire 4:3 aspect ratio, while 224p games, will have a small black border.
Also 4k is a multiple of 720 (3840x2160, multiply 720 by 3 and you have 2160).
This results in true square pixels in a 4k display.
1920x1080 isn't a multiple of anything related to old games. It's not a multiple of 240 and it's not a multiple of 224.
Now this is only a concern of people that want square pixels, shimering free experience (look at some background like the streets on strees of rage 2 while using non-integer resolutions and you will see the floor flickering, that sucks and this is caused by uneven pixels).
So, if guys are able to talk to some dev and ask for 720p rendering this would be really appreciated.