This uses handbrake

How it works is the conversion settings are the same as a YTS/Yiffy movie from 2017 (but with better audio, same quality as youtube audio actually). For YouTube, it does use ffmpeg to merge video+audio streams, but nothing else. FFmpeg converted videos simply do not work on the 360 media player, it hates them.
As for what your talking about, it's the same with H.264 video (which is what this converts to), you need the optional media update. That is included in the portable releases of Xbox 360 media center, and also available on the repo itself as well as on digiex. The optional media update mentioned previously has been modified to work on rgh/jtag back in the day, but it works on bad update consoles just fine after the console is in an exploited state. There are instructions on install it also on the homepage. Of course you can also do everything with a stock 360+live+installing the optional media update the official way as well. This just generates the media files.
What I've been doing is using Linux on my Mac mini Late 2012 to run a media sharing server. Gnome has this built in and can be enabled in the sharing section of the settings app. Then I ssh into the mac mini with my macbook air, cd into ~/Videos, and then run the command with a pasted youtube url. Then you get a downloaded youtube video in the highest resolution available that is playable on 360 (usually 1080p, but if only 720p is available you get that, etc.). Then I can start the video player, navigate to the video, and play it. When you download another video like this over ssh you need to exit out to the 'root' of the video player where it asks if you want to navigate to hdd, network server, usb, etc. Then go back into the network and it's there (with cover art icons and everything). Keeps your place in videos too. It's even better then the official YouTube app that requires live from what I can tell and haven't gotten to work with badupdate (that never did more then 720p iirc back in the day).
The server software in gnome linux is rygel, which I think is sorta buggy so sometimes you'll need to restart it for the 360 to be able to play anything after a lot of use. But besides that this has been a dream, I don't even need to copy files to it really but if you had like an unstable network connection you could just ftp to a USB or HDD if your console isn't stock with aurora. 360 can play 10Mb/s 1080p 30fps no problems which looks great on my 4k tv so why would I not use it. The files this gets you (besides the youtube high bitrate option) don't really go over 3Mb/s tho which is good for movies so that they don't go over the hardcoded 4GB filesize limit of the 360 video player.
I've also watched American Horror Story Season 1 on this, recursively converted the whole season in one command. Besides that, the old YTS/Yiffy releases work too just like back in the day with no conversion if you can find them. The newer ones (no idea of the cut off date) that contain the 5.1 sound instead of stereo need to be ran through the converter, as well as anything else basically just won't work with the usual error codes until you run them through x360mc.
There is a 'faster' way I could 'fix' the newer YTS/Yiffy releases with 5.1 sound by only converting the audio to stereo in theory, but that was out of scope (like the windows release) as a lot more work in regards to the portable releases/setup/build system and youtube features was required before v1.0 made it out. This seems really simple but it's not