Homebrew Help me understand Read & Write speeds of the WiiU

[jacobsson]

Hi.
I would like to understand why some apps can achieve almost 15MB/s in write speeds while others are limited to crippling 120KB/s.

Example
Obs! I'm using MB/s as in megabytes per second in my measurements.

WUP installer: When installing from SD card to USB hardrive it works just fine, see here: ~100MB installed in 7s.
FTPiiu everywhere: transfer files to USB-hard drive is limited to 120KB/s

I've been doing some small tests regarding transfer speeds between the PC -> SD-card, NAND and WiiU USB formatted storage using FTP.
Here some results using ftpiiu everywhere:

Write speeds
FTP: PC -> SD = 1.25MB/s
FTP: PC -> NAND = 120KB/s
FTP: PC -> WiiU HDD = 120KB/s
WUP: SD -> WiiU HDD =14.2MB/s

Read speeds
FTP: SD -> PC = 1.25MB/s
FTP: NAND -> PC = 1.25MB/s
FTP: WiiU HDD -> PC =1.25MB/s

Observations:

• Writing to anything other than SD-card is ridiculously slow
• Reading anywhere from the WiiU back to the PC seems to achieve the same result of around 1.25MB/s
• Using WLAN or ethernet gives the same max speed of ~1.25MB/s

Some conclusions
To me it seems like the network interface is saturated around the speed of ~1.25MB/s and the overall write speeds seems severely crippled when using homebrew.
I took a look into the source code of WUP-installer but I didn't get any smarter

My question:
What makes the WUP installers transfer speeds so much faster compared to other homebrew like ftpiiu everywhere (using IOSUHAX)?

Feel free to add your own test results here so that we can discuss them.

 

[depaul]

Best luck @jacobsson. You are addressing the real problem here.
I'm not a dev. I think it might have something to do with homebrew development library.

Probably WUP installer using a faster I/O function whereas other homebrew like FTPiiU use slow I/O functions.

 

[jacobsson]

@depaul Thanks man. Let's hope that someone more knowledgeable than me can shed some lights on this!
I think this hasn't really been a big issue since we're so used manually moving files on the PC to the SD.

 

[Deleted User]

We can now unlock USB-Speed. However: It´s for testing right now. Only use it on your own risk. To unlock full USB-speeds, you have to manually change some code, before you compile a WiiU-file (such as RPL or RPX).
This unlocks your USB. In order to PROFIT from fast USN-writing & reading, an SSD is required, since HDDs are limited to max 500 iops.

 

[Deleted User]

Btw: Only if you circumvent the OS of WiiU, you can use full USB-Speed.

WUP installer doesn´t run in the OS of WiiU, so full speed is possible.
The unlocking of USB pretty much just circumvents the OS.
However: Unlocking Network-speed is a lot harder. The problem is because WiiU encrypts the whole OS. It´s a software-encryption. Software-encryption is allways slow.
When you run WUP installer, no data is encrypted = fast.
If you run anything in WiiU´s OS (ex.Retroarch), you encrypt it automatically & thus can´t use full speed.
Every RPX/RPL has to manually patched in order to circumvent software-encryption of WiiU´s "standard procedure".

 

[jacobsson]

Hi @TheChosen. First of all, thanks for replying! What you're saying makes a lot of sense to me.

Is there a guide how to manually patch my elf and rpx files? (PM is fine too if you don't want to share a premature solution publicly)

Thanks man!

 

[Deleted User]

There are some links which describe, how it´s done on "Loadiine" as an example to try it on (which is what i´m testing right now, using an Intel-SSD, i have everything ready). Theoretically it could work on any rpx/rpl-file if you change some setting and write "0" into each field where it says "256". However there´s a potential of harming your system.
It could e.g. work on Retroarch (if you use different code/rewrite the code), or on other homebrews & yes, even on linux.
However unlocking network (so that it works without any encryption) requires a custom network-library, so that´s unlikely for now.
So yeah: It simply means you better don´t rely on WiiU´s own library, or it´s locked to slow loading.
So you either use own written code to circumvent WiiU-OS, or you use other compatible, foreign code. Then you´ll get unlocked reading/writing speed.

 

[jacobsson]

Theoretically it could work on any rpx/rpl-file if you change some setting and write "0" into each field where it says "256". However there´s a potential of harming your system.
I

Thanks for the info, this sounds really promising! Any chance you'd would do some kind of step by step or link to cover procedure?
I'd really love to test it out!

 

[CMDreamer]

Two things:
1. Read speed will always be faster than write speed.
2. Wired connection speed will always be faster than wireless connection speed.

When comparing r/w speeds of two diferent devices, SD (card) < USB-FD (flash drive) < USB-HDD (hard-drive).

 

[depaul]

Hi. I've just tried the excellent tool Dumpling released here:
https://gbatemp.net/threads/release-dumpling.562741/

Trying to transfer saves data from USB to SD, speed can easily reach 12 MB/s.
https://ibb.co/n7CNztV

It might not seem much, but for Wii U it's better than the 1.25MB limitation we have in Retroarch and FTPiiU.

Maybe the author, @Lia, might know how to achieve such speed ? Maybe I'll try to ask them for help ;p

 

[Deleted member 381889]

I've managed to get over 20MB/s write speeds with my dumpling tool, mentioned above. It seems that FTP is just slow.

I got better speeds dumping to FAT32 USB rather than to SD, but this may vary.

 

[depaul]

Thanks . It's great since it reaches real USB 2.0 speed... maybe my 12 MB limitation is only due to my slow SD write speed (since I'm doing transfer Wii U USB to SD).

So maybe FT2SD has those fast I/O functions :
https://github.com/dimok789/ft2sd/tree/master/src
https://github.com/dimok789/ft2sd/blob/master/src/fs/fs_utils.c

int MountFS(void *pClient, void *pCmd, char **mount_path);
int UmountFS(void *pClient, void *pCmd, const char *mountPath);
int LoadFileToMem(const char *filepath, u8 **inbuffer, u32 *size);

 

[jacobsson]

Two things:
1. Read speed will always be faster than write speed.
2. Wired connection speed will always be faster than wireless connection speed.

If you look at the results of the original post you can see that this isn't the case. Something is very off with by how Nintendo implemented their network drivers. What weird is using ftp to write to nand /WiiU drive only gives 120KB/s, not even reaching the max network speed of 1.25MB/s.

 

[V10lator]

WUP installer doesn´t run in the OS of WiiU, so full speed is possible.

What are you even trying to say here? WUP Installer is a normal RPX, it gets loaded by the OS RPX loader. It uses OS libraries like most other homebrews do. In fact it uses a OS library (MCP) to install the content - there are no filesystem writing codes in WUP Installer itself...

WUP Installer or better the (Wii U OS) MCP library reaches top speeds of around 40 Mbit/s (average speed is at around 20 Mbit/s, so in line with the tool written by @Lia which, you guess it, also uses OS functions to read/write files.

max network speed of 1.25MB/s.

Max network speed is around 8 MB/s (Mbit/s) or ~1 Mb/s (Mbyte/s). Are you sure your measurings are in Mbit/s?

//EDIT: To be more precisely network speed seems to be around 4 Mbit/s per core, maxed at 8 Mbit/s. So multitasking is a good thing for everything using the network but I doubt FTPiiU is using more than core 1.

//EDIT²: FTPiiU does everything (UI, network I/O, filesystem I/O, ...) in one thread, that's why it's so slow. Anyway, if I see it right the tool from @Lia is doing the same iosuhax mounts FTPiiU does but it's written against modern WUT. So maybe I'll find the time to use this to update FTPiiU to newest WUT and after that look at multitasking or other speed improvements. If any other dev wants to do that, too: Just do it as I'm not sure if I'll find the time for it.

 

[jacobsson]

What are you even trying to say here? WUP Installer is a normal RPX, it gets loaded by the OS RPX loader. It uses OS libraries like most other homebrews do. In fact it uses a OS library (MCP) to install the content - there are no filesystem writing codes in WUP Installer itself...

WUP Installer or better the (Wii U OS) MCP library reaches top speeds of around 40 Mbit/s (average speed is at around 20 Mbit/s, so in line with the tool written by @Lia which, you guess it, also uses OS functions to read/write files.



Max network speed is around 8 MB/s (Mbit/s) or ~1 Mb/s (Mbyte/s). Are you sure your measurings are in Mbit/s?

//EDIT: To be more precisely network speed seems to be around 4 Mbit/s per core, maxed at 8 Mbit/s. So multitasking is a good thing for everything using the network but I doubt FTPiiU is using more than core 1.

//EDIT²: FTPiiU does everything (UI, network I/O, filesystem I/O, ...) in one thread, that's why it's so slow. Anyway, if I see it right the tool from @Lia is doing the same iosuhax mounts FTPiiU does but it's written against modern WUT. So maybe I'll find the time to use this to update FTPiiU to newest WUT and after that look at multitasking or other speed improvements. If any other dev wants to do that, too: Just do it as I'm not sure if I'll find the time for it.

It's important to address that the use of MB stands for megabyte (hence capital B), times eight would give you megabits, makes comparisons with the WUP installer example easier. I should probably clarify this in the first post.

So I'm using kilobytes and megabytes in my measurement. 1.25 MegaBytes per second (which seems to support your claim of around 8-10Mbit/s) seems to saturate they network interface of the ftp everywhere.

Your point about lack of multi core implementation in ftpiiu is very interesting!

Do you have any guess why FTPing to Nand and WiiU formated storage only gives 120KB/s (around 1Mbit/s) compared to the SD cards 1.25MB/s (10Mbit/s)? Seems really odd to me.

Thanks for contributing to the discussion.

 

[jacobsson]

I've managed to get over 20MB/s write speeds with my dumpling tool, mentioned above. It seems that FTP is just slow.

I got better speeds dumping to FAT32 USB rather than to SD, but this may vary.

I think you're totally right about that. It seems like the network interface only allows for ~10Mbit/s (1.25MB/s) transfer speed to the SD card. What I don't understand is why it only gets 1Mbit/s writing to NAND and WiiU formatted USB.

Could it be that using 1 core (single thread) to write to encrypted drivers is too much workload vs the unencrypted SD?

RetroArch is also suffering from extremely slow write speeds when using WiiU formatted HDD, I have not had the pleasure to look at you Dumplings tool yet, but you obviously achieves very good read/write speeds between the different storage medias.

EDIT: This commit message made me laugh! (love eating dumplings, great name! )

 

[V10lator]

It's important to address that the use of MB stands for megabyte (hence capital B), times eight would give you megabits, makes comparisons with the WUP installer example easier. I should probably clarify this in the first post.

So I'm using kilobytes and megabytes in my measurement. 1.25 MegaBytes per second (which seems to support your claim of around 8-10Mbit/s) seems to saturate they network interface of the ftp everywhere.

Thanks for clarification.

Do you have any guess why FTPing to Nand and WiiU formated storage only gives 120KB/s (around 1Mbit/s) compared to the SD cards 1.25MB/s (10Mbit/s)? Seems really odd to me.

No but it would be great to adress this.

Could it be that using 1 core (single thread) to write to encrypted drivers is too much workload vs the unencrypted SD?

You might have answered your own question here. That would also explain why I see MCP installing to a USB drive with just around 20-50 Mbit/s while Lias tool reaches crazy 20 MB/s with a FAT32 USB drive. Still faster than FTPiiU through but using MCP involves multitasking (the function WUP Installer uses is called MCP_InstallTitleAsync() - what I just said is not 100% correct but close enough).

I have not had the pleasure to look at you Dumplings tool yet, but you obviously achieves very good read/write speeds between the different storage medias.

Comparing it with FTPiiU might really reveal some other bottlenecks of FTPiiU but I also didn't find the time yet.

//EDIT: I'm currently in the process of writing a download tool and while doing so I added download to Wii U formatted USB (thanks to @Lia, looking at your sources helped a lot). Now I compared singlethreaded download to SD (via POSIX filesystem API, direct path access, so /vol/external01/...) with singlethreaded download to USB (via libiosuhax + POSIX filesystem API, indirect path access, so usb:/...) with the result that the USB download is maxed at around 1,3 Mbit(s) (some rare spikes up to around 2 Mbit/s) while the SD downloads are maxed at around 4 Mbit/s.

//EDIT²: Average download speed to USB is around 600 Kbit/s. I guess encryption has a lot do do with that (and maybe the fact that it's indirect writing via libiosuhax / IOCTLing data to hacked MCP).

//EDIT³: Looking at libiosuhax sources I don't think multitasking will help as it uses locking to feed the data to the hacked MCP thread. So data access is singlethreaded with libiosuhax anyway. @Lia do you have speed results from reading/writing to Wii U formatted storage?

 

[jacobsson]

@V10lator Awesome stuff dude, thanks you so much for investigating this, a download/file manager with some more performance would be highly desirable!
I was thinking just making a mod to Dumpling just to be able to choose one folder to copy from SD to the HDD. Currently I have ~120GB Retroarch data that I want to move from my SD card.

 

[V10lator]

I was thinking just making a mod to Dumpling just to be able to choose one folder to copy from SD to the HDD.

Do that! After that slowly mod it more and more and maybe in the end you'll be the one writing a good file manager for the Wii U?

 

[jacobsson]

Do that! After that slowly mod it more and more and maybe in the end you'll be the one writing a good file manager for the Wii U?

I'll try to make a PoC of this after I've looked into to some Retroarch related issues

Please don't stop making your idea, I'm not even close to as a good coder as you