Homebrew WiiU FTP Server (use WUT, fix inject failures and increase bandwith)

[Van154]

Hi, guys! I also use WiiUFTPserver and I like it more than ftpiiu. These are the issues that I have:
* After using WiiUFTPserver as HBL app. other apps are not loading. They simply return to HBL menu.
* I can copy small sized files to WiiU SDcard, but when I copy file over 1Gb, system hangs for indefinite time after transferring this file.
* There are some strange messages appear while copying files: "warning when settings file's rights rc=-196629"

WiiUFTPserver v8.2 has the same issues as 8.0. If I'm not mistaken, issues with hanging and warnings appeared after updating to 8.0 from v 7.x.

I use Haxchi, TotalCommander as a client. PC via Ethernet, WiiU via WiFi. Router uses 802.11bgn mixed mode

 

[Laf111]

* After using WiiUFTPserver as HBL app. other apps are not loading. They simply return to HBL menu.

Wow, i duno what happened here...

If you start the other apps before WiiuFtpServer, do they start fine ?

Could you please check if a log file was created under /storage_slc/log (the last numbered one) and have a try using MOCHA ?

* I can copy small sized files to WiiU SDcard, but when I copy file over 1Gb, system hangs for indefinite time after transferring this file.

Copy to SDcard has horrible performances, Why not using the network to transfer the file ???
After all, it is the essence of this app.

Also : the max size of a file to be manipulated on the wii-U must be less than 1GB.

The Wii-U have only 1GB of RAM for apps.
So trying to load in memory a file bigger than than will fail. (i don't know how it is really done in the low layers of the WUT & IOSUHAX libraries)

* There are some strange messages appear while copying files: "warning when settings file's rights rc=-196629"

Seems to be a consequence of copying large file on the SDCard or trying to manipulate a too large file... You should also have "failed to open file" messages, no?

 

[V10lator]

https://github.com/Laf111/WiiUFtpServer/releases/tag/V8-2 (add a CRC calculation and a tool to check CRC)

I don't get why this is needed. I mean FTP runs over TCP and TCP does checksuming already, doesn't it? Also I'm doing important FTP uploads to various servers since 15+ years. Probably transfered terrabytes of data over way more unstable connections than LAN but never ever needed a CRC check.

when I copy file over 1Gb, system hangs for indefinite time after transferring this file.

Wii U FTP Server is using too large buffers. So it looks like it's tranfairing extremely fast but in reality it stalls whenever the buffers need to get flushed (like at the end of such a large transfer).

 

[Laf111]

I mean FTP runs over TCP and TCP does checksuming already, doesn't it

Yes but not enought to detect CRC errors : http://noahdavids.org/self_published/CRC_and_checksum.html

I also use FTP for years without any problems and no worries about that but after some research it appears that CRC errors happens on every network. A faulty Ethernet cable can cause them. It is considered as an issue when over 1% of the traffic is affected.

That's why pro FTP programs (client & server) implement the command XCRC to exchange hash value between the client and the server to be sure that no CRC errors occurs during the transfer on critical data.

Wii U FTP Server is using too large buffers. So it looks like it's tranfairing extremely fast but in reality it stalls whenever the buffers need to get flushed (like at the end of such a large transfer).

When uploading small files, you'll see that the"stall time" is not as long as it is with big files that use the whole preallocated buffer (> 84MB) so i don't think that it has something to do with it.

Only uploads use the whole buffer (~83MB preallocated), and it is mainly because i found it faster to receive all the data then write it to file.

The "stall" time is relative to the size of the file downloaded but i think mainly caused by IOSUHAX_FSA_ChangeMode() to put the rights on the file (and now it also include the CRC32 calculation time).

 

[V10lator]

Yes but not enought to detect CRC errors : http://noahdavids.org/self_published/CRC_and_checksum.html

Sorry to say but that side is amateur and not professional:

it would appear that the Ethernet CRC will detect the vast majority of errors. Unfortunately, "vast majority" is not "all"

So adding a CRC check on top of that CRC check will help cause...? Also read the RFCs/standarts for yourself: They are designe since more than 50 years. Do you really think such issues would slip unnoted?

Any errors in the higher layer software of these devices or transient failures of the hardware (memory or bus)

I'm still seeing corruption issues even when connecting the Wii U directly to the PC, so no "higher layer software of these devices" involved. But even if there's a switch in between: Do you really think software beeing used all around the globe, handling terrabytes of data in a second, will suddendly fail on a ~3 Mbit/s link?

So, if there is an undetectable corrupted segment on the network once every 34 hours or even once a month, why aren't the databases hopelessly corrupt? Why aren't applications randomly failing? Well, in many cases a corrupt segment is no big deal.

But in many other cases it would be a big deal. [EDIT]Also the site doesn't give a reply to the question "why aren't the databases hopelessly corrupt? Why aren't applications randomly failing?". Really: Think about it: Tech giants like Google, Facebook and so on are syncing databases all around the globe without issues. So "many cases" is a joke. In most commercial/important cases it would be a big deal - [/EDIT] Like my example:

I'm doing important FTP uploads to various servers since 15+ years. Probably transfered terrabytes of data over way more unstable connections than LAN

To correct myself: Was probably petabytes of data I transfaired without an issue... Also we see more corruptions than "once every 34 hours or even once a month". I don't want to be mean but this really seems to be a bug in your software. Please don't try to find such poor excuses. I know how it feels to have corruption issues. Hell, my software is affected regulary, too, but I try my best to fix my codes instead of saying it's the fault of others.

buffer (~83MB preallocated)

That's way too large. What you see as speed improvement is relaxing IOSU so it has more time for the network transfair but when the buffer gets flushed IOSU will be stalled by that, so the download will be stalled, too. That's not only at the end of the file but whenever the buffer needs to be flushed... FTP clients just don't calculate these stalls into their speed measurements..
I'm still searching for ways to relax IOSU to do this async stuff (DL from network card to PPC and write from PPC to storage medium) faster but... It seems around 3.3 Mbit/s is a hardware limitation... Even Nintendo software struggles at this value... Would love to have more devs than me to try to break through this instead of, well, cheating with this large buffer strategy.

Again: I don't want to sound mean. Really sorry if I do. I just thing you're distracting yourself a bit.

 

[Laf111]

Sorry to say but that side is amateur and not professional:

Sorry, the page i linked wasn't one of those i consulted in the first place.
I looked in CISCO forum, if i remember well...

I can't answer to your question but here seems to be an explaination : https://stackoverflow.com/questions...smission-errors-sometimes-if-the-data-is-usin But i don't think that the FTP command XCRC was created for nothing

I think the issue#10 on CRC error is closed in V8-2 (was caused because i resized the internal file's buffer).
It generated only few 3 errors on 9000 files transferred (more when using Wifi 802g and even more in 802n)

But anyway, in case of injecting critical files (like system ones), a CRC check is a good thing.


You're right : the writing time (flushing the buffer) is not taken into account on client side.
But isn't it always the case with files smaller than the buffer?
(unless you're writing per byte which doesn't make sense)

I used the server side measure to benchmark (dividing the number of bytes received by the duration of the transfer and so the flushing time is taken into account). It gives me always better speed when i load all the data into RAM then flush the buffer in a row.

I might explain this because of the write/flush sequence which i think is faster when the size of the file is "big enought".
(like when benchmarking a HDD, you'll always get a better speed when transferring a "big" single file that many small files)

But i'm not sure of that and it seems you know more things than i am...

 

[V10lator]

@Laf111 IIRC Maschell told that 1 MB buffers are perfect for writing: It's a multiple of a HDDs block size (512b or 4k) but still small enough to allow fast flushes. From personal testing I also got to this value. Then again not many homebrews do network and storage I/O at the same time, esp not as fast as Wii U FTP Server.

I also don't know much but the Wii U uses two processors, right? I think it was Quarky who told the ARM CPU (running IOSU) is pretty weak and bottlenecks the PPCs I/O (as for security reasons USB and network card aren't wired to the PPC but to the ARM. So the data goes from the network card to ARM, from there to the PPC, then back to ARM and finally to the storage medium. This was the hardware side. On software side (simplified) : IOSU -> CafeOS -> Wii U FTP Server -> libiosuhax -> WUPServer). So my guess is that the download slows down the writing and vice-versa. Bigger buffers compensate for this as they delay writing, so the ARM is able to use all of his "power" for the download, but then again you get large flushing times which annoys users and in the end it should take a similiar amount of time as it doesn't make the ARM faster.

WUPServer is a part of the CFW running on the ARM/IOSU. That's also why libIOSUhax...

BTW: A unconfirmed long term goal of Aroma CFW will be to allow IOSU plugins. Well, in fact you can do this with Tiramisu already but it's complicated (Bloopair is implemented into the Wii Us Pro Controller driver on IOSU, for example). I wonder what would happen if the whole FTP server is implemented on the ARM... Like will it make things slower cause the ARM has to do more stuff or will it make things faster as the hardware can be talked to directly?

//EDIT: Thought about the last statement some more... Doing it the Tiramisu way should work on Aroma, too, and would give an FTP server running always in the background... Maybe a small normal homebrew which uses libIOSUhax or some self-implemented IPC to start/stop/restart the server and read logs... I'm not requesting this through as it will be damn complicated to write.

 

[godreborn]

an ftp server always running in the background is something I mentioned in the tiramisu thread, since I think it would be a good idea. wup server was able to run on the home screen, but it was slow as hell. that's how I changed the theme on my vwii, now it's painless.

 

[Laf111]

V9 is out

Comes with a CRC checker in python tested on WINDOWS and Linux.
It uses a pool of threads and appears to be 30 times faster that CRC Checker V1-2 (windows shell)

Time for me to finalize Cemu2WiiU now the FTP server is stable.

 

[Laf111]

V9-1

2022/03/15 V9-1 :

> add symlinks resolution support for the ones found in updates and DLC folders (fix transfer errors on those files)
> move CRC32 calculation after the analysis of the return code of fwrite() in net::recv_to_file
> fix poor transfer speeds to SDCard with using libFat (it also fix app hang/cripple when activating the log file)
HARD FIXED the max simultaneous transfers to SDCard to 4 (to avoid write errors)
> fix issue#14 : Transfer large size file on SDcard, WIIUFTPSERVER disconnect?
> rollback to IOSUHAX_FSA_ChangeMode because rights was not correctly set using system call.
> lower the preallocated transfer buffer to ~13MB
So the total RAM used is ~110MB (when transferring 8 files)
It lowers the ending closing connection time but also gives extra KB/s
> add folder deletion support
> fix a regression on renaming file process
> crcChecker V2-1 fix "TypeError: string operation on non-string array" that could occurs when treating the report

 

[Laf111]

V9-2

2022/03/19 V9-2 :
> remove DEBUG traces left in ftp.c and vrt.c
> add a 10s timeout for the CRC32 Warning
> final tweaks to optimize multi transfers (numbers of transfers launched simultaneously)
> display current transfer speed stats when no transfer is active and when only a browse connection is opened
> build.sh : fix build date not replaced in meta.xml
> add ./toWUP/createChannel.sh (Linux shell version)
> add music to channel version

 

[Laf111]

2022/04/01 V9-3 :

> use libFat and libIOSUHAX versions from Crementif (thanks!!!) : better timestamps and symlinks support
> now symlinks are identified in FTP client browser
> fix regression Cyberduck connection failure #15
> fix FileZilla client fail to delete a folder #16
> fix WinScp fail to list root #17
> fix sd drive mount twice after the NAND files backup
> add the ftp_RMD command (equivalent to ftp_DELE)
> update meta.xml template
> fix case for folders names in ./toWUP/createChannel.bat
> fix release date patching in meta.xml in build.sh

 

[godreborn]

@Laf111 , my friend, I just installed your new ftp server. I held off on installing, since I thought my wii u had some major problems with overheating. I replaced the fan, and it hasn't overheated since. anyway, I put it on the nand now, since it's much easier to uninstall, then install a new version whenever a new version is out. there seems to be no logic to the ordering of things in system settings, unless it's based on title id (who's going to know that!?). I found a bug though. I always install a new channel version, then test it by deleting the wiiu ftp server install folder. it will not let me delete its folder. it will delete the contents inside, just not the folder itself. the previous version I had installed (forgot which it was) didn't have this problem.

 

[Laf111]

my friend, I just installed your new ftp server

Hey,
Thank you for your feedback : https://github.com/Laf111/WiiUFtpServer/issues/18

 

[Laf111]

I found a bug

Hey,

The V10 fix it :

2022/04/20 V10 :
> fix issue "Still rare CRC errors on upload (about 1 file on 9000 randomly) #19 ":
replace the preallocate buffer by an ontime allocation + use the transfer buffer as internal's file buffer (setvbuf)
> multi transfers enhancement : double TRANSFER_BUFFER_SIZE for upload and lower the buffer used for download to the min
+final tweaks on threads priorities to maximize the number of connections opened simultaneously on the 3 cores
> update crcChecker.py to V2-2 :
- fix answer "no" not taken into account when choosing to not use an report that already exists in the script's folder
- add a try catch for permissions errors
- fix failure when root = folder to check
> move IOSUHAX_FSA_ChangeMode() call in the transfer thread (for uploads)
> fix issue "Folder left after deletetion #18"
> fix "Crash when exiting if no connections were opened #20"

 

[godreborn]

Hey,

The V10 fix it :

2022/04/20 V10 :
> fix issue "Still rare CRC errors on upload (about 1 file on 9000 randomly) #19 ":
replace the preallocate buffer by an ontime allocation + use the transfer buffer as internal's file buffer (setvbuf)
> multi transfers enhancement : double TRANSFER_BUFFER_SIZE for upload and lower the buffer used for download to the min
+final tweaks on threads priorities to maximize the number of connections opened simultaneously on the 3 cores
> update crcChecker.py to V2-2 :
- fix answer "no" not taken into account when choosing to not use an report that already exists in the script's folder
- add a try catch for permissions errors
- fix failure when root = folder to check
> move IOSUHAX_FSA_ChangeMode() call in the transfer thread (for uploads)
> fix issue "Folder left after deletetion #18"
> fix "Crash when exiting if no connections were opened #20"

we have success.

 

[Laf111]

2022/05/18 V11 :

• fix issue " Still some CRC errors occur when uploading on SDCard #21 ": remove sevbuf use to fix the last CRC errors and limit to only 1 upload to sdCard (simultaneous transfers leads in CRC errors)
• multi transfer enhancement : reduce open/close connection time with
  • using a static preallocated array of threads (before it was allocated statically when creating the connection)
  • manually set cpu and priority of transfer threads
• now use also a small buffer for UL operations (before setting manually CPU and priority, a large buffer was used to let other connections start)
• tweak the buffers sizes used for DL / UL : use 25% DL vs 75% UL of the reserved memory for buffer socket operations (instead of 50/50). This last setting increase DL speeds obtain when multi-transferring. Using a small buffer for upload reduced the "speed displayed" on UL operation (now take the fwrite time into account).

This version gives max speeds over 4M/s on both ways and treat a large number of files more quickly.

 

[godreborn]

@Laf111 , I'm trying to build your ftp server, but there's a problem. I can not make libfat. there's an error that seems to point to the code or I'm using too new/old a version of devkitpro or something. here's what happens when building ftp server:

- WiiUFtpServer 11.0.0 -
========================

checking env ...
-----------------------------------------------------
more: cannot open makefile: No such file or directory
=====================================================
building...
-----------------------------------------------------
clean ...
controllers.c
ftp.c
main.c
nandBackup.c
net.c
virtualpath.c
vrt.c
linking ... WiiUFtpServer.elf
/opt/devkitpro/devkitPPC/bin/../lib/gcc/powerpc-eabi/11.2.0/../../../../powerpc-eabi/bin/ld: cannot find -lfat
collect2: error: ld returned 1 exit status
make[1]: *** [/opt/devkitpro/wut/share/wut_rules:78: /home/godreborn/WiiUFtpServer-11/WiiUFtpServer.elf] Error 1
make: *** [Makefile:112: build] Error 2
ERRORS happened when building RPX file, exit 2
godreborn@GODREBORN-LAPTOP:~/WiiUFtpServer-11$

it can't create the rpx, and I think it's due to libfat. could you send me the library files of libfat? or what's created from make?

 

[godreborn]

@Laf111 , nm, I got it. I noticed you had a dependencies zip file, so I got libfat from there. that was the problem:

 

[Laf111]

could you send me the library files

Hey,

You can get the download links in the BUILD section of the README.MD :

To build from scratch :​

​

- install [devkitPro](https://github.com/devkitPro/installer/releases/latest) (in DEVKITPRO_PATH)​

- get [libWUT](https://github.com/devkitPro/wut) ​

- get [libIOSUHAX](https://github.com/Crementif/libiosuhax) (crementif version)​

- get [libFat](https://github.com/Crementif/libfat) (because using only WUT+IOSUHAX cripple SDCard performance, crementif version)​

I use versions of libFat and libIOSUHAX from @Crementif because he fixed things for symlinks support.