Homebrew Help me understand Read & Write speeds of the WiiU

V10lator · May 15, 2020

jacobsson said:
I'm not even close to as a good coder as you

You don't know my skills and I don't know yours, so don't say that.

Please don't stop making your idea

I won't, don't worry. It's just not a file manager or general downloading tool, it's more like a specialized downloader combined with WUP installer...

BTW: I made a very interresting finding: Untill some time ago my tool did all in one thread. Now, in the slow preperation of multitasking, I wrote an I/O queue. So there's the main thread and the I/O thread (both still on the same core as this is just preperation work).
What's interresting is that download speeds completely single threaded where maxing out at around ~4Mb/s, min speed 300Kb/s, averaging somewhere in between... Now with this I/O queue they are min 3.9Mb/s, max 4.9Mb/s, most of the time stable at 4.5Mb/s... Again: This is all on one core...

jacobsson · May 16, 2020

V10lator said:
You don't know my skills and I don't know yours, so don't say that.

I won't, don't worry. It's just not a file manager or general downloading tool, it's more like a specialized downloader combined with WUP installer...

BTW: I made a very interresting finding: Untill some time ago my tool did all in one thread. Now, in the slow preperation of multitasking, I wrote an I/O queue. So there's the main thread and the I/O thread (both still on the same core as this is just preperation work).
What's interresting is that download speeds completely single threaded where maxing out at around ~4Mb/s, min speed 300Kb/s, averaging somewhere in between... Now with this I/O queue they are min 3.9Mb/s, max 4.9Mb/s, most of the time stable at 4.5Mb/s... Again: This is all on one core...

Wow, that's quite some improvement!
Are these speeds to SD or HDD?

V10lator · May 16, 2020

jacobsson said:
Are these speeds to SD or HDD?

That was to SD. I just did the same test to USB and it seemed to be stable at 4 Mb/s but froze after some time, so my codes need improvement I guess.

//EDIT: The LED on the USB enclosure is still blinking, so it seems just the UI froze but it's still downloading in the background.

//EDIT²: And the UI unfroze. Seems like there's some bad slowdown somewhere in my codes.

//EDIT³: And frozen again. My queue codes really don't like USB.

//EDIT⁴: Most likely these codes are freezing the UI for some time:

Code:

    // Flush half of queue in case it grows too large
    ioBufSize += size;
    if(ioBufSize > 128 * 1024 * 1024)
    {
        debugPrintf("Shrinking...");
        while(ioBufSize > 64 * 1024 * 1024)
            executeIOQueue(false);
    }

//EDIT⁵: Was able to get the ASAN from @QuarkTheAwesome ( https://pastebin.com/QZMhUsP9 ) working in this multitasked environment and as a result I was able to build a debug build (where debugPrintf really does something instead of being a NULL macro). This showed me that it's not the shrinking of the queue freezing the UI but the flushing happening at the end of the download:

Code:

void flushIOQueue()
{
    debugPrintf("Flushing...");
    lockIOQueueAgressive();
    while(writeQueue != NULL)
        executeIOQueue(false);
 
    OSFastMutex_Unlock(&writeQueueMutex);
}

So I guess the download is fast as long as it goes to RAM but writing to USB is still slow.

//EDIT⁶: In case you're courious / want to test for yourself, these are my I/O queue codes: https://pastebin.com/PNeNCmWL - Note that it's still kind of a PoC. cthread_wrapper is just a C wrapper for https://github.com/wiiu-env/libgui/blob/master/include/gui/system/CThread.h as my tool already uses libgui so using CThread sounded like a good idea (but still: PoC style). I'm downloading like this:

Code:

    CURLcode ret = curl_easy_setopt(curl, CURLOPT_URL, url);
 
    char curlError[CURL_ERROR_SIZE];
    curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curlError);
    curlError[0] = '\0';
 
    long fileSize;
    if(!toRam && fileExists)
    {
        fileSize = getFilesize(fp);
        ret |= curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, headerCallback);
        ret |= curl_easy_setopt(curl, CURLOPT_WRITEHEADER, &fileSize);
    }
 
    ret |= curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, addToIOQueue)
    ret |= curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);

    ret |= curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, progressCallback);
    ret |= curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, curl);
    ret |= curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
 
    ret |= curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
 
    if(ret != CURLE_OK)
    {
        fclose(fp);
        curl_easy_cleanup(curl);
        debugPrintf("curl_easy_setopt error: %s", curlError);
        return 1;
    }
 
    debugPrintf("Calling curl_easy_perform()");
    ret = curl_easy_perform(curl);
    debugPrintf("curl_easy_perform() returned: %d", ret);
    addToIOQueue(NULL, 0, 0, fp);
    flushIOQueue();

Ofc. don't forget to call initIOThread() and shutdownIOThread().

//EDIT⁷: I guess you know how a C wrapper for a C++ API looks like but in case some other dev don't: Look at this cpp code:

Code:

#include <stdint.h>

#include <coreinit/thread.h>
#include <gui/system/CThread.h>

#include <cthread_wrapper.h>

CTHREAD createCThread(CThreadCallback callback, void *callbackArgs, CThreadAttributes iAttr, int32_t iPriority, int32_t iStackSize)
{
    return (CTHREAD)CThread::create((CThread::Callback)callback, callbackArgs, iAttr, iPriority, iStackSize);
}

OSThread *getOSThread(const CTHREAD thread)
{
    return (OSThread *)((CThread *)thread)->getThread();
}

void executeCThread(const CTHREAD thread)
{
    ((CThread *)thread)->executeThread();
}

combined with this header file:

Code:

/****************************************************************
 * This file is part of NUSspli.                                *
 * (c) 2020 V10lator <[email protected]>                        *
 *                                                              *
 * Licensed under the MIT license (see LICENSE.txt for details) *
 ****************************************************************/

#ifndef NUSSPLI_CTHREAD_WRAPPER_H
#define NUSSPLI_CTHREAD_WRAPPER_H

#include <wut-fixups.h>

#include <coreinit/thread.h>

#include <stdint.h>

#ifdef __cplusplus
    extern "C" {
#endif

typedef void *CTHREAD;
typedef void (* CThreadCallback)(CTHREAD thread, void *arg);

// Enum copied from https://github.com/wiiu-env/libgui/blob/master/include/gui/system/CThread.h
typedef enum
{
    CThreadAttributeNone        = 0x07,
    CThreadAttributeAffCore0    = 0x01,
    CThreadAttributeAffCore1    = 0x02,
    CThreadAttributeAffCore2    = 0x04,
    CThreadAttributeDetach        = 0x08,
    CThreadAttributePinnedAff    = 0x10
} CThreadAttributes;

CTHREAD createCThread(CThreadCallback callback, void *callbackArgs, CThreadAttributes iAttr, int32_t iPriority, int32_t iStackSize);
OSThread *getOSThread(const CTHREAD thread);
void executeCThread(const CTHREAD thread);
void suspendCThread(const CTHREAD thread);
void resumeCThread(const CTHREAD thread);
void setCThreadPriority(const CTHREAD thread, const int32_t prio);
bool isCThreadSuspended(const CTHREAD thread);
bool isCThreadTerminated(const CTHREAD thread);
bool isCThreadRunning(const CTHREAD thread);
void shutdownCThread(const CTHREAD thread);

#ifdef __cplusplus
    }
#endif

#endif // ifndef ifndef NUSSPLI_CTHREAD_WRAPPER_H

//EDIT⁸: The ASAN isn't thread save or some other weird thing happened:

Code:

Calling curl_easy_perform()
0x01049438    0x90830000    stw        r4,0(r3)
0x0104943c    0x2c0c0000    cmpwi      cr0,r12,0x0
0x01049440    0x9184000c    stw        r12,12(r4)
0x01049444    0x4182ffe8    b          2,12,0x104942c
0x01049448    0x908c0008    stw        r4,8(r12)
0x0104944c    0x7c832378    or         r3,r4,r4
0x01049450    0x4e800020    bclr       20,0
0x01049454    0x81830000    lwz        r12,0(r3)
0x01049458    0x38000000    addi       r0,r0,0

Address:      Back Chain    LR Save
0x11373a30:   0x11373a48    0x00001df8 <unknown>+0x0
0x11373a48:   0x11373a60    0x0104a35c coreinit.rpl|MEMFreeToExpHeap+0x58
0x11373a60:   0x11373a88    0x0d0158f4 homebrew_launcher|ASANFree+0x1e4
0x11373a88:   0x11373ab8    0x0d00e1e4 homebrew_launcher|executeIOQueue+0x138
0x11373ab8:   0x11373ad0    0x0d00e2ac homebrew_launcher|ioThreadMain+0x30
0x11373ad0:   0x11373ae8    0x0d044e64 homebrew_launcher|_ZN7CThread13executeThreadEv+0x6c
0x11373ae8:   0x11373b00    0x0d04548c homebrew_launcher|_ZN7CThread14threadCallbackEiPPKc+0x54
0x11373b00:   0x11373b18    0x0103c494 coreinit.rpl|__OSTestAssistReadPhysical32+0x6c
0x11373b18:   0x00000000    0x01041d6c coreinit.rpl|OSCheckThreadStackUsage+0xb0

(stacktrace produced by libWHBs crash handler / https://github.com/devkitPro/wut/blob/master/libraries/libwhb/include/whb/crash.h )

jacobsson · May 16, 2020

@V10lator Wow, thanks man! I really appreciate you sharing, I'd love to follow your process, would consider putting up github repo?

V10lator · May 16, 2020

jacobsson said:
@V10latorI'd love to follow your process, would consider putting up github repo?

There are licensing issues I need to resolve before publishing all sources but I'm working on it.

//EDIT: The issues beeing the tool stemming from WUPDownloaders codebase, so it's MIT licensed but libGui (and WUT) are GPL:
https://github.com/wiiu-env/libgui/issues/8
https://github.com/devkitPro/wut/issues/122

//EDIT²: See all codes shared above as public domain even if there's something other written inside of the files.

ploggy · Jan 10, 2021

Sorry for the bump but I was wondering if this is still being looked into?

piratesephiroth · Jan 10, 2021

TheChosen said:
Btw: Only if you circumvent the OS of WiiU, you can use full USB-Speed.

WUP installer doesn´t run in the OS of WiiU, so full speed is possible.
The unlocking of USB pretty much just circumvents the OS.
However: Unlocking Network-speed is a lot harder. The problem is because WiiU encrypts the whole OS. It´s a software-encryption. Software-encryption is allways slow.
When you run WUP installer, no data is encrypted = fast.
If you run anything in WiiU´s OS (ex.Retroarch), you encrypt it automatically & thus can´t use full speed.
Every RPX/RPL has to manually patched in order to circumvent software-encryption of WiiU´s "standard procedure".

this guy's bullcrap cracks me up everytime

QuarkTheAwesome · Jan 13, 2021

https://github.com/devkitPro/wut/pull/159 got merged into wut recently, which provides a pretty significant boost to sd card access for non-iosuhax wut applications. @Maschell was also looking into some strategies involving setvbuf to improve performance of file operations that don't use buffering (fgetc is particularly painful, since by default it makes a new call to the IOSU for every. single. byte.) You can compare Maschell's numbers from the GitHub issue at your lesiure.
I know this thread is more about networking speeds but I figured this was worth mentioning - the point about setvbuf and fgetc is especially important for ported code, GTA3 got some orders-of-magnitude improvements doing this

ploggy · Jan 13, 2021

QuarkTheAwesome said:
https://github.com/devkitPro/wut/pull/159 got merged into wut recently, which provides a pretty significant boost to sd card access for non-iosuhax wut applications. @Maschell was also looking into some strategies involving setvbuf to improve performance of file operations that don't use buffering (fgetc is particularly painful, since by default it makes a new call to the IOSU for every. single. byte.) You can compare Maschell's numbers from the GitHub issue at your lesiure.
I know this thread is more about networking speeds but I figured this was worth mentioning - the point about setvbuf and fgetc is especially important for ported code, GTA3 got some orders-of-magnitude improvements doing this

I saw this wut update being talked about on discord a while ago and of course my mind went straight to Retroarch

This would be perfect for RA no? how would one go about implementing this?

QuarkTheAwesome · Jan 13, 2021

ploggy said:
I saw this wut update being talked about on discord a while ago and of course my mind went straight to Retroarch
This would be perfect for RA no? how would one go about implementing this?

It looks like RA does a similar thing where it always reallocates a temp buffer for alignment reasons: https://github.com/libretro/RetroArch/blob/master/wiiu/fs/sd_fat_devoptab.c#L347
so the wut change to only do this when actually needed could probably be implemented, yeah. Then it'd be a case of adjusting the parent app to either align all its filesystem buffers nicely or use setvbuf where appropriate.
Interestingly, the libiosuhax version of this code seems to pay no attention to alignment at all: https://github.com/libretro/RetroArch/blob/master/deps/libiosuhax/iosuhax_devoptab.c#L349
presumably the iosu side has been changed up to not need aligned buffers. I do wonder what this means performance-wise, since I would expect Nintendo had a reason to require alignment in the first place...

QuarkTheAwesome · Jan 13, 2021

small update: it looks like 3ds and vita already have the setvbuf optimisation. enabling that for wiiu should provide a decent general speedup for iosuhax, and paired with the alignment buffer changes non-iosuhax setups should get some benefit
https://github.com/libretro/RetroArch/blob/master/libretro-common/vfs/vfs_implementation.c#L436

ploggy · Jan 13, 2021

QuarkTheAwesome said:
small update: it looks like 3ds and vita already have the setvbuf optimisation. enabling that for wiiu should provide a decent general speedup for iosuhax, and paired with the alignment buffer changes non-iosuhax setups should get some benefit
https://github.com/libretro/RetroArch/blob/master/libretro-common/vfs/vfs_implementation.c#L436

This is great stuff, I only wish I understood it and could implement it. but now that it's been posted someone can come in and take a swing at it

gblues still does stuff on WiiU RA now and again

Thanks.

jacobsson · Jan 13, 2021

@QuarkTheAwesome and @ploggy
Hi guys, sorry, I've been out of the scene lately. I'm very happy to see your correspondence in the thread! The news above could potentially lead to a significant and much needed performance lift for stuff like RA, thanks for sharing this Quark!

Next on my to-do list (when ever I get some more time) is to figure out how WiiMotes can be read as pointing/mouse device in the wiiU-version of RA, this would make the system one of coolest little hardware for MAME lightgun gaming.

ploggy · Jan 13, 2021

jacobsson said:
@QuarkTheAwesome and @ploggy
Hi guys, sorry, I've been out of the scene lately. I'm very happy to see your correspondence in the thread! The news above could potentially lead to a significant and much needed performance lift for stuff like RA, thanks for sharing this Quark!

Next on my to-do list (when ever I get some more time) is to figure out how WiiMotes can be read as pointing/mouse device in the wiiU-version of RA, this would make the system one of coolest little hardware for MAME lightgun gaming.

Agreed this WUT update could hopefully lead to a much needed boost to the SD performance

and since the setvbuf optimisation quark talked about is already in Retroarch for Vita/3Ds we wouldn't need to reinvent the wheel for WiiU.

IIRC we used to be able to play lightgun games with the WiiU Gamepad Touch screen (not on mame tho iirc?) snes/megadrive/mastersystem worked I think.. we could even navigate the RA UI with the touch screen.. but for some reason it doesnt work any more.

It would be great to play lightgun games with the Wiimote's (just seems like a natural thing to have on RA WiiU

Speaking of .. this maybe of interest to you jacobsson.
https://gbatemp.net/threads/retroarch-lightgun-support.548089/#post-9197476
Wiimpathy has done exactly that on Wii version of retroarch

depaul · Jan 13, 2021

Thanks guys if I may interfere in fact when @jacobsson implemented HDD access for WiiU retroarch, paradoxically the HDD turned out to be much slower than SD card.

Rom Street Fighter III 3rd strike:
-loads from sd card in 15 s
-loads from HDD in 45 s

So there is a big bottleneck regarding HDD I/O. Thanks!

jacobsson · Jan 14, 2021

depaul said:
Thanks guys if I may interfere in fact when @jacobsson implemented HDD access for WiiU retroarch, paradoxically the HDD turned out to be much slower than SD card.

Rom Street Fighter III 3rd strike:
-loads from sd card in 15 s
-loads from HDD in 45 s

So there is a big bottleneck regarding HDD I/O. Thanks!

I remember being so exited when finally getting it to work and then it showed to be...super slow!
But! Maybe the WUT changes will finally yield faster IO speeds for the HDD access as well, that would be extremely satisfying.

@ploggy
Thanks man! I'll try to diff the changes and see what's been done.

ploggy · Jan 15, 2021

jacobsson said:
I remember being so exited when finally getting it to work and then it showed to be...super slow!
But! Maybe the WUT changes will finally yield faster IO speeds for the HDD access as well, that would be extremely satisfying.

@ploggy
Thanks man! I'll try to diff the changes and see what's been done.

Thats what im thinking, those wut changes could help with the usb performance

No probs

CrisMod · Jan 15, 2021

jacobsson said:
@QuarkTheAwesome and @ploggy
Hi guys, sorry, I've been out of the scene lately. I'm very happy to see your correspondence in the thread! The news above could potentially lead to a significant and much needed performance lift for stuff like RA, thanks for sharing this Quark!

Next on my to-do list (when ever I get some more time) is to figure out how WiiMotes can be read as pointing/mouse device in the wiiU-version of RA, this would make the system one of coolest little hardware for MAME lightgun gaming.

If you are interested I'm using it already......with Wii gun accessories and the Wii remote connected to the PC combined to an infrared USB light bar to the monitor... it's amazing with all Arcade games guns from the past....

jacobsson · Jan 16, 2021

CrisMod said:
If you are interested I'm using it already......with Wii gun accessories and the Wii remote connected to the PC combined to an infrared USB light bar to the monitor... it's amazing with all Arcade games guns from the past....

Hi, thanks for the reply! I'm aware that this is possible on the PC, atm I'm trying to figure out why it doesn't work for RA on the wii u. Fixing this would make wii u a beast of light gun gaming, taking wii and wii u gun games into account.

N7Kopper · Jan 17, 2021

jacobsson said:
Hi, thanks for the reply! I'm aware that this is possible on the PC, atm I'm trying to figure out why it doesn't work for RA on the wii u. Fixing this would make wii u a beast of light gun gaming, taking wii and wii u gun games into account.

Especially if you could enable both the system's sensor bar and the GamePad's. (Even better if the user could do so at will)
Snatcher is best played with a lightgun for the fights, in all honesty.

Homebrew Help me understand Read & Write speeds of the WiiU

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

WAKA! WAKA!

I wish I could read

🦊

WAKA! WAKA!

🦊

🦊

WAKA! WAKA!

Well-Known Member

WAKA! WAKA!

Well-Known Member

Well-Known Member

WAKA! WAKA!

Well-Known Member

Well-Known Member

Lest we forget... what Nazi stood for.

Similar threads

Popular threads in this forum