Homebrew WIP libusbhsfs - USB Mass Storage Class Host + Filesystem Mounter static library.

[DarkMatterCore]

Thank you, this is awesome. I just want this implemented so I can play music and movies from a USB on the TV, that would be ill. Then, run some classic games on retroarch, all of that without using up space on the SD card. As a matter of fact I still use from time to time my Wii to do these things, so it will be a great, great addition to be able to do it at last on the Switch.

The Wii being able to pull this off like it's nothing it's exactly what motivated me to do this. I still have mine and I love it - the sentimental value it has to me can't be measured.

 

[kickmeh]

well done, finally UMS come in

just try an example build, seems need to declare of usbHsFsInitialize()

 

[cucholix]

This will off-load some files from my SD Great

 

[ploggy]

My SD card thanks you sincerely.. poor little guy.

 

[DarkMatterCore]

well done, finally UMS come in

just try an example build, seems need to declare of usbHsFsInitialize()

It actually means usbHsFsInitialize() ran, but couldn't finish all of its startup steps. 0x19C8C seems to be a USB error code.

Are you running any background sysmodules with USB access?

You can alternatively edit the Makefile from the example test app to link it against the debug version of the library (e.g. -lusbhsfsd instead of -lusbhsfs). That will generate a logfile in your SD card root directory with information about the function call that triggered the error code.

 

[kickmeh]

It actually means usbHsFsInitialize() ran, but couldn't finish all of its startup steps. 0x19C8C seems to be a USB error code.

Are you running any background sysmodules with USB access?

You can alternatively edit the Makefile from the example test app to link it against the debug version of the library (e.g. -lusbhsfsd instead of -lusbhsfs). That will generate a logfile in your SD card root directory with information about the function call that triggered the error code.

yeah i use ams_mitm.kip from xortroll over my hekate_ipl, so i think it ran background, ok i will edit my atmosphere configuration, also Makefile for logfile, thanks for advice and quick reply

 

[ber71]

Does this support GPT partitions? Or only MBR

 

[DarkMatterCore]

Does this support GPT partitions? Or only MBR

Currently both. FatFs implements partition table parsing on its own, and supports both MBR and GPT.

We'll be working on adding support for more filesystems soon. If their libraries don't provide partition table parsing capabilities on their own, it will be added to libusbhsfs.

 

[kickmeh]

ok seems listed everything on my card reader, also i do several test, i open on hbmenu i use full ram access mode
with usb 2.0 flashdrive on third time i launch .nro it wont listed anymore (no matter is exfat,fat32,ntfs,etc), idk maybe alot sysmodule i ran on my background procces
with usb 3.0 flashdrive is listed everytime (exfat)
with usb 3.0 external hdd is listed everytime (ntfs)
with usb 3.0 card read is listed everytime (fat32)

after this just adapted on file manager homebrew application (eg; nx-shell) then use borealis ui it will be great
thanks for ur work

 

[DarkMatterCore]

ok seems listed everything on my card reader, also i do several test, i open on hbmenu i use full ram access mode
with usb 2.0 flashdrive on third time i launch .nro it wont listed anymore (no matter is exfat,fat32,ntfs,etc), idk maybe alot sysmodule i ran on my background procces
with usb 3.0 flashdrive is listed everytime (exfat)
with usb 3.0 external hdd is listed everytime (ntfs)
with usb 3.0 card read is listed everytime (fat32)
View attachment 235200

after this just adapted on file manager homebrew application (eg; nx-shell) then use borealis ui it will be great
thanks for ur work

Check the logfile, maybe there's something wrong going on with the USB 2.0 flash drive you're using. If it's related to the BOT driver, I can fix it.

 

[duckbill007]

If run example with inserted usb drive it hangs until I unplug and reinsert usb drive.

 

[blawar]

NSAInstaller (or Where's my waifu Awoo-Installer) and Tinfoil?

Tinfoil has its own usb hard drive implementation that was released about a month ago (it does not use this library).

--------------------- MERGED ---------------------------

We performed benchmarks reading the first raw 128 MiB (concurrent reads) from each connected drive with logging enabled back when the Bulk-Only Transport driver was still being developed, and we got speeds between 4 and 6 seconds. We used portable Seagate HDDs and WD HDDs with custom enclosures. So we could be talking anywhere between ~21 and 32 MiB/s - which yes, is better than fsp-usb's performance.

Disabling debug logging makes transfers go even faster, but we can't properly do benchmarks without it. Even if you're using the example test application, there's a noticeable difference if you link it against the debug build of the library vs the release one.

I benchmarked Tinfoil's implementation earlier this year using a fast SSD, and achieved speeds between 30-35MB depending on file system and cluster size. 35MB was not stable, and used exfat with obscenely high cluster size of 16MB IIRC.

 

[Naminave]

has its own usb hard drive

let me guess. ams_mitm.kip from Xolltrol?

And please don't come ruin this thread.

 

[DarkMatterCore]

@Ghost92 Please stop. I just don't want the thread to be derailed.

Each developer is free to implement stuff like this in whichever way they want. This is just another option.

If they can do it better than us, then so be it. This is not a competition.

 

[Naminave]

Currently both. FatFs implements partition table parsing on its own, and supports both MBR and GPT.

We'll be working on adding support for more filesystems soon. If their libraries don't provide partition table parsing capabilities on their own, it will be added to libusbhsfs.

How likely is the partition to get corrupted with the first versions? I say this because early versions of the most popular Wii browser used to corrupt partitions and freeze.

How much does the symbols or special letters affect folders and files?

 

[DarkMatterCore]

How likely is the partition to get corrupted with the first versions? I say this because early versions of the most popular Wii browser used to corrupt partitions and freeze.

How much does the symbols or special letters affect folders and files?

Not very likely, as long as the developer using this library makes sure to close all open file/dir handles, and the user removes the USB Mass Storage device at a point where no I/O operations are being done (plug 'n play is supported). All FatFs writes are flushed directly to the USB Mass Storage devices.

UTF-8 path encoding is used for both input and output operations. It doesn't suffer from the same limitations as the fsp-srv service.

 

[Naminave]

Not very likely, as long as the developer using this library makes sure to close all open file/dir handles, and the user removes the USB Mass Storage device at a point where no I/O operations are being done (plug 'n play is supported). All FatFs writes are flushed directly to the USB Mass Storage devices.

UTF-8 path encoding is used for both input and output operations. It doesn't suffer from the same limitations as the fsp-srv service.

Good. I hope to see browsers like Apollo and players like TriPlayer. Although the latter uses Sys-Module to keep the playback in the background and even resume it upon waking from deep sleep and I don't know how well libusbhsfs would get along in sys-module implementations.

 

[DarkMatterCore]

Good. I hope to see browsers like Apollo and players like TriPlayer. Although the latter uses Sys-Module to keep the playback in the background and even resume it upon waking from deep sleep and I don't know how well libusbhsfs would get along in sys-module implementations.

Yeah, I don't think it would get along very well. It's definitely not suitable for anything running in the background. A dedicated buffer per drive helps achieve better speeds and simplifies concurrency (e.g. reading data from more than one drive at the same time).

Doesn't mean a fork of this library can't be created to cater such needs. It should be possible.

 

[blawar]

let me guess. ams_mitm.kip from Xolltrol?

View attachment 235240
And please don't come ruin this thread.

That ams_mitm.kip is no longer used, and it contains no USBHDD code. Many people will likely remember it as the kip that disabled BIS write protections in AMS for incognito. You can verify this by trying to install the kip--atmosphere will not boot as it was compiled from an old version of AMS.

Tinfoil's USBHDD driver is not a kip or service, but is a homebrew library similar to libusbhsfs and was written over a year ago but never released. And there were many people who helped me debug my implementation over the past year.

@DarkMatterCore You are right, FatFs is lacking when it comes GPR/MBR management. I ended up writing my own GPT/MBR parser and handling that manually to support multiple partitions per volume. Which other file systems are you looking to support? NTFS is the only other fs that is common on external hard drives, and ntfs-3g is a monolithic mess.

 

[DarkMatterCore]

@DarkMatterCore You are right, FatFs is lacking when it comes GPR/MBR management. I ended up writing my own GPT/MBR parser and handling that manually to support multiple partitions per volume. Which other file systems are you looking to support? NTFS is the only other fs that is common on external hard drives, and ntfs-3g is a monolithic mess.

Yeah. It's the same reason why more than one FAT volume isn't being mounted right now - the way it handles multiple partitions expects you to know how many partitions there are for each available drive beforehand.

EXT support is also in my plans.