Homebrew WIP libusbhsfs - USB Mass Storage Class Host + Filesystem Mounter static library.

[kickmeh]

i get warn, what is mean? is still right procces to build liblwext4?

Spoiler

 

[DarkMatterCore]

i get warn, what is mean? is still right procces to build liblwext4?

Spoiler

View attachment 239362

Looks good to me. The warnings highlighted in magenta are harmless.

 

[Chrscool8]

Just a heads up (so you don’t make the same mistake I did!) you won’t be able to just launch an app from a hdd. As soon as your app exits, the hdds will be unmounted so now that path that nxhbl has will be invalid (ie ums0:/app.nro). The quick workaround I thought of was just to copy the nro from hdd to sd then set the path to location on sd card. This is a bit mixed because the nro could be huge so it could take a while (slow launches), it might depend on loose files like config stuff and maybe the user doesn’t want to override the app on the sd card with the one of hdd (is it exists) so maybe copying to a temp folder would be better.

but yeah not like it can’t work, just not super straight forward. But adding hdd support as a way to copy / move apps to and from storages might be a better option imo, and would still be a very nice feature

Thanks for the input! I was thinking that might be the case and was banking on the copy idea being the backup. Appreciate your expertise!

 

[ShroomKing]

Was able to get the library working in pPlay if anyone wants to give it a try:
https://github.com/ShroomKing/pplay/releases/tag/3.1-usb1

 

[DarkMatterCore]

@ShroomKing Got a stack dump of that crash?

 

[DarkMatterCore]

Just released v0.2.2 to change the default mount behaviour for NTFS volumes. Should reduce the number of people complaining for NTFS volumes not being mounted if they weren't properly unmounted in the first place.

Developers, please keep in mind you can override the default mount flags at runtime before calling usbHsFsInitialize(), should you ever need to do so.

 

[mrdude]

Just released v0.2.2 to change the default mount behaviour for NTFS volumes. Should reduce the number of people complaining for NTFS volumes not being mounted if they weren't properly unmounted in the first place.

Developers, please keep in mind you can override the default mount flags at runtime before calling usbHsFsInitialize(), should you ever need to do so.

Thanks, I updated an modded tinfoil installer I am working on and this update is working well - tested on NTFS/Fat32/Exfat drives and no issues, I'll try a modded usb pen drive that's formatted to ext3/4 later, but so far so good.

 

[peteruk]

Thanks, I updated an modded tinfoil installer I am working on and this update is working well - tested on NTFS/Fat32/Exfat drives and no issues, I'll try a modded usb pen drive that's formatted to ext3/4 later, but so far so good.

You know it's gonna be asked, so it may as well be me are you planning on release your mod when finished and you're happy with it ?

 

[ShroomKing]

@ShroomKing Got a stack dump of that crash?

It freezes, then crashes when i exit from home menu but there's no stack dump.

 

[mrdude]

You know it's gonna be asked, so it may as well be me are you planning on release your mod when finished and you're happy with it ?

Yes, I'm pretty much done now - probably release today or tomorrow, I just fixed the last bug that I know about.

 

[DarkMatterCore]

It freezes, then crashes when i exit from home menu but there's no stack dump.

There should at least be some Atmosphère crash report logs in your SD card. You can use that information with addr2line and the ELF binary from your pplay build to build a stack dump, so to speak.

 

[ShroomKing]

There should at least be some Atmosphère crash report logs in your SD card. You can use that information with addr2line and the ELF binary from your pplay build to build a stack dump, so to speak.

Yeah i mean there are no crash logs, this particular crash doesn't generate any for some reason.
I wonder if it's an issue with my flash drive, seems way too slow at times. Unfortunately i don't have another.

 

[Naminave]

Was able to get the library working in pPlay if anyone wants to give it a try:
https://github.com/ShroomKing/pplay/releases/tag/3.1-usb1

Off-topic question: What is the maximum resolution or the proper format to reproduce?

--------------------- MERGED ---------------------------

Just released v0.2.2 to change the default mount behaviour for NTFS volumes. Should reduce the number of people complaining for NTFS volumes not being mounted if they weren't properly unmounted in the first place.

Developers, please keep in mind you can override the default mount flags at runtime before calling usbHsFsInitialize(), should you ever need to do so.

Are there plans to launch Drive formatting and repair functions?

 

[DarkMatterCore]

Are there plans to launch Drive formatting and repair functions?

Not at all.

 

[DarkMatterCore]

I haven't forgotten about this project just yet. I have been pretty busy since January, so I haven't been able to work on this and nxdumptool as much as I did last year.

That being said, I have been slowly working on improving libusbhsfs and squashing some nasty bugs in the process. Some of these changes include:

• A reimplementation of usbHsEpPostBuffer() to make it possible to use timeouts instead of indefinitely waiting for endpoint transfers to take place - which is pretty bad if an endpoint suddenly becomes stalled, resulting in those dreaded softlocks some of you guys have been experiencing.
• Thanks to the previous point, the Bulk-Only Transport driver could also be updated to accurately follow some small error correction guidelines that weren't being met.
• PKGBUILD scripts for NTFS-3G and lwext4 were updated as well to fix build issues experienced by some users under specific Linux distros.
• More data about each drive and mounted volume is now made available to homebrew applications using the library (VID, PID, mount flags and USB string descriptors).

Next up, here comes something I'm sure some people will be excited about: I have been working on a way to reenable USB 3.0 support under HOS 9.0.0+.

Some time ago, it was possible to use the "usb30_force_enabled" flag in Atmosphère's system_settings.ini to achieve this. However, HOS 9.0.0 introduced a hard dependency on the USB sysmodule in FS services, which means the USB sysmodule is now launched before Atmosphère is capable of parsing the INI file, rendering this flag completely useless.

Using ExeFS IPS patches is totally out of the question for the very same reason. And modifying Atmosphère to make it enable the "usb30_force_enabled" flag on its own doesn't work because SD card access is required at that point.

Looking for an alternate solution, I decided to modify Atmosphère to make it capable of hot-patching the USB sysmodule to override the flag check in its NSO before it's launched. And it worked! USB 3.0 access is again a possibility. Here's a dump of a raw UsbHsInterface element obtained from a USB 3.0 drive (HOS 11.0.2, AMS 0.18.1):

Spoiler

UsbHsInterface:
    (UsbHsInterfaceInfo) inf:
        (s32) ID: 0x50010000
        (u32) deviceID_2: 0x00000003
        (u32) unk_x8: 0x00000002
        (usb_interface_descriptor) interface_desc:
            (u8) bLength: 0x09
            (u8) bDescriptorType: 0x04
            (u8) bInterfaceNumber: 0x00
            (u8) bAlternateSetting: 0x00
            (u8) bNumEndpoints: 0x02
            (u8) bInterfaceClass: 0x08
            (u8) bInterfaceSubClass: 0x06
            (u8) bInterfaceProtocol: 0x50
            (u8) iInterface: 0x00
        (usb_endpoint_descriptor) input_endpoint_descs[0]:
            (u8) bLength: 0x07
            (u8) bDescriptorType: 0x05
            (u8) bEndpointAddress: 0x81
            (u8) bmAttributes: 0x02
            (u16) wMaxPacketSize: 0x0400
            (u8) bInterval: 0x00
        (usb_endpoint_descriptor) output_endpoint_descs[1]:
            (u8) bLength: 0x07
            (u8) bDescriptorType: 0x05
            (u8) bEndpointAddress: 0x02
            (u8) bmAttributes: 0x02
            (u16) wMaxPacketSize: 0x0400
            (u8) bInterval: 0x00
        (usb_ss_endpoint_companion_descriptor) input_ss_endpoint_companion_descs[0]:
            (u8) bLength: 0x06
            (u8) bDescriptorType: 0x30
            (u8) bMaxBurst: 0x0F
            (u8) bmAttributes: 0x00
            (u16) wBytesPerInterval: 0x0000
        (usb_ss_endpoint_companion_descriptor) output_ss_endpoint_companion_descs[1]:
            (u8) bLength: 0x06
            (u8) bDescriptorType: 0x30
            (u8) bMaxBurst: 0x0F
            (u8) bmAttributes: 0x00
            (u16) wBytesPerInterval: 0x0000
    (char) pathstr: "SsDevice3-/T2/P1/A01"
    (u32) busID: 0x00000004
    (u32) deviceID: 0x00000003
    (usb_device_descriptor) device_desc:
        (u8) bLength: 0x12
        (u8) bDescriptorType: 0x01
        (u16) bcdUSB: 0x0300
        (u8) bDeviceClass: 0x00
        (u8) bDeviceSubClass: 0x00
        (u8) bDeviceProtocol: 0x00
        (u8) bMaxPacketSize0: 0x09
        (u16) idVendor: 0x0BC2
        (u16) idProduct: 0x231A
        (u16) bcdDevice: 0x0708
        (u8) iManufacturer: 0x01
        (u8) iProduct: 0x02
        (u8) iSerialNumber: 0x03
        (u8) bNumConfigurations: 0x01
    (usb_config_descriptor) config_desc:
        (u8) bLength: 0x09
        (u8) bDescriptorType: 0x02
        (u16) wTotalLength: 0x0079
        (u8) bNumInterfaces: 0x01
        (u8) bConfigurationValue: 0x01
        (u8) iConfiguration: 0x00
        (u8) bmAttributes: 0x80
        (u8) MaxPower: 0x70
    (u64) timestamp: 0x0000000884CC1EDD

Pay special attention to the bcdUSB and wMaxPacketSize fields, as well as the availability of SuperSpeed endpoint companion descriptors.

Preliminary tests using the random reads carried out by NTFS-3G while getting filesystem stats from a mounted NTFS volume look very promising:

Spoiler

USB 2.1 (unmodified Atmosphère build):
[2021-03-01 01:11:36,7639944] -> statvfs() start.
[2021-03-01 01:11:42,135820152] -> statvfs() end.
Result: 5,371825752 seconds.

USB 3.0 (modified Atmosphère build):
[2021-03-01 01:13:19,911078844] -> statvfs() start.
[2021-03-01 01:13:22,876405504] -> statvfs() end.
Result: 2,96532666 seconds.

Furthermore, the current BOT driver works perfectly fine with BOT interfaces provided by USB 3.0 drives. I have yet to test sequential reads and writes, but judging by these small, random reads, I'm sure it'll be awesome.

Even though transfers already are faster than before, BOT doesn't support command queuing, so my endgoal is to implement a USB Attached SCSI (UAS) driver in libusbhsfs to fully take advantage of USB 3.0 speeds. Let me know what you think.

 

[masagrator]

Sounds like there is lately more demand for those type of patches that are applied before Atmosphere checks for config and exefs patches, but now there is no other way to apply it than recompiling loader in Atmosphere. Do you know if there is something that is in work that can fix this problem? Maybe applying to hekate bootloader/fusee-primary something that will load special patches alongside fusee-secondary so there is no need for fusee-secondary to have access to sdcard beforehand?
Or maybe fs reimplementation is only what we need for this case?

 

[DarkMatterCore]

Sounds like there is lately more demand for those type of patches that are applied before Atmosphere checks for config and exefs patches, but now there is no other way to apply it than recompiling loader in Atmosphere. Do you know if there is something that is in work that can fix this problem? Maybe applying to hekate bootloader/fusee-primary something that will load special patches alongside fusee-secondary so there is no need for fusee-secondary to have access to sdcard beforehand?
Or maybe fs reimplementation is only what we need for this case?

As far as I know, nothing is being done at this moment to provide an easy way to patch services launched before FS. I think Scires said he would try to come up with a solution, but he's currently busy with the tma2 reimplementation, so I'm sure it's hard for him to provide an ETA.

Both solutions would probably work fine btw. I don't think a full FS reimplementation is needed - dedicating a consistent/well-known RAM area to hold IPS patches for these services would be good enough for fusee-secondary, I guess.

 

[duckbill007]

How to test that USB3.0 enabling?

 

[DarkMatterCore]

How to test that USB3.0 enabling?

Patch locations 0x8718 and 0x87A8 in the uncompressed NSO with 05000014 (unconditional branch).

This will make the "usb30_force_enabled" checks always return 1.

 

[duckbill007]

Thank you. Is is for 11.0.1? Does this apply to usb:ds as well?