Homebrew WIP libusbhsfs - USB Mass Storage Class Host + Filesystem Mounter static library.

[DarkMatterCore]

Failed to fetch tweet https://twitter.com/DarkMatterCore/status/1341335146784501760

If no issues are discovered in the next few hours (and if I don't undergo another power cut), I'll make a proper release today.

 

[DarkMatterCore]

https://github.com/DarkMatterCore/libusbhsfs/releases/tag/v0.2.0

It has been a nice trip so far, and a great learning experience. Thank you all. I don't think I'll implement support for other filesystems at this point. I am, however, willing to:

* Implement other nice features, such as a NTFS write cache.
* Update the library if a libnx / HOS update breaks it.
* Fix any possible bugs that may be discovered down the road.

The ball is no longer in my park, which means it's now up to other devs to adopt and use the library if they wish to do so. It has been a fun project, and I really wish more people could take advantage of it.

I can finally continue my nxdumptool rewrite, which was stalled to properly develop libusbhsfs. Not that it matters anyway, since the reason the library exists in the first place is to be used in nxdumptool.

Happy holidays.

 

[hippy dave]

I don't think I'll implement support for other filesystems at this point.

What about Mac file systems?

...I'm not even serious I was going to suggest HFS+ as a joke, then I remembered they don't even use that any more, and have got some new FS with weird virtual volumes and stuff. Seems to work nice on the internal drive but I doubt there are many pressing reasons to use it on a USB drive.

Epic work, great contribution to Switch homebrew, thanks!

 

[DarkMatterCore]

What about Mac file systems?

...I'm not even serious I was going to suggest HFS+ as a joke, then I remembered they don't even use that any more, and have got some new FS with weird virtual volumes and stuff. Seems to work nice on the internal drive but I doubt there are many pressing reasons to use it on a USB drive.

Epic work, great contribution to Switch homebrew, thanks!

In all seriousness, I'm open to PRs to implement support for other filesystems.

 

[mrdude]

Nice, I'm modding awoo-installer to be able to have custom themes, and now I can implement the code from tinleaf to give it usb hdd support as well. Sweet. Thanks for posting the pre-built binaries of the version that supports ntfs, I couldn't get these to build on ubuntu as I kept getting an error I had missing dependencies (patch), even though they were installed. The fat version built fine though.

 

[DarkMatterCore]

Nice, I'm modding awoo-installer to be able to have custom themes, and now I can implement the code from tinleaf to give it usb hdd support as well. Sweet. Thanks for posting the pre-built binaries of the version that supports ntfs, I couldn't get these to build on ubuntu as I kept getting an error I had missing dependencies (patch), even though they were installed. The fat version built fine though.

Did you try using dkp-makepkg instead of makepkg?

You'll need to build and install both NTFS-3G and lwext4, anyway. Static libraries aren't statically linked to other libraries.

 

[mrdude]

Did you try using dkp-makepkg instead of makepkg?

You'll need to build and install both NTFS-3G and lwext4, anyway. Static libraries aren't statically linked to other libraries.

Yep, I tried that using dkp-makepkg, that's when I got the issue with dependencies.

I followed the readme and added -lntfs-3g -llwext4 to the makefile, and manually copied the files you posted in the release page into the relevant folders in portlibs - but I get this issue:

/opt/devkitpro/devkitA64/bin/../lib/gcc/aarch64-none-elf/10.2.0/../../../../aarch64-none-elf/bin/ld: cannot find -lntfs-3g
/opt/devkitpro/devkitA64/bin/../lib/gcc/aarch64-none-elf/10.2.0/../../../../aarch64-none-elf/bin/ld: cannot find -llwext4

I can build fine using the fat version you posted - am I missing some files?

 

[DarkMatterCore]

Yep, I tried that using dkp-makepkg, that's when I got the issue with dependencies.

I followed the readme and added -lntfs-3g -llwext4 to the makefile, and manually copied the files you posted in the release page into the relevant folders in portlibs - but I get this issue:

/opt/devkitpro/devkitA64/bin/../lib/gcc/aarch64-none-elf/10.2.0/../../../../aarch64-none-elf/bin/ld: cannot find -lntfs-3g
/opt/devkitpro/devkitA64/bin/../lib/gcc/aarch64-none-elf/10.2.0/../../../../aarch64-none-elf/bin/ld: cannot find -llwext4

I can build fine using the fat version you posted - am I missing some files?

Yes, the static library builds for NTFS-3G and lwext4.

Try running this instead:

dkp-makepkg -d -i --noconfirm

I don't really know why dependency checks fail for patch, but that should take care of the problem.

 

[DarkMatterCore]

Went ahead and removed the patch dependency from PKGBUILD scripts. It was pointless.

Also updated the src tarball from the v0.2.0 release.

 

[mrdude]

Went ahead and removed the patch dependency from PKGBUILD scripts. It was pointless.

Also updated the src tarball from the v0.2.0 release.

Thanks for updating.

Using this command works: dkp-makepkg -d -i --noconfirm

Also I think you messed up on liblwext4 as that was still asking for dependencies while using this command: dkp-makepkg -i --noconfirm, the other file worked fine.

 

[DarkMatterCore]

Thanks for updating.

Using this command works: dkp-makepkg -d -i --noconfirm

Also I think you messed up on liblwext4 as that was still asking for dependencies while using this command: dkp-makepkg -i --noconfirm, the other file worked fine.

Yeah, I was just informed it isn't needed to add cmake as a dependency. I'll take care of that too.

 

[mrdude]

Yeah, I was just informed it isn't needed to add cmake as a dependency. I'll take care of that too.

Thanks, that will make life easier for noobs like me, I have spent all day trying to get tinleaf to compile properly (with ntfs), now you modded those files, I was done in a few minutes :-). Now I can get back on to other stuff, thanks again.

 

[ShroomKing]

I figured out why ntfs-3g doesn't compile on mac even when skipping dependency checks.
You have to use the gnu version of make (install make from package manager) and symlink it to /usr/local/bin/make so it's used instead of the make bundled with xcode.

 

[DarkMatterCore]

Just released v0.2.1 as a quick bugfix for mount name ID unregistration while destroying filesystem contexts: https://github.com/DarkMatterCore/libusbhsfs/releases/tag/v0.2.1.

Also added a preprocessor macro to help developers generate a filesystem type string based on the filesystem type value from a UsbHsFsDevice element, and a Makefile fix for people who use the repository as a git submodule.

 

[Chrscool8]

Awesome work! I'm excited to try to add this to Homebrew Details in the future. Might be cool to manage and launch homebrews on external media.

 

[TheGodMauro]

Thanks for all your work dude, it's an incredibly necessary library for homebrew, which will surely come to be of use in the future.

And also...

https://twitter.com/DarkMatterCore/status/1341335146784501760

If no issues are discovered in the next few hours (and if I don't undergo another power cut), I'll make a proper release today.

My fellow paisano, I don't think many people in this forum can understand how real the power cut thing hits.

 

[DarkMatterCore]

Awesome work! I'm excited to try to add this to Homebrew Details in the future. Might be cool to manage and launch homebrews on external media.

That sounds awesome. I'll surely look forward to it if it ever becomes a thing.

My fellow paisano, I don't think many people in this forum can understand how real the power cut thing hits.

It does hit hard, yeah. I have (sadly) gotten used to it by now, though - there's not much that can be done beyond hoping it's not another 5-day blackout each time it happens, heh.

It's always a good thing to meet other paisanos around these places. We're definitely not too many, and the number gets extremely low when it comes to homebrew development.

Nice pfp, btw - been playing Yakuza Kiwami for some days now, thanks to a friend.

 

[TotalJustice]

Awesome work! I'm excited to try to add this to Homebrew Details in the future. Might be cool to manage and launch homebrews on external media.

Just a heads up (so you don’t make the same mistake I did!) you won’t be able to just launch an app from a hdd. As soon as your app exits, the hdds will be unmounted so now that path that nxhbl has will be invalid (ie ums0:/app.nro). The quick workaround I thought of was just to copy the nro from hdd to sd then set the path to location on sd card. This is a bit mixed because the nro could be huge so it could take a while (slow launches), it might depend on loose files like config stuff and maybe the user doesn’t want to override the app on the sd card with the one of hdd (is it exists) so maybe copying to a temp folder would be better.

but yeah not like it can’t work, just not super straight forward. But adding hdd support as a way to copy / move apps to and from storages might be a better option imo, and would still be a very nice feature

 

[duckbill007]

Yes. For app launch this should be a sysmodule.

 

[DarkMatterCore]

Even though the BOT driver from the library could easily be modified to work under a custom sysmodule/service context, lots of benefits would be lost, like proper support for multiple partitions, faster transfers and support for other filesystems beyond FAT, which are really good QoL features if I do say so myself.

I agree that certain benefits would be gained instead, like the possibility to generate FsFileSystem objects that can be used straight with libnx functions. But personally, the trade-off is just too big - it's better to let the driver live in userland (and Scires agrees that it's better to provide a library for this stuff).