Homebrew Homebrew app [Release] ftpsrv - A small ftp server

[TotalJustice]

The problem with case sensitivity is windows files system is case insensitive. So these four files will ended up with only two as the other two will be overwritten if the directory is copied.

View attachment 474514
The ':' character is replace by '_' perhaps by total commander but on the way back it will be a problem.

Maybe the reason why this line gives uid of 0,0 on my system maybe the reason why thomas save directory is empty
if (R_SUCCEEDED(accountTrySelectUserWithoutInteraction(&uid, false))) {

can you enable logs in the config.ini and do save: -> open thomas -> exit, then upload the /config/ftpsrv/log.ini here, maybe it will show some errors, hopefully anyway.

Post automatically merged: Dec 6, 2024

I believe Mac is also case insensitive.
Everything on windows is.

I suppose by and large all users will have this problem

Post automatically merged: Dec 6, 2024

@TotalJustice is it possible to support directory time stamp?

Post automatically merged: Dec 6, 2024

One other thing, in my case all of link's save has directory listing but some of them are empty with ftpsrv but they are not empty

no directory timestamp is not possible in hos, only files.

 

[TomSwitch]

can you enable logs in the config.ini and do save: -> open thomas -> exit, then upload the /config/ftpsrv/log.ini here, maybe it will show some errors, hopefully anyway.

Post automatically merged: Dec 6, 2024

no directory timestamp is not possible in hos, only files.

I was wrong about link save being empty, it is actually empty, DBI will show a size that seems to be perhaps a reserved size, when I choose it it showed blank

 

[TotalJustice]

i added cache, device and system save mounting in the latest commit. the application version now has hdd mounting too, thanks to https://github.com/DarkMatterCore/libusbhsfs.

as for the save naming issues...i've thought about this quite a bit and i can't come up with a good solution that both works with app/sys versions due to memory contraints. the zip idea is likely the best / only reasonable option. theres a few issues with using zips however:

• sysmod does not have much memory to work with, i'd likely have to write my own zip code (using store, not deflate).
• handling the upload of a zip save file.

i've had a quite look at the zip spec and i think(?) it can be created on the fly. it has local headers for each file, then the file data comes after and so on for every file. this would work with very limited memory. although theres that bit about the central_header at the end of the file, that might prove problematic.

i could work around this by creating the zip to file, then upload the zip by reading from file. kinda like using file as swap memory horribly inefficient, but it would work i guess. or write the central header to file, create the rest of zip on the fly, then upload the central_header as the last chunk.

handling the upload of zips, ie, extracting them into saves is 100% not happening tho. it would work fine with zips created by ftpsrv, but literally everything else will create / modify zips to use deflate (compression). its too much of a pain to handle. plus, i assume those messing with saves will be using ftpsrv to create save backups in the background, rather than restoring them. use jksv for that.

(why did nintendo allow for case sensitive save data...)

 

[JK_]

@TotalJustice
I'm assuming you know about the journal space thing too, right? 99.9% of games are lazy and allocate the default 32MB for both storage and journal space, but there's some that don't and trying to commit changes greater than the journal space will cause the commit to fail. New Pokemon Snap is the funnest ones to tackle, because it expands its save data a few times to fit more pictures and the save files themselves are way, way larger than its journal space will allow to commit at once.

 

[TotalJustice]

@impeeza I finished the zip code this morning, aside from it missing crc32, it all works what do you think the path structure should be? I was thinking:
/save:/zips/list_of_games.zip
And
/save:/games/game/save_files

 

[AlM2023]

@impeeza I finished the zip code this morning, aside from it missing crc32, it all works what do you think the path structure should be? I was thinking:
/save:/zips/list_of_games.zip
And
/save:/games/game/save_files

Can't wait for the new release!

 

[duckbill007]

While waiting you can use DBI for access saves through ftp.

 

[impeeza]

@impeeza I finished the zip code this morning, aside from it missing crc32, it all works what do you think the path structure should be? I was thinking:
/save:/zips/list_of_games.zip
And
/save:/games/game/save_files

For me /saves/SGzips/GameXYZ.zip will be nice

 

[TomSwitch]

@TotalJustice I tried with my other Switch, there is no problem seeing saves on both profiles on this Switch with the save: directory on ftpsrv. I tried with the link profile on the previous Switch I reported that link was ok and thomas was not and I found that a new game save I created today don't show up on link profile, the directory is empty while it is not. My fork of your first release and DBI both show there is file there.

If you tell me where to hack I can help to do some hacking to see if some discovery can be made on this mystery since it can be seen on my first Switch.

 

[TotalJustice]

@TotalJustice I tried with my other Switch, there is no problem seeing saves on both profiles on this Switch with the save: directory on ftpsrv. I tried with the link profile on the previous Switch I reported that link was ok and thomas was not and I found that a new game save I created today don't show up on link profile, the directory is empty while it is not. My fork of your first release and DBI both show there is file there.

If you tell me where to hack I can help to do some hacking to see if some discovery can be made on this mystery since it can be seen on my first Switch.

is "/save:/thomas [id]/" empty? or is "/save:/thomas [id]/game [id]/" empty? if its the latter, then i have literally no idea why its not working for you

 

[TomSwitch]

is "/save:/thomas [id]/" empty? or is "/save:/thomas [id]/game [id]/" empty? if its the latter, then i have literally no idea why its not working for you

First one, is empty like there is no save for this profile

 

[TotalJustice]

First one, is empty like there is no save for this profile
View attachment 474951
View attachment 474943

can you try this build please? it should list all the saves and more info about them. please post a screen cap of your client, or the log file. you will likely not be able to open the saves themselves, but they will at least be shown. if this works, ill know what to fix.

 

[TomSwitch]

can you try this build please? it should list all the saves and more info about them. please post a screen cap of your client, or the log file. you will likely not be able to open the saves themselves, but they will at least be shown. if this works, ill know what to fix.

both files and zips directory is empty for profile thomas

 

[TotalJustice]

both files and zips directory is empty for profile thomas

i give up

 

[TomSwitch]

i give up

Breaking news !!
The problem disappeared when I rename my profile from "thomas" to "tom"
I then rename if back to "thomas" and it is still working
I suspect it has something to do with writing than to do with the name but there is no way to verify since it is a one time thing

The mystery is not yet solved but perhaps that give some clue about what it may be. I guess no more means to test hypothesis until someone else get this problem.

Post automatically merged: Dec 9, 2024

uid changed.
This from earlier log file:
257 "/save:/thomas [10002DE2D2B002FBD2E9A2531D5FB38F]" opened.
Also now the second half of uid is the same between the two profile
257 "/save:/thomas [100027500A4F76EF 6547D9A94CB7A593]" opened.
257 "/save:/link [10015BF6D39365DB 6547D9A94CB7A593]" opened.
Does this ring any bell?

 

[duckbill007]

I have such issues in DBI with inforeader, so I switched to reading raw files.

 

[AlM2023]

Where can i download the latest and test, in the gethub still version 0.2.0!?

 

[TotalJustice]

i've released version 1.0.0, many changes and features added, mainly specific to the switch:

- added OPTS cmd, allows clients to support UTF8.
- replace sscanf with snprintf / strtoull.
- added session timeout, can be set in config.ini.
- make all sockets non-blocking. previously, accept and connect could block.
- remove virtual decive code from ftp server, instead, have the vfs backend handle any virtual devices.
- fix PORT polling.
- [Unistd] fix socket closing, see commit for details.
- [Nx] sysmod is 3x faster (10MiB read, 6MiB write).
- [Nx] option to set different port for app/sysmod in config.ini.
- [Nx] added save listing / mounting.
- [Nx] fix sysmod sockets running out of memory, causing a hang.
- [Nx] led now flashes during data transfer.
- [Nx] close the log file on exit in app version.
- [Nx] added content storage mounting.
- [Nx] added gamecard mounting, allows you to dump cert, ticket and ncas.
- [Nx] added bis storage file support.
- [Nx] added hdd mounting, only available in app version.
- [Nx] app romfs mounting, if launched via title override (holding R), you can mount the romfs of the current title, only available in app version.
- [Nx] qlaunch (home menu) romfs mounting, only available in app version.
- [Nx] offers saves as zip files.

ftpsrv-sysmod uses a lot more memory than the last version. if you ever want to check how much memory a system uses, you can use hactool.
`hactool -t pfs0 exefs.nsp --outdir out` and finally `hactool -t nso out/main`.
this will print the section sizes, text being code, bss being static memory etc.
the last value printed in hactool is the total size used for the sysmod (code + memory).
for example, sys-ftpd:
NSO0:
Build Id: 901BDB59F8548B5A3E06C31ED96610FB64298F54000000000000000000000000
Sections:
.text: 00000000-00033000
.rodata: 00033000-00044000
.rwdata: 00044000-00049000
.bss: 00049000-000fa000

and ftpsrv:
NSO0:
Build Id: 8A929DF7352A88C63E844544B7975A38F3AA37A1000000000000000000000000
Sections:
.text: 00000000-00028000
.rodata: 00028000-00032000
.rwdata: 00032000-00036000
.bss: 00036000-00094000

these values are in hex, so ftpsrv uses 592kib of memory. this is quite a lot more than the previous release, however it
was needed to prevent random freezes due to the sockets running out of memory. it should be stable now. the stress test has
been using rclone + filezilla syncing 1k large/small files.

https://github.com/ITotalJustice/ftpsrv/releases/tag/1.0.0

 

[AlM2023]

i've released version 1.0.0, many changes and features added, mainly specific to the switch:

- added OPTS cmd, allows clients to support UTF8.
- replace sscanf with snprintf / strtoull.
- added session timeout, can be set in config.ini.
- make all sockets non-blocking. previously, accept and connect could block.
- remove virtual decive code from ftp server, instead, have the vfs backend handle any virtual devices.
- fix PORT polling.
- [Unistd] fix socket closing, see commit for details.
- [Nx] sysmod is 3x faster (10MiB read, 6MiB write).
- [Nx] option to set different port for app/sysmod in config.ini.
- [Nx] added save listing / mounting.
- [Nx] fix sysmod sockets running out of memory, causing a hang.
- [Nx] led now flashes during data transfer.
- [Nx] close the log file on exit in app version.
- [Nx] added content storage mounting.
- [Nx] added gamecard mounting, allows you to dump cert, ticket and ncas.
- [Nx] added bis storage file support.
- [Nx] added hdd mounting, only available in app version.
- [Nx] app romfs mounting, if launched via title override (holding R), you can mount the romfs of the current title, only available in app version.
- [Nx] qlaunch (home menu) romfs mounting, only available in app version.
- [Nx] offers saves as zip files.

ftpsrv-sysmod uses a lot more memory than the last version. if you ever want to check how much memory a system uses, you can use hactool.
`hactool -t pfs0 exefs.nsp --outdir out` and finally `hactool -t nso out/main`.
this will print the section sizes, text being code, bss being static memory etc.
the last value printed in hactool is the total size used for the sysmod (code + memory).
for example, sys-ftpd:
NSO0:
Build Id: 901BDB59F8548B5A3E06C31ED96610FB64298F54000000000000000000000000
Sections:
.text: 00000000-00033000
.rodata: 00033000-00044000
.rwdata: 00044000-00049000
.bss: 00049000-000fa000

and ftpsrv:
NSO0:
Build Id: 8A929DF7352A88C63E844544B7975A38F3AA37A1000000000000000000000000
Sections:
.text: 00000000-00028000
.rodata: 00028000-00032000
.rwdata: 00032000-00036000
.bss: 00036000-00094000

these values are in hex, so ftpsrv uses 592kib of memory. this is quite a lot more than the previous release, however it
was needed to prevent random freezes due to the sockets running out of memory. it should be stable now. the stress test has
been using rclone + filezilla syncing 1k large/small files.

https://github.com/ITotalJustice/ftpsrv/releases/tag/1.0.0

Awesome! I have a suggestion: why not separate the configuration files for ftpsrv-sysmod and ftpsrv-app? This way, we can have different settings for each. For example, in ftpsrv-sysmod, I would set mount_devices=0, but when running ftpsrv-app, I would set mount_devices=1. This allows me to have game saves mounted and manage other end points only while on the app.

 

[nitrozz]

In breeze its show version 0.2.0 how to change that