SXOS and atmosphere sharing emunand + android 10 accessing FAT32

Inaki · Apr 26, 2021

Further ideas:

I know hekate is SXOS agnostic and that's fine. The thing is the SD partitioning part could add a checkbox or so to create this contiguous file based SXOS just after the partitioning is done and before the previous files are restored. These would just be the file entries describing the /SXOS/Emunand/... files contiguously and maybe a template /emuMMC/emummc.ini with the sector= pointing to the correct sector. Then we would just use EmuTool to restore our emunand backup to the space starting from that sector.

Another possibility is creating a windows program that does this and we would use it after partitioning and installing android and/or linux..

One last thing, any idea why this SD partitioning in hekate creates FAT32 partitions with a cluster size of 64KB instead of the suposedly preferred 32KB clusters ?

lordelan · Apr 26, 2021

You finally did it. I can't thank you enough! Been waiting for something like this for months especially since Android 10 was officially released to the public.
I asked many skilled people at many places (gbatemp, Discord, YouTube, couple of other forums, Telegram...) but no one was able to achieve what you just did.
As I told (and promised) you, this shall be finally yours:

Can't wait for my big card to arrive so I can test this!

Edit: Regarding your second post I guess it would be our best bet to mention @CTCaer here.

Edit 2: I also want to credit @evil_santa here as he helped me a lot understanding things under the surface and especially pointed out some technical difficulties that prevented us from achieving this.

Inaki · Apr 26, 2021

lordelan said:
You finally did it. I can't thank you enough! Been waiting for something like this for months especially since Android 10 was officially released to the public.
I asked many skilled people at many places (gbatemp, Discord, YouTube, couple of other forums, Telegram...) but no one was able to achieve what you just did.
As I told (and promised) you, this shall be finally yours:

Can't wait for my big card to arrive so I can test this!

Edit: Regarding your second post I guess it would be our best bet to mention @CTCaer here.

Edit 2: I also want to credit @evil_santa here as he helped me a lot understanding things under the surface and especially pointed out some technical difficulties that prevented us from achieving this.

Lol, thank you, is not that much though

lordelan · Apr 26, 2021

Inaki said:
Lol, thank you, is not that much though

Believe me, it is.
I can finally enjoy Android 10 on my Switch without loosing the ability to mount XCIs from USB (SX OS) and without the need to swap SD cards (god I hate such setups where this is my only choice).

Inaki · Apr 26, 2021

lordelan said:
You finally did it. I can't thank you enough! Been waiting for something like this for months especially since Android 10 was officially released to the public.
I asked many skilled people at many places (gbatemp, Discord, YouTube, couple of other forums, Telegram...) but no one was able to achieve what you just did.
As I told (and promised) you, this shall be finally yours:

Can't wait for my big card to arrive so I can test this!

Edit: Regarding your second post I guess it would be our best bet to mention @CTCaer here.

Edit 2: I also want to credit @evil_santa here as he helped me a lot understanding things under the surface and especially pointed out some technical difficulties that prevented us from achieving this.

Yeah, but I don't know, @CTCaer probably won't like the idea, since this is very sxos specific and its future is not even clear, so... I don't know if I will be able to do the mod myself, I could try I guess, that's the good thing about open source

I did have a few looks into hekate sources because I'd like to do an SXOS chainloader+patcher and I think RetroReloaded is closed source. I just want a minimalistic chainloader with framebuffer access ( that's the code part I looked into the most ) so this may be a good moment to play with hekate sources, all with all the respect to the developers.

lordelan · Apr 26, 2021

Inaki said:
Yeah, but I don't know, @CTCaer probably won't like the idea, since this is very sxos specific and its future is not even clear, so... I don't know if I will be able to do the mod myself, I could try I guess, that's the good thing about open source

I did have a few looks intp hekate because I'd like to do an SXOS chainloader+patcher and I think RetroReloaded is closed source. I just want a minimalistic chainloader with framebuffer access ( that's the code part I looked into the most ) so this may be a good moment to play with hejate sources, all with all the respect to the developers.

Yeah I too don't think CTCaer is going to "waste" any time on this but I figured I'd mention him at least once.
He still would accept a github pull request though I think, as long as it doesn't destroy other things or contains "unnecessary code".
His "migrating existing emuMMC" feature was it in the first place that enabled Atmosphère users to use an already existing partition based emuNAND from SX OS so I don't see what he would have against a proper pull request if you're able to provide one at some point in the future.

However, this is already satisfying enough. I looked through all the steps of your guide by now and have to admit it's still a genius approach to trick Atmosphère into using SX OS's filebased emuNAND and treat it like a partition based "only" by making sure it is continuous and putting the starting sector into the emummc.ini as you would for a partition based emuNAND.
So the medal is well deserved.
Speaking of deserving things, I really appreciate all the mentions you gave me (YouTube, your PDF, this thread) but I don't really deserve them since I didn't really provide anything to your nice solution at least. Still thanks for that.

As for RetroReloaded (which I didn't ever use), you might find what you're looking for in the code of "Argon NX" too. There's even a forked version called Argon NX-SE now as I saw recently. Hekate's source code however should help you too with that.

Edit: I'm still not sure if I'll manage to be able to obtain the correct offset calculation for my setup on the first try I'll be making but we'll see. If not I'll be coming back to this thread and you might point me where I miscalculated.

Inaki · Apr 26, 2021

lordelan said:
Yeah I too don't think CTCaer is going to "waste" any time on this but I figured I'd mention him at least once.
He still would accept a github pull request though I think, as long as it doesn't destroy other things or contains "unnecessary code".
His "migrating existing emuMMC" feature was it in the first place that enabled Atmosphère users to use an already existing partition based emuNAND from SX OS so I don't see what he would have against a proper pull request if you're able to provide one at some point in the future.

However, this is already satisfying enough. I looked through all the steps of your guide by now and have to admit it's still a genius approach to trick Atmosphère into using SX OS's filebased emuNAND and treat it like a partition based "only" by making sure it is continuous and putting the starting sector into the emummc.ini as you would for a partition based emuNAND.
So the medal is well deserved.
Speaking of deserving things, I really appreciate all the mentions you gave me (YouTube, your PDF, this thread) but I don't really deserve them since I didn't really provide anything to your nice solution at least. Still thanks for that.

As for RetroReloaded (which I didn't ever use), you might find what you're looking for in the code of "Argon NX" too. There's even a forked version called Argon NX-SE now as I saw recently. Hekate's source code however should help you too with that.

Edit: I'm still not sure if I'll manage to be able to obtain the correct offset calculation for my setup on the first try I'll be making but we'll see. If not I'll be coming back to this thread and you might point me where I miscalculated.

No worries, I can help you calculate it. Maybe I can write a small program to calculate and check that "ordered and contiguous" thing, step 4 is the hardest, so... EDIT: or, better, I may try writing a program to generate the file based emunand myself, ensuring order and continuity and creating the emuMMC/emummc.ini automatically... hmmm...

About a PR, yes, I thought about that, I coukd first try to do it myself and then, maybe do a PR.

As for crediting you, well, you explained it to me and encouraged me to do it, so...

lordelan · Apr 26, 2021

I wonder if we can even resize an emuNAND created by your guide like this:
https://gbatemp.net/threads/how-to-resize-your-emunand-mmc.587024/
Not a must have with a > 500 gb card but still everything you install inside the emuNAND goes to the SD card, not the "internal memory" of the emuNAND (which is still on the SD, yes, but inside the reserved emuNAND space).

Edit: Doesn't really matter though. I figured I can just move a few games I've installed to SD to the "internal memory" of the emuNAND so it goes to a space that is "already wasted" anyway but frees space on the SD in total.

Inaki · Apr 26, 2021

lordelan said:
I wonder if we can even resize an emuNAND created by your guide like this:
https://gbatemp.net/threads/how-to-resize-your-emunand-mmc.587024/
Not a must have with a > 500 gb card but still everything you install inside the emuNAND goes to the SD card, not the "internal memory" of the emuNAND (which is still on the SD, yes, but inside the reserved emuNAND space).

Edit: Doesn't really matter though. I figured I can just move a few games I've installed to SD to the "internal memory" of the emuNAND so it goes to a space that is "already wasted" anyway but frees space on the SD in total.

I thought about it too...

Two things here:

1) Technically, it depends on wether SX OS checks the existence of all the /SXOS/Emunand/ files. For hidden partition style emunand there is no size checks but what would it check since there is no files... it could check the USER partition size but there's no benefit for that. For file based emunand my guess is it won't complain if there's at least a full.00.bin file... but I don't know, if I programmed such a thing I would probably calculate all the physical offsets on boot time so... maybe it does need all the pieces. I was planning to check this and other things and I will. I think 60% vs 40% for we can cut vs we can not cut

EDIT: just had another couple of ideas:

- even if it cheks for existence of the files we could set their size to 0, so, if they have implemented such a precalculation, if they don't check the size, we could just leave 0 sized files.

- if they check sizes we can make the unneeded parts point to the full.00.bin sectors for example. This would make SXOS checks happy. I am waaay more certain that the developers didn't include checks against this trick

Also, and finally, this is not a problem since, with a reduced USER partition, Horizon OS will only access so far so the files above the actual emunand sizes won't be used at all.

So, even without testing I'd say probability of success is 95%

2) Keep in mind that, if this works, if you make, let's say a 6GB emunand, you would delete full.02.bin to full.07.bin and truncate full.01.bin to 2GB. And that would be fine, but if you then wanted to grow the emunand you would need to start over. You can think this is the same as with a hidden partition style emunand but it is not. In that case you can use Minitool Partition Wizard to shrink and/or move the FAT32 partition to make space for the bigger emunand, but in this case, once you lose "ordered and contiguous" length, you can't recover it anymore. A program that finds a given length contiguous space and creates the emunand files there would be desirable, both for this and in general...

Inaki · Apr 26, 2021

I found this, very promising, even if it happens to only work on given files, I could add code to do the same for multiple file sets. I need to study the code.

https://github.com/ike9000e/CFSpace

EDIT: it uses http://elm-chan.org/fsw/ff/00index_e.html which I'd rather use directly. EDIT2: amazing library and amazing FAT32 info on that site.

Inaki · Apr 27, 2021

Inaki said:
I thought about it too...

Two things here:

1) Technically, it depends on wether SX OS checks the existence of all the /SXOS/Emunand/ files. For hidden partition style emunand there is no size checks but what would it check since there is no files... it could check the USER partition size but there's no benefit for that. For file based emunand my guess is it won't complain if there's at least a full.00.bin file... but I don't know, if I programmed such a thing I would probably calculate all the physical offsets on boot time so... maybe it does need all the pieces. I was planning to check this and other things and I will. I think 60% vs 40% for we can cut vs we can not cut

EDIT: just had another couple of ideas:

- even if it cheks for existence of the files we could set their size to 0, so, if they have implemented such a precalculation, if they don't check the size, we could just leave 0 sized files.

- if they check sizes we can make the unneeded parts point to the full.00.bin sectors for example. This would make SXOS checks happy. I am waaay more certain that the developers didn't include checks against this trick Also, and finally, this is not a problem since, with a reduced USER partition, Horizon OS will only access so far so the files above the actual emunand sizes won't be used at all.

So, even without testing I'd say probability of success is 95%

2) Keep in mind that, if this works, if you make, let's say a 6GB emunand, you would delete full.02.bin to full.07.bin and truncate full.01.bin to 2GB. And that would be fine, but if you then wanted to grow the emunand you would need to start over. You can think this is the same as with a hidden partition style emunand but it is not. In that case you can use Minitool Partition Wizard to shrink and/or move the FAT32 partition to make space for the bigger emunand, but in this case, once you lose "ordered and contiguous" length, you can't recover it anymore. A program that finds a given length contiguous space and creates the emunand files there would be desirable, both for this and in general...

Bad news on this front, it seems I sinned for being overconfident

The thing is I tried the ideas for cutting off the excess of full.xx.bin files ( tried just deleting, zero-sizing and even making them point to full.00.bin ) but none of them work, black screen.

The emunand itself is fine, atmosphere works perfectly on it, but SXOS doesn't.

This means we are stuck with full sized emunand for now...

Also, I will be updating the guide for the backups and restores of the "ordered and contiguous" emunand since I noticed some issues using EmuTool...

lordelan · Apr 27, 2021

Great insights. Interesting indeed. That could work and safe some extra space.

Edit: Oh, just saw your latest post. Well okay then ... guess that's how it is. :/

Inaki · Apr 27, 2021

lordelan said:
Great insights. Interesting indeed. That could work and safe some extra space.

Edit: Oh, just saw your latest post. Well okay then ... guess that's how it is. :/

Yeah

I will try a bit more but...

The last possibility here would be to just leave all the full.xx.bin file entries intact and then create other file entries to alias those unused sectors or mark them as free, I don't know if it will work but it would be way more unprobable that SXOS checks go that far.

lordelan · Apr 27, 2021

Inaki said:
Yeah I will try a bit more but...

The last possibility here would be to just leave all the full.xx.bin file entries intact and then create other file entries to alias those unused sectors or mark them as free, I don't know if it will work but it would be way more unprobable that SXOS checks go that far.

No it's fine, really. Don't waste any further time on this. You've already done enough.
It's no problem to move a few games like Zelda and MK8D to the "internal memory" in order to save some space.

Inaki · Apr 27, 2021

lordelan said:
No it's fine, really. Don't waste any further time on this. You've already done enough.
It's no problem to move a few games like Zelda and MK8D to the "internal memory" in order to save some space.

A few hours ago, on my lunch break, I tested with an SD card that has this file based shared emunand thing and just restored an 8GB resized emunand to it, I didn't delete nor edit any full.xx.bin entry nor anything, just let it intact.

Well, atmosphere boots fine to emunand but SXOS black screens.

So, that means it is not an emunand file check issue but some internal emunand content check issue; maybe it just expects the emunand to be full sized ( or bigger than 8GB at least ).

lordelan · Apr 27, 2021

Inaki said:
A few hours ago, on my lunch break, I tested with an SD card that has this file based shared emunand thing and just restored an 8GB resized emunand to it, I didn't delete nor edit any full.xx.bin entry nor anything, just let it intact.

Well, atmosphere boots fine to emunand but SXOS black screens.

So, that means it is not an emunand file check issue but some internal emunand content check issue; maybe it just expects the emunand to be full sized ( or bigger than 8GB at least ).

Interesting. Thanks for testing! =)

Inaki · Apr 27, 2021

Updated the guide document to version 1.1:

// Changed backing up and restoring procedures.
// Added Tools & Software: section before Steps: section and removed Tools: section at the end.
// Fixed credits.

XxBakasuraxX · Apr 28, 2021

How much space do i need for sx os and atmosphere? Should i skip some steps if i want to do only for those cfw?

Inaki · Apr 28, 2021

XxBakasuraxX said:
How much space do i need for sx os and atmosphere? Should i skip some steps if i want to do only for those cfw?

This creates an emunand on files in the fat32 partition so 0, you don't need to create an emunand when partitioning the SD card. The file based 29GB emunand will be used by both sxos and atmosphere. You won't have an emunand partition.

I may need to update the document to give more background info and explain this... it seems it is a bit cryptic, maybe ?

Attachments

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Similar threads

Popular threads in this forum