1. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    In a the tutorial thread https://gbatemp.net/threads/quickes...emunand-to-sxos-style-microsd-emunand.586474/, fellow user @lordelan explained the problems leading to the impossibility of having SXOS and atmosphere share the same emunand when using them on a switch which also has android 10 installed.

    Basically, the problem is the combination of these three facts:
    • android 10 requires a GPT partition layout, which uses sectors 1,2,… to describe the partitions.
    • SXOS hidden partition style emunand is stored in the fixed sectors 1 ( TX NAND mark ) and starting on sector 2 ( actual emunand contents ).
    • Atmosphere can describe SXOS hidden partition style emunands and use them, but it needs the contents to be contiguous and it doesn’t support referencing a file based SXOS emunand ( of course some modifications to atmosphere could give this support and maybe some FAT32 file descriptor trickery would also do the trick, but that would be more work, I think ).
    So, the trick that turned out to be possible was: what if…
    • We create an SXOS file based style emunand STRUCTURE and arrange file data in the FAT32 partition so that the file entries are describing data for each file, yes, as it would be the case for a file based SXOS emunand, BUT we also make sure that data is contiguous on disk.
    • We create a hekate+atmosphere emummc.ini to use a hidden partition style emunand that uses sector=<sector where the “Ordered and Contiguous” file based SXOS emunand contents start>.
    And, after some research and testing, we got it working :)
    _____________________

    Updated document to version 1.1:

    // Changed backing up and restoring procedures.
    // Added Tools & Software: section before Steps: section and removed Tools: section at the end.
    // Fixed credits.
     

    Attached Files:

    Last edited by Inaki, Apr 27, 2021
    bulbasouri, evil_santa and lordelan like this.
  2. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Further ideas:

    I know hekate is SXOS agnostic and that's fine. The thing is the SD partitioning part could add a checkbox or so to create this contiguous file based SXOS just after the partitioning is done and before the previous files are restored. These would just be the file entries describing the /SXOS/Emunand/... files contiguously and maybe a template /emuMMC/emummc.ini with the sector= pointing to the correct sector. Then we would just use EmuTool to restore our emunand backup to the space starting from that sector.

    Another possibility is creating a windows program that does this and we would use it after partitioning and installing android and/or linux..

    One last thing, any idea why this SD partitioning in hekate creates FAT32 partitions with a cluster size of 64KB instead of the suposedly preferred 32KB clusters ?
     
    lordelan likes this.
  3. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    You finally did it. I can't thank you enough! Been waiting for something like this for months especially since Android 10 was officially released to the public.
    I asked many skilled people at many places (gbatemp, Discord, YouTube, couple of other forums, Telegram...) but no one was able to achieve what you just did.
    As I told (and promised) you, this shall be finally yours:

    [​IMG]

    Can't wait for my big card to arrive so I can test this!

    Edit: Regarding your second post I guess it would be our best bet to mention @CTCaer here.

    Edit 2: I also want to credit @evil_santa here as he helped me a lot understanding things under the surface and especially pointed out some technical difficulties that prevented us from achieving this.
     
    Last edited by lordelan, Apr 26, 2021
    evil_santa and Inaki like this.
  4. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Lol, thank you, is not that much though :)
     
    lordelan likes this.
  5. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    Believe me, it is.
    I can finally enjoy Android 10 on my Switch without loosing the ability to mount XCIs from USB (SX OS) and without the need to swap SD cards (god I hate such setups where this is my only choice).
     
    Inaki likes this.
  6. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Yeah, but I don't know, @CTCaer probably won't like the idea, since this is very sxos specific and its future is not even clear, so... I don't know if I will be able to do the mod myself, I could try I guess, that's the good thing about open source :)

    I did have a few looks into hekate sources because I'd like to do an SXOS chainloader+patcher and I think RetroReloaded is closed source. I just want a minimalistic chainloader with framebuffer access ( that's the code part I looked into the most ) so this may be a good moment to play with hekate sources, all with all the respect to the developers.
     
    Last edited by Inaki, Apr 26, 2021
  7. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    Yeah I too don't think CTCaer is going to "waste" any time on this but I figured I'd mention him at least once.
    He still would accept a github pull request though I think, as long as it doesn't destroy other things or contains "unnecessary code".
    His "migrating existing emuMMC" feature was it in the first place that enabled Atmosphère users to use an already existing partition based emuNAND from SX OS so I don't see what he would have against a proper pull request if you're able to provide one at some point in the future.

    However, this is already satisfying enough. I looked through all the steps of your guide by now and have to admit it's still a genius approach to trick Atmosphère into using SX OS's filebased emuNAND and treat it like a partition based "only" by making sure it is continuous and putting the starting sector into the emummc.ini as you would for a partition based emuNAND.
    So the medal is well deserved.
    Speaking of deserving things, I really appreciate all the mentions you gave me (YouTube, your PDF, this thread) but I don't really deserve them since I didn't really provide anything to your nice solution at least. Still thanks for that.

    As for RetroReloaded (which I didn't ever use), you might find what you're looking for in the code of "Argon NX" too. There's even a forked version called Argon NX-SE now as I saw recently. Hekate's source code however should help you too with that. :)

    Edit: I'm still not sure if I'll manage to be able to obtain the correct offset calculation for my setup on the first try I'll be making but we'll see. If not I'll be coming back to this thread and you might point me where I miscalculated. :P
     
    Last edited by lordelan, Apr 26, 2021
    Inaki likes this.
  8. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    No worries, I can help you calculate it. Maybe I can write a small program to calculate and check that "ordered and contiguous" thing, step 4 is the hardest, so... EDIT: or, better, I may try writing a program to generate the file based emunand myself, ensuring order and continuity and creating the emuMMC/emummc.ini automatically... hmmm...

    About a PR, yes, I thought about that, I coukd first try to do it myself and then, maybe do a PR.

    As for crediting you, well, you explained it to me and encouraged me to do it, so... :D
     
    Last edited by Inaki, Apr 26, 2021
    lordelan likes this.
  9. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    I wonder if we can even resize an emuNAND created by your guide like this:
    https://gbatemp.net/threads/how-to-resize-your-emunand-mmc.587024/
    Not a must have with a > 500 gb card but still everything you install inside the emuNAND goes to the SD card, not the "internal memory" of the emuNAND (which is still on the SD, yes, but inside the reserved emuNAND space).

    Edit: Doesn't really matter though. I figured I can just move a few games I've installed to SD to the "internal memory" of the emuNAND so it goes to a space that is "already wasted" anyway but frees space on the SD in total. :D
     
    Last edited by lordelan, Apr 26, 2021
  10. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    I thought about it too...

    Two things here:

    1) Technically, it depends on wether SX OS checks the existence of all the /SXOS/Emunand/ files. For hidden partition style emunand there is no size checks but what would it check since there is no files... it could check the USER partition size but there's no benefit for that. For file based emunand my guess is it won't complain if there's at least a full.00.bin file... but I don't know, if I programmed such a thing I would probably calculate all the physical offsets on boot time so... maybe it does need all the pieces. I was planning to check this and other things and I will. I think 60% vs 40% for we can cut vs we can not cut :)

    EDIT: just had another couple of ideas:

    - even if it cheks for existence of the files we could set their size to 0, so, if they have implemented such a precalculation, if they don't check the size, we could just leave 0 sized files.

    - if they check sizes we can make the unneeded parts point to the full.00.bin sectors for example. This would make SXOS checks happy. I am waaay more certain that the developers didn't include checks against this trick ;) Also, and finally, this is not a problem since, with a reduced USER partition, Horizon OS will only access so far so the files above the actual emunand sizes won't be used at all.

    So, even without testing I'd say probability of success is 95% :D

    2) Keep in mind that, if this works, if you make, let's say a 6GB emunand, you would delete full.02.bin to full.07.bin and truncate full.01.bin to 2GB. And that would be fine, but if you then wanted to grow the emunand you would need to start over. You can think this is the same as with a hidden partition style emunand but it is not. In that case you can use Minitool Partition Wizard to shrink and/or move the FAT32 partition to make space for the bigger emunand, but in this case, once you lose "ordered and contiguous" length, you can't recover it anymore. A program that finds a given length contiguous space and creates the emunand files there would be desirable, both for this and in general...
     
    Last edited by Inaki, Apr 26, 2021
    lordelan likes this.
  11. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Last edited by Inaki, Apr 27, 2021
    lordelan likes this.
  12. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Bad news on this front, it seems I sinned for being overconfident :(

    The thing is I tried the ideas for cutting off the excess of full.xx.bin files ( tried just deleting, zero-sizing and even making them point to full.00.bin ) but none of them work, black screen.

    The emunand itself is fine, atmosphere works perfectly on it, but SXOS doesn't.

    This means we are stuck with full sized emunand for now...

    Also, I will be updating the guide for the backups and restores of the "ordered and contiguous" emunand since I noticed some issues using EmuTool...
     
  13. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    Great insights. Interesting indeed. That could work and safe some extra space. :toot:

    Edit: Oh, just saw your latest post. Well okay then ... guess that's how it is. :/
     
    Inaki likes this.
  14. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Yeah :( I will try a bit more but...

    The last possibility here would be to just leave all the full.xx.bin file entries intact and then create other file entries to alias those unused sectors or mark them as free, I don't know if it will work but it would be way more unprobable that SXOS checks go that far.
     
  15. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    No it's fine, really. Don't waste any further time on this. You've already done enough.
    It's no problem to move a few games like Zelda and MK8D to the "internal memory" in order to save some space. :)
     
    Inaki likes this.
  16. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    A few hours ago, on my lunch break, I tested with an SD card that has this file based shared emunand thing and just restored an 8GB resized emunand to it, I didn't delete nor edit any full.xx.bin entry nor anything, just let it intact.

    Well, atmosphere boots fine to emunand but SXOS black screens.

    So, that means it is not an emunand file check issue but some internal emunand content check issue; maybe it just expects the emunand to be full sized ( or bigger than 8GB at least ).
     
    lordelan likes this.
  17. lordelan

    lordelan GBAtemp Psycho!
    Member

    Joined:
    Jan 4, 2015
    Messages:
    4,978
    Country:
    Germany
    Interesting. Thanks for testing! =)
     
    Inaki likes this.
  18. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    Updated the guide document to version 1.1:


    // Changed backing up and restoring procedures.
    // Added Tools & Software: section before Steps: section and removed Tools: section at the end.
    // Fixed credits.
     
    lordelan likes this.
  19. XxBakasuraxX

    XxBakasuraxX Newbie
    Newcomer

    Joined:
    Apr 8, 2021
    Messages:
    7
    Country:
    Argentina
    How much space do i need for sx os and atmosphere? Should i skip some steps if i want to do only for those cfw?
     
  20. Inaki

    OP Inaki GBAtemp Regular
    Member

    Joined:
    Jan 23, 2014
    Messages:
    256
    Country:
    This creates an emunand on files in the fat32 partition so 0, you don't need to create an emunand when partitioning the SD card. The file based 29GB emunand will be used by both sxos and atmosphere. You won't have an emunand partition.

    I may need to update the document to give more background info and explain this... it seems it is a bit cryptic, maybe ?
     
    Last edited by Inaki, Apr 28, 2021
Draft saved Draft deleted
Loading...

Hide similar threads Similar threads with keywords - atmosphere, accessing, sharing