Partition based emuMMC and L4T Ubuntu on the same SD card

Deleted-172301 · Jul 29, 2019

Probably going to play around with Android later. I was banned shortly after setting this up lol. So I can probably blow away my emunand... Once my 256gb emmc comes in I’ll be playing with that to figure out how to install android/linux to it so I can dedicate my SD to only games and cfw stuff.

Android relies on several partitions in order to segment different functions and allow for easy upgradability. So system, boot, recovery, can all be overwritten fairly quickly. When any android os info is installed/upgraded. It remains static (ROM). Its the user space that changes how that os info works. It also houses ALL user/new data. A factory reset on android is basically just erasing the user partition and starting from scratch.

I’m sure if we write the 2gb image file to a 32gb partition it would work fine. You would just have to make sure the user partition uses the remaining space as thats where android installs apps and saves user data to.

Cyan · Jul 29, 2019

I analyzed the android image to get the partitions table.
it's using GPT, with a fallback FAT32 as first MBR partition.
I think L4T might work, being linux it should work with GPT. I suggested in android thread to put it on second partition so you don't have to edit the l4t boot file. android partitions should not have their starting sectors hardcoded and should rely on the table.

You won't be able to boot stock/cfw, as it'll require MBR, unless :
- the switch reads first partition, whatever the type (bypassing the fact it's EE, and using it as fallback as intended)
- you mess with the table again and change the type from EE to 0C, but it's not a conventional GPT table. some apps might fail to see the GPT located on sector 1, or might list the FAT32 partition twice.

well, without proper test we won't know, so if you have time let us know what you did

edit:
My android post: https://gbatemp.net/posts/8733850/

Deleted-172301 · Jul 29, 2019

I was able to find my flash drive so my other 128gb sd is free for testing. Going to play with it some this week. Going to try CFW, L4T and android

lordelan · Jul 29, 2019

Hi you two

1. I decided to go with Ubuntu rather than Android when the switchtoot team told me in their Discord that it won't be possible to have Android on your Horizon SD since it's GPT.
If any of you would figure out how to achieve this, it would be awesome. Cyan you already made some suggestions which sound interesting.

2. I messed up my Ubuntu once again (I failed at creating the swapfile properly and ended up with zero space).
What would be then fastest way to start over with Ubuntu?
Any dd commands or something like this?

Otherwise I would go the CloneZilla route again.

Cyan · Jul 29, 2019

I would use dd to extract the linux part of the image, then use dd (or another tool) to overwrite the partition.
you could even try to do both steps at once, but I didn't try so I can't tell you if it's working fine or not.

extract linux partition from sd.img to linux.img (Tested, working):
dd if=sd.img skip=978944 of=linux.img bs=512 count=15798272

writing the linux to sd (untested):
dd if=linux.img bs=512M of=<Disk you found with dd --info> --progress

or both at once (untested):
dd if=sd.img skip=978944 <Disk you found with dd --info> bs=512 count=15798272 --progress

note: using 1 and 2nd commands to extractlinux, then write it with 512MB block is maybe faster than doing 512Kb copy/paste (3rd command).

Some tools might be able to load the full image, display partitions located in the bin, and let you clone one to your SD's partition.
MiniPartition, or DiskGenius? I used neither, but tabzer used the later, he can confirm.

I don't know linux at all, how to set it up nor how to update or install packages. I'll probably mess things too. I guess I'll have to overwrite the partition soon enough

tabzer · Jul 29, 2019

@Cyan what do you know about gpt/mbr hybrids?

--------------------- MERGED ---------------------------

Cyan said:
Some tools might be able to load the full image, display partitions located in the bin, and let you clone one to your SD's partition.
MiniPartition, or DiskGenius? I used neither, but tabzer used the later, he can confirm.

You can extract the l4t img file using 7zip. You can then use OSFMount to mount it (to windows). Then you can use Diskgenius to clone the L4T ext4 mount to a partition that you have created. As long as it is the second partition, then boot.scr shouldn't need any changing.

Deleted-172301 · Jul 29, 2019

lordelan said:
Hi you two

1. I decided to go with Ubuntu rather than Android when the switchtoot team told me in their Discord that it won't be possible to have Android on your Horizon SD since it's GPT.
If any of you would figure out how to achieve this, it would be awesome. Cyan you already made some suggestions which sound interesting.

2. I messed up my Ubuntu once again (I failed at creating the swapfile properly and ended up with zero space).
What would be then fastest way to start over with Ubuntu?
Any dd commands or something like this?
Otherwise I would go the CloneZilla route again.

Cyan has the right idea. What I did was have a separate persistent live ubuntu boot usb available. I booted into it, plugged my sd in. Then I used Gparted to create all the needed partitions. Used the disks tool built into ubuntu to “dd” flash the 1.img contained in the L4T image to the ext4 partition. Its pretty easy, but it takes some work if you aren’t used to it.

From there you reformat the fat32 partition in windows to avoid the bs gprted does to it.

This was done after I made sure SXOS created the hidden emunand partition, and that hekate was able to boot into it.
——

I’m still really curious about creating a new bootable android partition manually. I poked the dev on XDA about writing directly to the EMMC, but I haven't gotten word on that. So details on whats involved will be minimal, but my mantra for this stuff is pretty much to avoid having to redo your SD card 500 times. (Especially because thats what I did prior to getting L4T working lol)

Cyan · Jul 29, 2019

only theoretical, I never messed with it on computers (only on Wii/vWii).
all I know is things I've read about it, and how USBLoaderGX works to detect partitions on both tables.

the GPT still contain the MBR at sector 0 for backward compatibility.
the MBR contains 4 primary partitions, with their type (FAT32, ntfs, etc.). if the type is "EE" it means there's a GPT partition table located on Sector 1.

Usually, the MBR with GPT contains a "fallback" partition in case an app is not working with GPT.
the first partition is still defined in MBR, for example the first FAT32 is both defined in MBR and GPT.

sector 0 MBR
1. type EE (instead of 0C) : FAT32 start sector 2048
2. type EE (GPT) : start sector1
3.
4.

Partition 1 is the falback, in case an old app doesn't understand GPT at all, it'll be able to at least mount the first partition, but only if "EE" type doesn't bother it. if it's checking for "0C" specifically, it'll never mount that first partition.
Some apps/tools also check the first sector of the partition to determine its real format, because some partitioning tool doesn't edit the MBR/GPT when formatting a partition to new format, leaving the old Type code mismatching its current format.

Sector 1 GPT header
some info about the GPT itself, first/last usable sector, etc.

Sector 1 to xx : up to 128 partitions
- FAT32 at sector 2048, same as defined in MBR.
- partition 2
- partition 3
etc.
until "partition info" is 0000000, meaning there's no more partitions in the table.

USBLoaderGX works like that : it mount sector0, checks if partition type is "EE", if it is then it skips the MBR and jump to sector 1 to list partitions.

if we mess with the type to fool the switch by setting the first partition type to "0C" instead of EE then :
- apps expecting GPT by checking the first partition will think there's no GPT at all and won't work.
- apps still mounting all partition and jumping to GPT at first found "EE" might list more partitions than present, listing the FAT32 twice, messing with partition numbers and being dangerous for the data integrity.

It can work if the app checks if LBA (first sector) has already been assigned to a partition and skip it

USBGX is not doing it and expect EE on MBR's first partition. I also just found a bug on usbgx sources by checking how GPT is working, could be bad...

everything is based on assumption and hope that failsafe are coded in all apps. if not, it can be done, but only if developers accord themselves on a specific protocol and create tools to manage SD partition for the Switch.
A user already tried to contact Hekate and Atmosphere team, but they didn't thought a good idea to work together in a new format (they prefer using ini files to set eMMC sectors info, instead of storing it in MBR), so I'm not sure everybody will see the idea as welcome. I feel like each team want to do it their own way, and don't try to work together to make something easy for users and compatible with everything at the same time.
I know I look like I'm giving lesson, and I don't do much. I only want to give ideas, in case someone find them useful. I'm not criticizing works and time spend on each tools. you guys do great things !

Deleted-172301 · Jul 30, 2019

After playing around a bit. I’ve concluded that this will be a bit more involved than I previously assumed.

The Hekate setup for android is found in the “hos_data” image. This is found after you extract the full android image. You’ll have to extract hos_data to get those files before you can boot. My attempt to run android from the restored partition didn't work. There are a few boot files that we might have to edit as well in order to get android working.

I also found that GPT FAT does not get read by hekate... so Cyan’s info has been confirmed. I’ll try and make a hybrid partition table later on. If we include the L4T partitions and windows FAT on the MBR side. I think we can accomplish this. I found a gentoo guide that seems to indicate that we can do a pentaboot if we try hard enough.

At the moment my SD card has about 10 partitions on it lol.

Cyan · Jul 30, 2019

GPT HOS_DATA is FAT32.
hekate should be able to mount it to read the ini file, so I thought GPT was compatible with hekate.

could you share the ini file? I'm curious to see what's inside to boot android.
official hekate doesn't contain the android boot, so it's only provided with the android image. I didn't extract files from it to get the file, so if you have it I'll be interested

thank you for your tests so far (even though it looks not correct to me, GPT FAT should be working with hekate, or else it wouldn't find the ini)
or maybe it uses the MBR (EE) FAT, which is why it's also listed there.

what did you exactly do?

another thing I didn't say about GPT : usually, a second and cloned partition table is located on the last sectors of the device. a backup table, in case the first is not readable or present. I don't know if it has its importance here. it depends which app you use and which table they check.

lordelan · Jul 30, 2019

~~Tito~~ said:
At the moment my SD card has about 10 partitions on it lol.

You're pushing it too far.

Seriously, this is a good find and a nice approach. This thread turned into one of the most interesting ones thanks to Cyan and you.
I'm almost sure you will find a way to have it on the same SD.

tabzer · Jul 30, 2019

IF only the Android image used 2 partitions (FAT32/EXT4) then it would be a lot easier. If we could bundle Android's five+ partitions into one and throw it onto a fourth partition, it could adhere to the MBR ruleset and we'd be able to boot every OS without the concerns of a hybrid mbr/gpt fragility. A wrapper of some sort.

Another hypothetical is using L4T's partition to share space with Android to help reduce the partitions necessary, or vice versa. If emunand/emummc has good success with file based systems, that is another partition we don't need.

--------------------- MERGED ---------------------------

Cyan said:
could you share the ini file? I'm curious to see what's inside to boot android.

[Switchroot Android]
payload=switchroot_android/coreboot.rom

inside switchroot_android is coreboot.rom and boot.scr

Besides the hekate ini to call it, those are the only two files that extra besides hekate/nyx in the FAT32 partition.

--------------------- MERGED ---------------------------

Alsot, boot.scr decoded:

mmc dev 1
mmc info serialno
setenv bootargs 'log_buf_len=4M access=m2 androidboot.bootreason=recovery androidboot.hardware=icosa androidboot.console=ttyGS0 console=tty0 androidboot.selinux=permissive fbcon=primary:0 androidboot.serialno='${serialno}
if gpio input 190; then
part start mmc 1 4 aistart
part size mmc 1 4 aisize
else
part start mmc 1 5 aistart
part size mmc 1 5 aisize
fi
mmc read 0x98000000 ${aistart} ${aisize}
boota 0x98000000

--------------------- MERGED ---------------------------

If I look at the sd card through a partition manager, there is a 1MB protected MBR in partition 0 a 2GB FAT32 partition in partition 1, and then the rest of the SD card is "unallocated", even though it is obviously being used. So it does appear that it is already utilizing a gpt/mbr hybdrid approach.

Maybe the easiest thing to do is to find out how the original android image (that we use to etch to SD) was populated, and redo the process with our own defined choices, maybe adding our own 2nd or 3rd partitions.

Cyan · Jul 30, 2019

I extracted the MBR and GPT tables of the 16GB image, and got the sector/size of each partition.
it's not a hybrid GPT, it's a normal GPT (with MBR first partition set to EE).
hybrid GPT contains original type (non EE) MBR partitions, which would be require for L4T/MMC. (maybe even stock/CFW)

what you need :
MBR:
1. type 0C (DOS FAT32), instead of EE (EE = jump to GPT don't read MBR, 0C = hybrid)
2. type EE GPT protected 1MB
3. unaloc
4. unaloc

GPT
all 7 partitions (from FAT to user)

I see the boot.scr contains some addresses to jump to. 0x98000000 is not the start of a partition (probably syscall address), so the jump is probably calculated (mmc 1 4 ? is that some sort of hekate's GPT parser? sector 1, partition 4, which is Boot partition located at 0xA73800)
then I wonder what the GPIO branch means, why 1 5 ?

tabzer · Jul 30, 2019

Cyan said:
I see the boot.scr contains some addresses to jump to. 0x98000000 is not the start of a partition (probably address of sys command), so the jump is probably calculated (mmc 1 4 ? is that some sort of hekate's GPT parser? sector 1, partition 4, which is Boot partition located at 0xA73800)
then I wonder what the GPIO branch means, why 1 5 ?

I think mmc 1 4 is TWRP while mmc 1 5 is the main OS partition.

Cyan · Jul 30, 2019

ah, maybe you are right. though, partition 5 is recovery. I don't know android enough so maybe recovery is part of the boot chain.

I thought hekate could boot android using partition type or partition name, instead of partition number, so putting ext4 first was the idea, but I see having ext4 before android will require boot.scr edition too if it's launching hardcoded partition number.
so placing ext4 at the end is easier, you don't need to move many partitions.

so the idea would be:
"<FAT><androids>" (from android image) + <ext4><mmc>
edit l4t boot.scr to put proper partition info to boot.
android.scr won't need to be edited.

though, I'd still prefer fat to be at the end, so it's bigger.

what would be great is to have a partition list in hekate and use a scr file using arguments/variables based on picked partition. (or something more user proof).

tabzer · Jul 30, 2019

Cyan said:
ah, maybe you are right. though, partition 5 is recovery. I don't know android enough so maybe recovery is part of the boot chain.

the idea to have ext4 before android will require boot.scr edition too if it's launching hardcoded partition number.
so placing ext4 at the end is easier, you don't need to move many partitions.

so the idea would be:
"<FAT><androids>" (from android image) + <ext4><mmc>
edit l4t boot.scr to put proper partition info to boot.
android.scr won't need to be edited.

though, I'd still prefer fat to be ad the end, so it's bigger.

I assumed that because TWRP required manual input of (power + vol +), that the "if gpio input 150" argument was a reference to that.

Though the final product looks different from a partition manager (appears as 2 partitions, and tons of unallocated space), the original image file has these defined (for 16GB based image):

Partition Type Start Sect. Size
0 FAT32 2048 2.00 GB
1 EXT4 4196352 1.06 GB
2 EXT4 6418432 2.17 GB
3 Empty partition 10958848 70.00 MB
4 Empty partition 11102208 70.00 MB
5 Empty partition 11245568 70.00 MB
6 EXT4 11307008 8.61 GB

Perhaps the empty partitions are just placeholders for recovery/bootloader/etc or whatever was discarded.

Bylaws said this in response to the notion of creating a image file:

>Mount l4t as loopback and use partprobe
>Then shrink user data on android sd
>(use gparted)
>Then create a new partition on sd and DD l4t part 2 to it
>Then write and run pinned gdisk commands

I don't know how to manage most of that, so it'd take a lot of learning on my part.

lordelan · Jul 30, 2019

tabzer said:
Bylaws said this in response to the notion of creating a image file:

>Mount l4t as loopback and use partprobe
>Then shrink user data on android sd
>(use gparted)
>Then create a new partition on sd and DD l4t part 2 to it
>Then write and run pinned gdisk commands

I don't know how to manage most of that, so it'd take a lot of learning on my part.

Saw that on Discord as well.
Not sure about the first step. The other steps seem doable to me.

Cyan · Jul 30, 2019

Yes, now your table is what I found too when checking values in hexeditor.

I don't think the partitions are empty, they are just formatted in a format not recognized by your manager, so it's like "raw" or empty.
same as "ext4?" it's not sure it's ext4, because it's maybe a modified version used by android. maybe it checked the first sector of the partition to detect the format.

I also don't understand what he said about loopback yet, but I'll learn

shrinking user data is needed only if you filled your SD card, but if you picked the 16GB image to put on a 256GB SD card, you don't need to resize the android user partition, you have plenty of free space to create your l4t ext4 partition.
no idea what pinned dgisk command does, I'l learn that too.
but theoretically, it's not hard, it's what I said so far. create a ext4 for l4t at the end, and edit boot.scr to boot partition 7 instead of 2

tabzer · Jul 30, 2019

This is the pinned message, for reference:

Here are the old instructions when testers had to setup the hybrid mbr manually. But I'm not 100% sure it's done the same way today

# gdisk /dev/<sdcard>
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): r
Recovery/transformation command (? for help): h
Type from one to three GPT partition numbers, separated by spaces, to be
added to the hybrid MBR, in sequence: 1
Place EFI GPT (0xEE) partition first in MBR (good for GRUB)? (Y/N): n
Enter an MBR hex code (default 83): ee
Set the bootable flag? (Y/N): n
Unused partition space(s) found. Use one to protect more partitions? (Y/N): n
Recovery/transformation command (? for help): w

--------------------- MERGED ---------------------------

Also verified that the EXT4? are indeed EXT4 partitions. I will change that.

--------------------- MERGED ---------------------------

Cyan said:
I don't think the partitions are empty, they are just formatted in a format not recognized by your manager, so it's like "raw" or empty.

It was mentioned that booting the bootloader and recovery wasn't supported by twrp for lineageOS, so I took that to insinuate that data that might be on those defined partitions is irrelevant. It's a guess.

Cyan · Jul 30, 2019

Apparently the gdisk command is used to copy some GPT info into MBR.
I would have done it using hexeditor, but a tool is good too.

the log uses "ee" (GPT) as partition type for the FAT partition it copied to MBR.
This is where I'd suggest to set it to 0C (FAT32) so the first partition is also seen by the switch, l4t and eMMC. note that not all FAT partition are 0C, there are more possible codes.

I would have done it this way :
Physically:
FAT32 (for android, too small for switch/cfw)
Androids
L4T ext4
MMC
Another FAT32 ? (just an idea (bad), for switch & MMC)

Partition info:
MBR 1. FAT32 (maybe even the second FAT32 for Switch to use!)
MBR 2. Ext4 (for L4T to use)
MBR 3. GPT
MBR 4. empty

GPT 1. FAT32 (first partition, for android)
GPT 2-6 : Android partition
GPT 7 : empty or 2nd FAT32 (if it's useful to list both in android? really no idea how android works and whether it understand hybrid or not)
GPT 8-128 : empty

MBR would have the first or second FAT+LT4 info, hekate uses that one to read boot.scr file, usable by the switch, l4t and emmc.
GPT would have the FAT again (like the current android image), and all androids partitions.

Idea for different FAT32 partitions :
the one in MBR is seen by stock/cfw/eMMC/hekate. contains games (nsp, xci, homebrew, lakka, l4t user data)
the one in GPT is seen by android as "external sd card" ?
no idea if once in linux (l4t) it can mount both MBR and GPT partitions, at best it'll see only the FAT32 from MBR, you won't be able to move files between them. ideally it can see both and doesn't conflict. at worse, it can see both and conflicts (mount them twice). but linux should let the user mount/unmount partition at will, just choose the one you want, right?

Deleted-172301

Guest

GBATemp's lurking knight

Deleted-172301

Guest

Well-Known Member

GBATemp's lurking knight

Enlightened and bored.

Deleted-172301

Guest

GBATemp's lurking knight

Deleted-172301

Guest

GBATemp's lurking knight

Well-Known Member

Enlightened and bored.

GBATemp's lurking knight

Enlightened and bored.

GBATemp's lurking knight

Enlightened and bored.

Well-Known Member

GBATemp's lurking knight

Enlightened and bored.

GBATemp's lurking knight

Similar threads