Hacking 4 TB Seagate / DIOS MIOS issues

[Elliander]

Dude give up. Wii software expects USB drives to have less than 2^32 sectors which means a 512B sector drive has to be smaller than 2TB. Supporting more than 2^32 sectors is harder than adding support for 4KB sectors.

If that's true than WiiMC wouldn't have worked at all in recent tests, but it does.

 

[edwardbirkholz05]

No, it couldn't access the second partition because it's (at least mostly) beyond the 2^32 sector boundary. Everything beyond 2TB is unreachable so it's a waste of time trying to make it work.

 

[Elliander]

No, it couldn't access the second partition because it's (at least mostly) beyond the 2^32 sector boundary. Everything beyond 2TB is unreachable so it's a waste of time trying to make it work.

You are missing my point. You said:

Wii software expects USB drives to have less than 2^32 sectors which means a 512B sector drive has to be smaller than 2TB.

But that statement implies a drive issue, not a partition issue. If it expects the drive to have such and so sectors it shouldn't work at all. Even if that is the issue, USB Loader GX had problems seeing the first partition as well which suggests a software issue separate from that.

Let's assume for a moment that you are correct. That it cannot see a PARTITION larger than that. Well, why couldn't it - in software - be told which partition to look at?In any case, I still haven't ruled out the problem being that I broke the theoretical rules for 32k partition sizes. I have to run a few more tests to be certain that it isn't a factor.

EDIT: Don't worry, this next test might instead add support to your theory. If you are right then it will see the first two partitions, but no more. If it only sees the first partition again or none at all that wouldn't be compatible with your explanation

 

[edwardbirkholz05]

http://en.wikipedia.org/wiki/SCSI_Read_Capacity_Command http://en.wikipedia.org/wiki/SCSI_Read_Commands#Read_.2810.29 Both of those commands only use 4 bytes for LBA (sector) values and they are the only ones used by wii software. You can mess around with partitions as much as you like but it won't work.

 

[Elliander]

This time I created 4 partitions. 953861.33 MB (931.51 GB) each. All are Fat32 with 32k cluster size. I named the partitions DIOS MIOS, WII_1, WII_2, WII_3. That way I will be able to identify the order later if I need to. I placed on the DIOS MIOS partition one Gamecube Game and one Wii game along with all other folders. On each of the following partitions I created just a wbfs folder and copied onto it one wii game each. That way if any games show up I would know which partitons are active.

Observation: After converting to Hybrid MBR/GPT the 4th partition was gone and was instead reported as unallocated space. When I went to recreate it a notification explained that there were no free MBR slots, so in Hybrid mode and MBR only mode there is a maximum of 3 partitions. I decided to leave the 4th partition unallocated for the purposes of this test.

Results:

Hybdrid MBR/GPT: WiiMC freezes. USB Loader GX produced errors: "Exception (DSI) Occurred!) Stack Dump, Code Dump"

MBR Only: WiiMC loads first partition only. Waited a few minutes to be sure that it wasn't going to load anymore. USB Loader GX produced same error.

GPT Only: WiiMC loads first partition only. Waited a few minutes to be sure that it wasn't going to load anymore. USB Loader GX produced same error.

Since it was suggested that the problem is related to having greater than 2^32 sectors I decided to perform an additional test. This time I created just one FAT32 GPT partition with a 32k cluster size just to see if anything will work. For this test I also allowed Windows to create it's 128 MB GPT partition and I also allowed the partition to be just 1.99 TB. I placed just one each Gamecube and Wii game on the drive along with all the other files.

GPT Only: Same as before

I would have to agree that the fact that the drive itself is larger than 2^32 sectors is why USB Loader GX isn't working, but I don't believe with the assumption that the Wii itself can't handle it. It's obvious that the Wii is detecting the drive, and although I didn't do extensive tests with WiiMC it's at least proven to be able to obtain some information from the drive. USB Loader GX also clearly detects it's presence, otherwise it wouldn't initialize the drive.

http://en.wikipedia.org/wiki/SCSI_Read_Capacity_Command http://en.wikipedia.org/wiki/SCSI_Read_Commands#Read_.2810.29 Both of those commands only use 4 bytes for LBA (sector) values and they are the only ones used by wii software. You can mess around with partitions as much as you like but it won't work.

Again, the Wii does detect the drive, and does obtain information from the drive. Just because a program running on the Wii has problems because of that doesn't mean the Wii itself or all programs running on the Wii must have the same problem.

_______

Cyan: What are your thoughts on this? At this point I have exhausted all of my options. Unless you think that there is anything that could be done in software there isn't anything more that I can do. It really all comes down to what you think would be easier: Getting DIOS MIOS to work with GPT/4k sectors - or getting USB Loader GX to read 512 byte drives larger than 2 TB properly.

_______

EDIT:

WBFS Test:

Although WBFS partitions are not useful for loading Gamecube Games or EmuNAND games I figured it would be an important test just to see to what extent USB Loader GX dislikes 512 byte drives. Unfortunately, no matter how many partition combinations I tried, it will only work in MBR only and even then only with a maximum of 2 TB. (I figured that if the test works with that I could try harder, but if it fails at 2 TB there wouldn't be a point)

Wii Backup Manager would and says complete, but doesn't actually accomplish it. WBFS Manager 3.0 was able to format it to WBFS, but it limited it to 128 GB for some reason so I had to delete the volume to start over. wbfsGUI.v14.3 is what was able to format the 2 TB correctly. Afterwards Wii Backup Manager was able to see all 2 TB (which I prefer using)

I put one game onto the drive and then ran the test with USB Loader GX... and... IT WORKS!!!

USB Loader GX has no problems loading up without any errors and displaying the games on a WBFS partition on a 512 byte hard drive. That in and of itself disproves the idea that the Wii is incapable of accessing a 512 byte drive. Sure, the partition is limited to 2 TB, but that's only because WBFS is limited by MBR and I am too sleep deprived to put forth the effort of trying to get it to see more.

 

[Elliander]

Since the WBFS volume works, I decided to test to see if the same volume would work at FAT32. In other words, only one 2 TB volume and no space allocated above the 2 TB point. MiniTool wouldn't allow me to format the volume as it was so first I formatted it to NTFS within Windows with a 32k cluster size and then I formatted it to FAT32 with a 32k cluster size within MiniTool.

2 TB volume only, MBR: WiiMC works. USB Loader GX loads correctly and displays games. Wii games load and play perfectly. When attempting to load Gamecube games the screen just goes black and the USB drive powers down. No error messages, had to manually power off Wii. To verify that it wasn't my settings I then plugged in my working drive and the same GC game loaded with no problems.

I then converted the drive to GPT (using gdisk) without making any other changes or adding any new volumes and ran the test again.

2 TB FAT32 only, GPT: WiiMC works. USB Loader GX loads correctly and displays games. Wii games load and play perfectly! When attempting to load Gamecube games an error is produced explaining that the 'main gamecube path' must be set to a primary partition. (no mention of 512, good sign. Also no mention of MBR, which is unexpected.) Verified the paths were correctly set. Verified that the partition is set to primary. (Not that I expected GC to work in GPT, but it seems like an odd response.)

Then I formatted the remaining space to FAT32 32k without making any other changes or adding any files to it to see what happens. (I'm trying to establish the true boundaries of compatibility here) I also reset settings in USB Loader GX to default before plugging in the drive just to see if there would be any difference since the last time with a similar configuration.

2 TB FAT32 and 1.6 TB FAT32, GPT: WiiMC works for the first partition only. USB Loader GX... works? Games load and play perfectly too (I find that very strange because in a previous test a similar set up didn't!). Gamecube games produce same error as before.

This time I added a single Wii game to the second partition to see if there is any difference, just in case maybe the presence of data beyond 2 TB was causing a problem.

2 TB FAT32 and 1.6 TB FAT32, GPT, data on second partition: USB Loader GX does load correctly, but the Wii game on the second partition didn't appear on the list at all. Went into global settings to enable multiple partitions and even restarted the Wii, but made no difference. USB Loader GX could not see at all the second partition.

This time I wanted to see if it's a limitation with total allocated space, or if for some reason it's limited to one partition no matter what I do. I set it to MBR, but after the test the computer read it as GPT

2 FAT32 partitions 949 GB each, MBR: WiiMC loads first partition only, as expected. USB Loader GX works. It can see and play games from both partitions perfectly. Gamecube games load the black screen again.

This time when I went to switch it to GPT I noticed something odd:

That's not a Hybrid MBR, yet both are present, and gdisk actually gives me a choice of which to initialize. Even so windows won't let me create beyond the 2 TB limit so I am interpreting this as an MBR only, but it's not the same as previous MBR tests. To be sure of the results I converted to MBR only so that GPT isn't even present, but the results were the same.

2 FAT32 partitions 949 GB each, GPT: WiiMC loads first partition only, as expected. USB Loader GX works. It can see and play games from both partitions perfectly. Gamecube games show the error about the primary partition.

I then created a third partition with the remaining space and place one game onto.

3 FAT32 partitions, GPT: WiiMC loads first partition only. USB Loader GX crashes with "Exception (DS!) Occurred!" There has to be some reason why under some scenarios USB Loader simply ignores a partition and while in others it crashes.

For a last test I wanted to create a single 3.7 TB partition set to GPT. I have used USB Extreme in the past to create a FAT32 partition that is 3.7 TB in size, but unfortunately it doesn't appear to work with a 512 byte drive. I also couldn't figure out how to get WBFS any larger. Instead I just formatted it to NTFS, 32k cluster size, just to see how the Wii would respond. I really just wanted to find out if USB Loader GX would at least see data beyond 2 TB.

3.7 TB NTFS, GPT: WiiMC Works. USB Loader GX Works!!! Wii games play perfectly, despite the fact that it's an NTFS partition using the entire volume. The Gamecube games show the same primary partition error message.

Conclusions: It's obvious that the Wii is perfectly capable of handling larges larger than 2 TB with a 512 byte sector size. WiiMC will work with any size partition I threw at it from 950 GB to 3.7 TB from FAT32 to NTFS as MBR or GPT, but no matter what I try it can only ever see the first partition. USB Loader GX can also see all 3.7 TB, but it seems to be limited to a maximum of two partitions and if it's using FAT32 it won't see any partitions beyond the 2 TB mark. The errors I got before were probably more related to some bug in settings, which is why changing it to defaults fixed that. Or it could be that because the previous MBR tests were previous in Hybrid mode that something related to that caused the problems. (the last time it happened did follow a convert between MBR and GPT) Unfortunately, DIOS MIOS does not appear to work yet on the drive even set to MBR, Primary, Fat32, 512 byte sector size, 32k cluster, and 950 GB partition. The fact that it works for the Wii games suggests that it should be possible for DIOS MIOS to play at least up to 2 TB. Also, the fact that both WiiMC and USB Loader GX works up to 3.7 TB means that it should be possible to get USB Loader GX to load up to that point in FAT32 mode. There is no reason to believe that it can't work.

You can mess around with partitions as much as you like but it won't work.

You were saying?

*cough*smallerdrive*cough*

No thanks

You didn't forfeit and are still working hard to make it compatible!

Can you take a look at these results and give me your thoughts on if USB Loader GX could be made to work with DIOS MIOS under any tested configuration?

EDIT: Seeing the unexpected success of NTFS gave me an idea for one additional test. I created one FAT32, 32k, 950 GB Partition and the rest I formatted to NTFS, 32k. I placed the gamecube game on the first partition only and a wii game on each partition.

950 GB FAT32 & 2.7 TB NTFS, GPT: WiiMC loads first partition only, but USB Loader GX loads both. Wii games from both partitions load and play perfectly. The Gamecube game gave the primary partition error.

Since DIOS MIOS requires FAT32, but the Wii games will load from NTFS, this seems like the perfect combination. I can access the entire drive. All that's left is to find a way to get USB Loader GX to work in hybrid MBR/GPT mode, because in MBR only I wouldn't be able to use the entire drive.

EDIT 2: I ran a final test to see if the problem really is the Hybrid MBR/GPT. Since each partition has to be under 2 TB I reworked them before converting to Hybrid mode.

2 TB FAT32 & 1.7 TB NTFS, Hybrid MBR/GPT: WiiMC freezes on the "Please Wait Loading..." prompt. USB Loader GX produces the "Exception (DSI) Error!".

It's confirmed. USB Loader GX really doesn't like Hybrid MBR/GPT.

 

[Rydian]

Mind doing a test? Pad the drive to the point that games are on the last fourth of it (defrag first of course just to make sure things are going in order), and see if anything can access them at that point.

 

[Elliander]

Mind doing a test? Pad the drive to the point that games are on the last fourth of it (defrag first of course just to make sure things are going in order), and see if anything can access them at that point.

Sure, no problem. Using which Setup? 950 GB FAT32 & 2.7 TB NTFS, GPT?

I was just in the middle of trying to test of FAT32/WBFS, but I am having so much trouble getting it to format that I would actually prefer to give it a break for a while and just transfer files over to it instead.

EDIT: I don't have enough games to pad it that far. Is it it OK if I just pad it with useless video files and such, then defrag, then add a few games to test?

 

[Rydian]

Ah, I misread your earlier post, thinking it was FAT32 and not NTFS, nevermind. NTFS going large isn't the concern, it's getting FAT32 that large, since DIOS MIOS doesn't support NTFS. If you do get a FAT32 partition that large that has stuff bootable, I'd like to see if data near the end can be read or if it has the cutoff point anyways even if the partition itself can be made larger.

 

[Elliander]

Ah, I misread your earlier post, thinking it was FAT32 and not NTFS, nevermind. NTFS going large isn't the concern, it's getting FAT32 that large, since DIOS MIOS doesn't support NTFS. If you do get a FAT32 partition that large that has stuff bootable, I'd like to see if data near the end can be read or if it has the cutoff point anyways even if the partition itself can be made larger.

Oh, you were referring to the attempt at USB Extreme Tools? Ya, I wasn't able to get that to work on this drive in 512 byte mode, but I will be sure to let you know the next time I use that program. I thought you were wanting me to test the partitions to get data to fill beyond the 2 TB mark to verify something along those lines.

 

[Rydian]

Yeah that's what I meant, see if data beyond the 2TB mark can be read (other than what's already known for NTFS). I have a feeling it won't, due to the mathematical limits people have posted in this thread a few times already.

 

[Elliander]

Yeah that's what I meant, see if data beyond the 2TB mark can be read (other than what's already known for NTFS). I have a feeling it won't, due to the mathematical limits people have posted in this thread a few times already.

Well, the only real mathematical limit that was a concern is the supposed idea that a 512 byte drive would never work on the Wii if it's larger than 2 TB, so if you would like I can still pad the drive and test it out anyway.

Also, I finally got in the mail a new enclosure to test out. I am going to need to test it out anyway with data transfers and such.

 

[Elliander]

The Thermaltake 5G Active Cooling definitely does not work. Every SATA hdd I have from 500 GB to 4 TB shows up as unformatted when used because it presents the drives as 4k. (If a drive is formatted in 512 bytes, and is then presented as 4k, it won't work without a reformat. And Vice versa.) For this unit there is no firmware update, which is a shame because I really liked the design. There are a few others I want to test out too.

 

[edwardbirkholz05]

If you had paid attention, I was saying you will never get a 4TB HDD with 512B sectors to work properly because no wii programs will be able to reach beyond 2TB. That doesn't mean it won't work at all, just that it will never work as anything larger than a 2TB drive. All your tests have proven that to be correct. The 3.7GB NTFS test only worked because all the data was at the start of the drive. You've wasted time and money (your 4TB is effectively crippled to 2TB) and if you really want to ignore what is in the code and stubbornly believe you've succeeded then I look forward to the day that a wii program tries to write beyond 2TB and the sector address overflows back to your MBR or partition table...

 

[Elliander]

If you had paid attention, I was saying you will never get a 4TB HDD with 512B sectors to work properly because no wii programs will be able to reach beyond 2TB. That doesn't mean it won't work at all, just that it will never work as anything larger than a 2TB drive.

And again, you were wrong.

All your tests have proven that to be correct.

How did my tests prove that?

The 3.7GB NTFS test only worked because all the data was at the start of the drive. You've wasted time and money (your 4TB is effectively crippled to 2TB)

Nope, that's not it at all. Wii games run perfectly from that partition. Of course, I already mentioned that I am in the process of padding the data to run additional tests on it. It also worked with NTFS being the second partition so that will save me time. I am going to be using Disk Scrubber software to fill up the drive to that point for this purpose. Also, when all the tests are done, if I so choose I can easily convert this back to a 4k drive so I don't see how I crippled anything.

and if you really want to ignore what is in the code

Actually, I am pretty sure that it will require some updates for USB Loader GX to really work with this for Gamecube games. I pretty much accepted that a few days ago, but the reason I pushed forward with the tests anyway was because I needed to be sure to what extent the limitations exist so that if and when someone decides to update the correct software they know exactly where the problem is.

and stubbornly believe you've succeeded then I look forward to the day that a wii program tries to write beyond 2TB and the sector address overflows back to your MBR or partition table...

Why are you so dead set against this anyway? It's one thing to make a suggestion on what to test for, but it's another to say "this will never work" without any real evidence. People used to say that about SDXC on the Wii. When people assume something is impossible just because one person says it is, well, that's pretty silly.

 

[edwardbirkholz05]

Show me any test that proves you were able to read data beyond 2TB. The read_10 command (which I DID show you as evidence) simply does not allow it, if you took the time to understand that instead of throwing shit at the wall to see what sticks you would see why you're wasting your time.

 

[Elliander]

Show me any test that proves you were able to read data beyond 2TB.

You said:

All your tests have proven that to be correct.

If when I ask:

How did my tests prove that?

You essentially answer with an identically reversed question, *sigh* whatever. You'll see the results of the data padding like everyone else when it's done and not before.

The read_10 command (which I DID show you as evidence) simply does not allow it,

No. You quoted Wikipedia, and didn't really talk about it in relation to the Wii. All you said was:

Both of those commands only use 4 bytes for LBA (sector) values and they are the only ones used by wii software.

I actually did read up on that page,and I did a few searches about it, but didn't really find what you were talking about. You are talking about the way a hard drive is formatted, but there are many ways of addressing sectors even on the same sector size scale. I really can't find any mention anywhere as to why your source in any way limits the maximum size of the hard drive.

In any case GPT was designed to circumvent the 2 TB limit. In fact, even without the emergence of 4k drives the GPT standard would have been more than enough to allow the growth of hard drive capacities. That's the reason why, in MBR only mode, the Wii will NOT see more than 2 TB total. Unfortunately, DIOS MIOS requires the use of MBR while Wii Loading does not. The point of trying to get it to use a Hybrid MBR/GPT is that DIOS MIOS will work within that 2 TB limit as MBR only while the Wii games will occupy the volume space beyond it as GPT.

For a 512 byte sector size disk using GPT the true capacity limit is 9.4 Zeta bytes. For a 4k disk using MBR the true capacity limit is 16 TB. Looking at all the physical rules, again, I see no evidence that a 512 byte disk can never go beyond 2 TB.

if you took the time to understand that instead of throwing shit at the wall to see what sticks you would see why you're wasting your time.

I am not trying to see what sticks, I am trying to identify the real (as opposed to imagined) boundaries so that ways around them can hopefully be found. Computer Science might not be my current major, but back when I was in a computer related field this is the general approach I was taught to take. Sure, it's been a good 12 to 14 years, but it's still good.

 

[Elliander]

I just finished setting up the new test. I set up a 950 GB FAT32 partition and a 2.7 TB NTFS partition. After verifying that games run on each partition with the Wii I set it up with "Hard Disc Scrubber". I chose this one because it will fill up the drive quickly with text files of random data which I will interrupt after it has written 1.5 TB to the NTFS partition. That should be enough junk data. I will then place a few games on that partition and will test to see if they load. If that test is successful I will then have USB Loader GX extra the NAND to an EmuNAND on that partition and then test that EmuNAND. It might take me a few days to complete this process.

The purpose of this test is to verify if the Wii is capable of reading and/or writing data to a 512 byte sector size hard drive beyond the 2 TB mark.

 

[edwardbirkholz05]

The read_10 command has NOTHING to do with how the drive is partitioned or formatted, it is used to access the raw data stored in the sectors of the drive. It uses a 32-bit value for the target sector, so the highest sector that can be used is sector 4,294,967,295. A 4TB drive with 512B sectors will have 8,589,934,592 total sectors. Therefore it is IMPOSSIBLE for all the sectors of your drive to be readable using the read_10 command, which is what ALL wii software uses. The same goes for a lot of other software/hardware devices which is one of the reasons why the entire industry is moving to 4KB sectors - to keep the total sector count below 2^32. So it's not "impossible just because one person says it is", all the drive manufacturers also know it's a dead end. Why do you think you had so much trouble finding a 512B sector drive larger than 2TB in the first place?

 

[Cyan]

I don't know how much test you did with DIOS MIOS, but sometime you need to do more than one try to check if it's working fine. For example random Wii shutting down is an issue with working drives too.

I told you to use an old version of USBLoaderGX to do your tests, to prevent the "check" for correct setup, and don't have error messages preventing you to boot DM. (or even don't use it at all, boot GC games using DML Booter).

You told "it's asking for primary !! so it's good it didn't tell anything about 512, but strange it didn't tell anything about MBR"
You don't understand how a program is working (sequentially), nor what you are talking about (what primary is).

Primary is one of the MBR partition type.
If you use a GPT, it will NOT be primary anymore.
Primary = MBR
If you set the drive as GPT, of course it will tell you don't have a primary partition anymore.


The USBLoader doesn't tell you ALL the error in one single message, it's checking each settings one by one and stop on the first wrong setup.
If you see "you need primary", It doesn't mean that other setup are correct. it only means that the first check failed, and it stopped at the first check. (no need to check/display other error, it will already not work if you are in GPT)

I can wait for all the checks to complete and display a single message with all found problems, but it's too big to fit the warning's window.

Spoiler

About this picture :
MBR entry is stored on Sector 0
GPT entry is stored at two different places on the drive : Sector 1 and Final Sector-34 (at the end of the drive)

This picture means either:
1. You removed the GPT, but it only deleted the data located at Sector1 and kept the data on Final-sector
2. You don't have a "GPT partition" type stored in one of the 4 MBR entries, but the GPT Sector1 is still there.
In both case, it means your partition manager is not doing a complete job when you delete GPT.



To do what you want to achieve, You need a hybrid mode if you want to use all the partitions:

in MBR you need to list ONLY the <2TB partitions
in GPT, you need to list the <2TB and the >2TB partitions
the FAT32 partition must be the first one in the MBR table.
Do not create a protective partition (or be sure it's not stored on the first slot of MBR)

2 TB FAT32 Primary (MBR) (& protective) & GPT partition presence information
2 TB FAT32 GPT & 1.7 TB NTFS GPT

But even with the hybrid mode, USBLoaderGX will warn you about the GPT type, because GPT is loaded first and the first partition will be "seen" as a GPT even if it exists as MBR too.
if I remove the error message (and make it a warning only for example), it will still boot the MIOS, and DIOS MIOS will check the MBR only and will find the FAT32 partition.


FIRST, you need to be sure that the drive is compatible with DIOS MIOS before doing more partition tests, or else you won't know if the problem is the partition or the drive.
Format a single <1TB FAT32 MBR primary partition again, and test DIOS MIOS with any loader.
The wii might shut down, or just black screen, try few times, with different DM version (and if possible with USBgecko).

If it doesn't work more than 10 times in a row, the drive is most likely not compatible, and doing more partitioning tests are useless.


And based on eward's post, you won't be able to read anything past the 2TB.
the tests you did proved that you couldn't load data past the 2TB mark.