The What

By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).

I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.

The Why

While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.

Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.

By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.

Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.

The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.

UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.

The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.

The How

While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.

So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.

I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.

Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.

I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.

Instructions

1. Dump your eid root key using IrisMan/MultiMan/etc
2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
6. Generate your hdd keys: ./ps3hdd_keygen.sh
7. Become root since most of this requires it: sudo -s
8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
9. Make virtual byte swapped encrypted device
   1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
   2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.

Source code

For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:

bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)

kpartx: https://git.opensvc.com/multipath-tools/.git/

ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

Credits

I would never have figured this out if it weren't for others' hard work.

Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.

Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.

Photos

Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.

 

[Nimrod-002]

In any case, I looked into the PS3 open source sdk a bit and I’m pretty sure I can write a tool to do this that could allow you to choose whatever percent you want as well as toggle between space and time modes.

that would be great. looking forward to it whenever you get the time to do it.

 

[einsteinx2]

Did a bunch of researching and I'm pretty sure I can hit the ground running with it this week. I also found someone that created a Docker container with the whole build environment all setup that should save me a bunch of time. Once I can get a proof of concept working, it won't take long to polish it up for release. Just need some people to beta test it first before setting it loose and potentially losing peoples' data.

 

[SG854]

Can someone make an instructional video

 

[bootmonster]

Can someone make an instructional video

I’d just wait for the tool, will be much simpler than manual editing.

--------------------- MERGED ---------------------------

Did a bunch of researching and I'm pretty sure I can hit the ground running with it this week. I also found someone that created a Docker container with the whole build environment all setup that should save me a bunch of time. Once I can get a proof of concept working, it won't take long to polish it up for release. Just need some people to beta test it first before setting it loose and potentially losing peoples' data.

Happy to beta test this have a couple of spare ps3s and drives kicking about

 

[BORTZ]

This is pretty cool. I personally never installed enough stuff installed on my PS3 at one time to notice. I probably will never hack/cfw my PS3 so I really have no need to worry about this. But still, very very cool.

 

[duwen]

Cool work!
Probably won't me of much use to me right now, as I scaled back what I was using on my PS3 and now only have 120Gb internal - reclaiming any of that 8% of wasted space really wouldn't be worth it, but for anything over half a Tb this will be invaluable.

 

[tbb043]

8% is not enough to bother with on my dinky ps3 drive (already use a 1tb external for it, along with 360 and wii on the same external.) But otherwise, cool.

 

[einsteinx2]

8% is not enough to bother with on my dinky ps3 drive (already use a 1tb external for it, along with 360 and wii on the same external.) But otherwise, cool.

I'm interested to hear your experience with your external drive. Have you run into many slowdown/stuttering issues with games or has it generally worked well? I generally prefer to keep things internal, hence my work on expanding the maximum internal space possible, but it would be nice to have a "spill over" drive that contains some games that I would otherwise not be able to have.

 

[Nimrod-002]

I'm interested to hear your experience with your external drive. Have you run into many slowdown/stuttering issues with games or has it generally worked well? I generally prefer to keep things internal, hence my work on expanding the maximum internal space possible, but it would be nice to have a "spill over" drive that contains some games that I would otherwise not be able to have.

In my experience its more convenient to share the game folder through Ethernet.

 

[MiCH431]

While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.

The PS3 CPU is big endian, while x86 CPUs are little endian. If the tool is not aware of this it won't match the superblock magic because it's written "backwards".

 

[einsteinx2]

The PS3 CPU is big endian, while x86 CPUs are little endian. If the tool is not aware of this it won't match the superblock magic because it's written "backwards".

Thanks for the info, that's very likely the reason that it didn't work. It's worth noting that tunefs was removed from the apt repos years ago as being no longer supported and presumably having issues, so I wasn't too hopeful that it would work to begin with.

I downloaded an old copy, and while it seemed to work on new test partition images I created, it didn't work for the PS3 partition. The file command worked fine however, so presumably it takes endianness into account.

 

[smf]

In my experience its more convenient to share the game folder through Ethernet.

I agree, ps3netsrv is by far the best way to go.

Laptop drives tend to go bad very easily. Instead you can put all your ISO on a multi drive raid setup as big as you want.

 

[einsteinx2]

Btw guys good news on the PS3 app progress. I have a working development environment now and I found a fantastic starter project called PS3 GamePad Test that shows everything from how to create an XMB icon, a pkg installer, read controller inputs, and write text and graphics to the screen. I'm really busy at the moment, but later next week I'm going to try and get a proof of concept working that displays the current minimum free space and optimization settings on the drive so I can confirm I can access the data I need. Once I have that working, it'll be simple to add the ability to set it. So I'm hoping to have something to release in a couple weeks.

Also my SATA data extension cable arrived, so I can start testing with a 3.5" 2TB drive I have available to see if I can get it to use all of the space, that is once I get the proof of concept of expanding a drive image from my 40GB drive to my 160GB drive. If I can do that, I should be able to expand to any size up to 2TB. Note this would almost definitely be something that would only work on a computer with dumped HDD keys, but if I get it working, I'll make it into a bit of a more user friendly tool than just a set of instructions.

I'll update when I make more progress, or if it turns out to be a dead end and just won't work.

 

[tbb043]

I'm interested to hear your experience with your external drive. Have you run into many slowdown/stuttering issues with games or has it generally worked well? I generally prefer to keep things internal, hence my work on expanding the maximum internal space possible, but it would be nice to have a "spill over" drive that contains some games that I would otherwise not be able to have.

I can't swear it's got absolutely no performance hits, but I've never noticed any, certainly nothing that would make me avoid using an external drive. Main issue is I have to make sure it's connected and powered (I use ac power, not powered by USB). With PS3 in particular, there's also the issue of games with files over 4GB in size, but multiman seems to handle that sort of thing fairly seamlessly, assuming there's room for the files on the internal PS3 drive.

 

[oxitran]

http://www.psx-place.com/threads/ps3-1-75-tb-internal-hdd-upgrade-formatting-utility.11989/page-2 This is as far as things seem to have gotter in terms of adding a better formatting utility to the PS3 4.46+. If you modify the HDD HPA table you can trick 4.46 into formatting it at 1.83tb which gives you around 1.5tb usable space. I would be nice to reclaim the OS allocated space for another 100gb back, but the issue that we had in 1.83tb formatting was with drive corruptions. If the ps3 needs to fix the corrupt drive then it cant, and will cause a complete loss of data.

This method would need to account for corruptions too. As the ps3 isnt going to know how to fix a HDD formatted on a pc and expanded to 2tb.

 

[einsteinx2]

From what I understand of that method, it's really just using the Sony-created formatting utility that for whatever reason allowed 1.75TB or so drives on a particular firmware version. The bit they're doing on the computer is just making a 2TB drive appear smaller to allow the formatting utility to work. My ultimate plan, which may totally not work, is to have a small HDD image and a tool that writes that image to the new HDD, then manually modifies the partition information to expand the partition to fill the drive as well as update the vflash to update it to the new drive's serial number so the PS3 allows it.

This tool, if it works, would allow for three pretty cool things:

1. The use of up to 2 full TB of space which is the theoretical maximum that the PS3 could support due to the use of MBR and 512 byte sectors (that would actually require a 3TB drive once they're thin enough, as 2TB drives are really more like 1.81TB, but that's another story)
2. Migrating a smaller PS3 drive directly to a larger drive without having to back it up, format the new drive, and reload everything
3. Making clones of your PS3 drive to use as hot swap backups in case of drive problems (currently doesn't work because the drive serial number is stored in vflash and the PS3 won't use the drive if it doesn't match)

For the 2TB use case, I'm definitely concerned about the potential corruption issues. The PS3 absolutely won't be able to run it's drive scan utility, however, in reality, other than the encryption there is really nothing special about the PS3 hdd's formatting. The GameOS partition is just a standard UFS2 partition used in FreeBSD which the PS3 OS is based off of. Hypothetically a FreeBSD boot disc could be created specifically to run drive scans and repairs, though the current tools for decrypting PS3 drives are only written for Linux, so maybe UFS2 repair tools on a Linux boot disc would be better, but whichever. Though the downside of course is that it would need to be run on a computer, not the PS3, unless we could make a PS3 linux boot disc for it (maybe possible, I haven't looked into that at all).

To be totally honest though, for my personal use case, I only care about having the max amount of space possible. I plan to keep everything backed up on a second drive (either a drive clone or something else) and periodically back up my game saves (maybe there's already a utility to do that automatically? if not, it would be something I'd be interested in writing). Then if the drive craps out, I'll just restore it or swap to my clone drive while I rebuild the old one.

I know for others though, it would certainly be a showstopper, but if I can get it all working, just the ability to keep clone drives and to easily upgrade to larger drives would be very useful to many people I think.

EDIT: Just took a closer look at one of the posts in the 1.75TB thread:

Sorry if this is off topic, and doesn't really solve the problem of having to downgrade to use larger drives. But I have discovered a way to use as much as 1.86TB. No such disk exists, and as you're aware, the PS3 can't simply partition the disk to use only the max supported space on a 2TB. However, it IS possible to modify a disk's firmware to make it appear to the PS3 as a smaller disk than it really is. To do this you have to take advantage of a feature set in the ATA specification to create a "host protected area" on the disk that is hidden from the PS3 (and most operating systems). A 2TB disk has 3907029168 sectors. Reducing this by 7% to 3633537126 sectors will allow a PS3 running 4.46 firmware to format and successfully boot using such a disk. That seems to be the absolute limit. Any higher than that, the PS3 will format a larger size than that, but it will black screen on boot (believe me, I tried through trial and error, and had to blank the first few sectors on the disk after each attempt to make the PS3 boot again each time I'd formatted with a larger size than the PS3's kernel would accept as valid, in between each time I'd changed the max sector count!) Upgrading the PS3's firmware as far as the latest (at time of writing) Rebug lite 4.82 from that point on, and it remains fully functional.

Interestingly it seems that the PS3's build in formatter on that version will actually format 2TB drives, they just don't boot... So it's hard to say if there's just a bug in the formatter or if there's some check in the kernel on boot blocking things, or who knows what. That may mean it's unlikely any method I use will work, but I'll just try and get all the partition resizing and serial number changing working first as a proof of concept, then if I can get that working well I'll test it with a 2TB drive. That way, even if it doesn't work, we still get a useful tool out of it, and working with smaller drives will be faster anyway for testing.

 

[Nimrod-002]

did the objective shift from unlocking 8% disk space by locally editing superblock?

 

[einsteinx2]

Not so much a shift as much as I have 2 objectives. First I'd like to write the utility to set the minimum free space directly on the PS3 to make it easier for others to do it. I have everything I need to get started on that, just haven't had the time to write the proof of concept app yet due to work.

Then longer term, I want to create utilities for the PC to migrate and format drives and potentially format 2TB drives if the PS3 kernel will read them. That was something I wanted to do since I first realized the power of manually editing partition tables, but it's much more complicated and more of a long term project to play around with.

I expect to have the POC app that just displays the min free space and optimization type within another week or so if I can find a few hours to write it. That will confirm that I can access the correct locations, then I'll add support for changing the values, which should be very quick (basically the write to the locations I'm already reading from).

I don't expect to have the other tools working for quite a while. That's just something I'm going to experiment with in my spare time, and if I'm lucky I'll get something working within a few months and will share my progress along the way.

 

[Deleted member 388531]

Not so much a shift as much as I have 2 objectives. First I'd like to write the utility to set the minimum free space directly on the PS3 to make it easier for others to do it. I have everything I need to get started on that, just haven't had the time to write the proof of concept app yet due to work.

Then longer term, I want to create utilities for the PC to migrate and format drives and potentially format 2TB drives if the PS3 kernel will read them. That was something I wanted to do since I first realized the power of manually editing partition tables, but it's much more complicated and more of a long term project to play around with.

I expect to have the POC app that just displays the min free space and optimization type within another week or so if I can find a few hours to write it. That will confirm that I can access the correct locations, then I'll add support for changing the values, which should be very quick (basically the write to the locations I'm already reading from).

I don't expect to have the other tools working for quite a while. That's just something I'm going to experiment with in my spare time, and if I'm lucky I'll get something working within a few months and will share my progress along the way.

Thank you for your hard work.

 

[bootmonster]

Any joy with this?