Unlock up to 8% extra total space on the PS3 internal hard drive

Discussion in 'PS3 - Tutorials' started by einsteinx2, Sep 21, 2018.

  1. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    The What

    By default, even on custom firmware, the PS3 reserves 8% of your total internal hard drive space. From my searching online, I don't believe anyone has ever successfully unlocked that wasted space, so I decided to give it a shot. Turns out not only is it possible, but it's relatively easy thanks to some existing tools created by the community. This guide will explain how to reclaim that wasted space by manually modifying the metadata of the UFS2 formatted GameOS partition using Linux, as well as the potential cons (primarily performance, though I haven't experienced any performance issues yet myself).

    I believe the pros outweigh the cons though, and have been using this now without issue on my personal PS3 with a 1.5TB drive for a few days now, installing tons of games without a hitch.

    The Why

    While external drives used with the PS3 are all FAT32 formatted (or NTFS if you have CFW), the GameOS partition on the internal hard drive is formatted using the UFS2 filesystem with a layer of encryption on top.

    Like other *nix file systems such as Ext2/3/4, UFS2 can reserve part of the drive's space to only be used by the system or the root user. This is to reduce fragmentation and also prevent the drive from completely filling up, possibly freezing the computer.

    By default, UFS2 reserves 8% of any drive's free space, in this case meaning it can't be used by the PS3 for installing games. This is why when you first install that shiny new 1.5TB hard drive you will see that not only is it only actually 1.36 TB because hard drive manufacturers really love counting in base 10, but you also lose another 111.4 GB.

    Now generally having this reserved space is not a bad thing, however the problem is that as drives grow larger, the amount of wasted space becomes larger than some of the smaller PS3's entire hard drives! Clearly that much isn't needed to prevent fragmentation.

    The tunefs documentation mentions that "the file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold." Note that these numbers are already higher than the 8% default that all UFS tools as well as Sony use. Also note that it says as free space drops to 0%, performance may be up to 3 times slower than normal. However, it's unclear whether that only affects newly written--presumably more fragmented files--or all files. It seems like this is a worthy tradeoff, especially since if you do notice any performance issues you can simply delete some games or other data to free up the space. By changing the minimum free space, performance is not changed at all unless you choose to fill the drive completely...but at least you can now make that choice.

    UFS2 supports two write optimization modes: time and space. Time optimization is the default--and is used by the PS3--and allows for faster writes at the expense of potentially higher fragmentation (though generally only as the drive reaches capacity). Since we're allowing ourselves to fill almost the entire drive, these instructions also change the optimization mode to space. That means that the filesystem will spend more time during writes to ensure the files are less fragmented, insinuating that it will shuffle blocks around to make contiguous space.

    The tunefs documentation says that "the file system can either try to minimize the time spent allocating blocks, or it can attempt to minimize the space fragmentation on the disk. Optimization for space has much higher overhead for file writes. The kernel normally changes the preference automatically as the percent fragmentation changes on the filesystem." However, since the PS3 uses a custom operating system and this documentation is from FreeBSD, I think it's best to manually set the space option anyway. I have not noticed any performance degradation so far since making this change, though I still have plenty of free space on my drive at the moment.

    The How

    While it's possible to mount a PS3 hard drive in Linux and view its decrypted partitions, unfortunately the tunefs.ufs tool doesn't appear to work. It always complains that it can't find the superblock. However, the file command does work fine to show the UFS2 filesystem info when tested on a dump of the start of the partition.

    So instead of using the tunefs utility to change the minimum free space and optimization type, I wrote a script to manually scan and test single byte changes to a dump of the partition's superblock using the file command until I found the correct locations to change.

    I tested this on 2 hard drives of vastly different sizes: 160 GB and 1.5 TB. On both drives, I found that the superblock was located in the same location--the standard 128 block aka 65,536 byte offset. I also found that the locations to set the minimum free space percentage and optimization type were in the same place on both drives--byte 65,599 and 65,667 respectively. However, I highly recommend running the find_ps3_ufs2_byte_locations script anyway just to confirm before making any changes to your drive.

    Once you know the correct offsets, changing the values is simple. To adjust the minimum free space, simply write the percent in hex to byte 65,599--e.g. for 1% free space write 0x01 or for the default 8% write 0x08. To change the optimization type, write a hex 0x01 to byte 65,667--the default is 0x00 for time optimization.

    I think this should be possible to do as a PS3 homebrew app so that it can just be done directly on the device without even removing the drive. Unfortunately I have zero experience with writing PS3 homebrew, but maybe someone else with more experience can use this information to do it. I'd love to see this as an option in the REBUG custom firmware settings.

    Instructions

    1. Dump your eid root key using IrisMan/MultiMan/etc
    2. Setup a computer or virtual machine with Ubuntu 16.04. The rest of these steps are done on that machine. I tested using Parallels on a MacBook Pro, but it should work on just about anything as well as other distros.
    3. Clone my repository: git clone https://github.com/einsteinx2/PS3-Reclaim-HDD-Space.git
    4. Change to the new directory as we'll do all of the work there: cd PS3-Reclaim-HDD-Space
    5. Rename your eid root key file to eid_root_key.bin and place it in the PS3-Reclaim-HDD-Space directory
    6. Generate your hdd keys: ./ps3hdd_keygen.sh
    7. Become root since most of this requires it: sudo -s
    8. Find the device name: fdisk -l (In my case, using an external USB enclosure, it was /dev/sdb)
    9. Make virtual byte swapped encrypted device
      1. If you have a drive 1TB or less (not confirmed the exact limit): ./makedev bswap16.512 /dev/sdb
      2. If you have a drive larger than 1TB (or maybe it's 1TB and larger, I don't have a 1TB drive to test): ./makedev bswap16.1024 /dev/sdb
    10. Create decrypted device: cryptsetup create -c aes-xts-plain64 -d ./hdd_key.bin -s 256 ps3hdd_crypt /dev/nbd0
    11. Map decrypted partitions: ./kpartx -a /dev/mapper/ps3hdd_crypt
    12. View decrypted partitions (ps3hdd_crypt2 is the UFS2 GameOS partition): ls -la /dev/mapper/
    13. View current free space: [ -d /mnt/PS3GameOS ] || mkdir /mnt/PS3GameOS && mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
    14. Dump the superblock of the GameOS partition: dd if=/dev/mapper/ps3hdd_crypt2 bs=512 count=256 of=GameOS_superblock.img
    15. Confirm the seek values for the next 2 commands: ./find_ps3_ufs2_byte_locations GameOS_superblock.img
    16. Set minimum free space to 1%: printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65599 count=1 conv=notrunc
    17. Set optimization type to "space": printf '\x01' | dd of=/dev/mapper/ps3hdd_crypt2 bs=1 seek=65667 count=1 conv=notrunc
    18. View the now larger free space: mount -t ufs -o ufstype=ufs2,ro /dev/mapper/ps3hdd_crypt2 /mnt/PS3GameOS && df -h | grep "Avail\|ps3hdd_crypt2" && umount /mnt/PS3GameOS
    19. Disconnect device: kpartx -d /dev/mapper/ps3hdd_crypt && cryptsetup remove ps3hdd_crypt && ./stop-nbd0
    20. Pop the drive back in your PS3 and enjoy the extra space! Note that I left 1% reserved space rather than going all the way to 0% to ensure that the drive never completely fills up, as I'm unsure what problems that would cause for the PS3's operating system.

    Source code

    For ease of use, this repo contains precompiled binaries for Ubuntu 16.04 64bit. If you need or prefer to compile yourself, here are the tools used:

    bswap16: https://github.com/sguerrini97/nbdcpp (note that for >1TB drives you must change <unsigned BS=512> to <unsigned BS=1024>)

    kpartx: https://git.opensvc.com/multipath-tools/.git/

    ps3hdd_keygen.sh: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

    Credits

    I would never have figured this out if it weren't for others' hard work.

    Huge thanks to Berion at PSX-Place for the hdd key generation script as well as pointing me to the information on mounting a PS3 HDD in Linux. His post here contains the script and the link: http://www.psx-place.com/threads/hdd-keys-generating-scripts.10610/page-2#post-125197

    Huge thanks to sguerrini97 at Playstation Hax for implemnenting PS3 hard drive mounting support for modern Linux kernels. Here's the post about it: https://playstationhax.xyz/forums/topic/4671-mounting-ps3-hdd-on-newer-linux-kernels and the GitHub repo: https://github.com/sguerrini97/nbdcpp.

    Thanks to dsroche for writing the original nbdcpp implementation that sguerrini97 forked, and thanks to Glevand for the original work on mounting the PS3 hard drive in OtherOS and for the great information here on the PS3 dev wiki: http://www.psdevwiki.com/ps3/Mounting_HDD_on_PC. Also thanks to anyone else that worked on PS3 hard drive mounting or anything else I'm not aware of.

    Photos

    Here are the before and after photos. They were taken immediately before removing the drives and immediately after reinserting them.

    [​IMG] [​IMG]

    [​IMG] [​IMG]
     
    Last edited by einsteinx2, Sep 26, 2018
  2. EmBlaze

    EmBlaze GBAtemp Regular

    Member
    3
    Feb 25, 2018
    Canada
    Wow, great work you've done there! If I was still actively installing games on my PS3 Slim, then I may really try it out. But right now, there's no need.

    I have a 320GB HDD, and when free space goes below ~7-10GB (if I recall correctly), my dad complains that Sports Champions 2 lags... Just my experience, not going to speculate.

    I wonder if the instructions will work with Windows Subsystem for Linux lol. Thanks for sharing such interesting and useful news!
     
    einsteinx2 likes this.
  3. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    Thanks! When I popped in my 1.5TB drive and realized how much space I was missing out on, I was obsessed with unlocking it. Searched around and found great info on mounting the hard drive, but nothing on adjusting the free space. Had a hunch this would do it and was honestly surprised it worked the first try haha.

    Also great to hear (assuming your memory is correct) that it took all the way down to around 3% free space before you saw performance issues. It may be an even lower percentage for larger drives. I'm still in the process of filling my drive up, so I'll be keeping an eye on performance and will report back when I get it closer to full.

    Unfortunately this will almost definitely not work on WSL, since this requires some linux kernel level stuff and WSL basically runs the Linux userland on the Windows kernel (or rather it translates POSIX calls to the NT kernel I think, but in any case I doubt they implemented the stuff needed for this to work). Luckily it worked great for me in a VM, so it should be trivial to do on any computer using VirtualBox, VMWare, or Parallels.

    — Posts automatically merged - Please don't double post! —

    Err whoops, I forgot that that also counts the 8% reserved space, so it was more like 10-11%, which is pretty much in line with the docs. So it's very possible that I'll start seeing performance issues as I get down to 10% free space and lower. I guess we'll see if this was actually useful or not once I get to the end of my drive. It's worth noting though that I also changed the drive optimization from "time" to "space" which supposedly reduces fragmentation at the expense of write speeds. Assuming the filesystem driver that the PS3 is using is something relatively standard and actually respects that setting, that may prevent some of the performance problems. Also just the fact that the drive is so much larger, so 10% free space is about 1/3 of your entire drive leaving a lot more room for less fragmentation. I'll find out in a few days when all my games finish installing haha.

    Anyway I also have a suspicion that it may be possible to use a similar technique of manually editing drive metadata to create a tool to format 2TB drives with all of the space available. The PS3 uses MBR (master boot record) and 512kb sectors, which has a hard limit of 2TB for total drive size, so it's unlikely we'll ever be able to use larger internals (plus there are currently no 2.5" drives larger than 2TB that are less than 15mm so they won't fit anyway). However, I think it should be possible to manually create the partitions, or maybe write a raw drive image from a smaller drive and write a tool to manually adjust the partitions to allow full use of the 2TB without using the PS3's built in formatting tool. That's my next project if I can find the money for a 2TB drive :)

    If I can get 2TB drives working, we can just leave the reserved space at 8% (or even raise it to ensure high performance!) and still get way more space than is possible now. It would be more complicated, but I don't see any reason it shouldn't work...
     
    Last edited by einsteinx2, Sep 21, 2018
    DarthDub and EmBlaze like this.
  4. TR_mahmutpek

    TR_mahmutpek medic

    Member
    5
    Jul 28, 2015
    Turkey
    Holy mother... Real hardwork, thx for tutorial! If it will be ℅100 safe, I will do it. But I have 1TB drive:ha:
     
    einsteinx2 likes this.
  5. Alex4nder001

    Alex4nder001 Part-time thug

    Member
    3
    Jul 28, 2017
    United Kingdom
    I have a 1TB drive, and the free space was about 650GB to start with, wtf?
     
    einsteinx2 likes this.
  6. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    Haha thanks. While nothing is 100% safe, if you're somewhat familiar with Linux and you follow the instructions you shouldn't have any issues. Eventually I'd like to write something to just run right on your PS3 to make it full proof, but I'm holding off spending more time on this until I use it on my own machine for a while and see if I run into any performance issues. As far as the actual changes it's making, they're totally reversible and as far as I can tell, has no chance of causing any kind of corruption or anything like that as it's just manipulating a standard feature of the UFS2 filesystem the PS3 uses.

    Also, no reason not to do it on a 1TB drive, my whole reason for working on this was to get more space on my 1.5TB :)

    650GB seems a bit low, but basically this is what happens: The hard drive makers count in base 10, while computer operating systems generally count in base 2. They do this for marketing reasons to be deceptive. Basically a 1 TB drive is not 1 TB, it's 1 trillion bytes. 1 TB is actually 1,099,511,627,776 bytes, or 1024 GB. 1,000,000,000,000 bytes is actually around 931 GB, so you're losing close to 100 GB right away due to marketing bullshit. Then on top of that, the PS3 reserves 8% of the space, so immediately you'll see 856 GB free minus whatever space the PS3 uses for the OS and whatnot.

    So you're somehow missing another 200 GB, not sure what happened there.
     
    Last edited by einsteinx2, Sep 21, 2018
    tbb043, djDougs and TR_mahmutpek like this.
  7. Alex4nder001

    Alex4nder001 Part-time thug

    Member
    3
    Jul 28, 2017
    United Kingdom
    Actually, it’s 1024GB, so it really is truly 1TB.
     
    einsteinx2 likes this.
  8. EddieEddie90

    EddieEddie90 GBAtemp Regular

    Member
    3
    Apr 24, 2016
    Mexico
    Can I do this in my Super Slim PS3 with HAN?
     
    einsteinx2 likes this.
  9. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    As long as you can dump your eid root key or your hdd keys then you can do it. Otherwise make and model of PS3 should make no difference. I don’t have experience with HAN, as I have a slim with REBUG, but if there is any tool to dump/backup your keys then you’re good to go.

    The modification itself has to do with the filesytem that all PS3’s use, so as long as you can decrypt and mount it to make the required changes, the PS3 should recognize it.
     
    Last edited by einsteinx2, Sep 22, 2018
  10. seren3

    seren3 GBAtemp Regular

    Member
    2
    Jul 16, 2018
    United States
    Wow, amazing work!
     
    einsteinx2 likes this.
  11. uyjulian

    uyjulian Homebrewer

    Member
    7
    Nov 26, 2012
    United States
    United States
    No. The abbreviation units are usually misleading thanks to the storage device manufacturers. They may be base 2 or base 10.

    For accurate comparison purposes, use the space in bytes, not any other unit.
     
  12. aphirst

    aphirst "Aren't girls with pee-pees just men?"

    Member
    4
    Oct 22, 2007
    United Kingdom
    Hull
    It's common convention to use MB/GB/TB to mean million/billion/trillion bytes (SI), but to use MiB/GiB/TiB to mean 1024* each previous amount (binary).
     
  13. bootmonster

    bootmonster GBAtemp Fan

    Member
    5
    Oct 26, 2002
    United States
    Fantastic stuff! Now to break the ~1.8tb drive limit!
     
    einsteinx2 likes this.
  14. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    Already starting work on that now ;)

    2TB will likely be the hard max, but at least it’s something! Also once 3TB drives get thinner, it should be possible to get even more as a 2TB drive is really more like 1862GB. A 3TB drive would allow for maxing out the true limit of 2048GB for almost 200 extra GB (at the expense of wasting space, but who cares when they get as cheap as 2TB are now. It’s already possible to do this on the original XBOX which has the same 2TB drive limit using custom formatting tools.
     
  15. Nimrod-002

    Nimrod-002 GBAtemp Regular

    Member
    3
    Dec 8, 2016
    Morocco
    i was always interested in doing this but the procedure is too drastic, i hope someone make a homebrew for this for a one click fix.
     
    SpiffyJUNIOR likes this.
  16. Nimrod-002

    Nimrod-002 GBAtemp Regular

    Member
    3
    Dec 8, 2016
    Morocco
    irisman has an integrated file manager/hex editor so wouldnt it be possible to do this on console?
     
  17. uyjulian

    uyjulian Homebrewer

    Member
    7
    Nov 26, 2012
    United States
    United States
    It might be possible to do this on console with OtherOS++
     
  18. Nimrod-002

    Nimrod-002 GBAtemp Regular

    Member
    3
    Dec 8, 2016
    Morocco
    no i dont think otheros has gameos region access
     
  19. uyjulian

    uyjulian Homebrewer

    Member
    7
    Nov 26, 2012
    United States
    United States
    yes it does
     
    einsteinx2 likes this.
  20. einsteinx2
    OP

    einsteinx2 Advanced Member

    Newcomer
    2
    May 3, 2018
    Colombia
    If I understand it correctly, without modification, Irisman can’t be used as it’s showing the mounted partitions not the raw partition devices. In any case, I looked into the PS3 open source sdk a bit and I’m pretty sure I can write a tool to do this that could allow you to choose whatever percent you want as well as toggle between space and time modes.

    I’m pretty busy at work this week but I’ll be messing around with it at night. I’ll post progress on that as I make any.

    It definitely can be done in OtherOS++ Linux. In fact there is a page on the psdevwiki showing how to do similar steps to create the partition devices and after that the same dd commands from my tutorial can be used.
     
Loading...