Hardware How does the PS4 and PS5 load game data?

[biolizardshadow]

My understanding is that from the 32bit generation onward that data was loaded from the disk into system ram where the CPU would run everything instead of trying to have the CPU read data directly from the disk which was a lot slower and not really possible. Now the PS5 has the nvme slot where you can plugin an external SSD to increase the amount of storage on the system. I watch Reviewtechusa on youtube and he seems to think that if you buy some cheap SSD that doesn't meet recommended read write speeds that the game will not run properly or something along those lines. So can the CPU load data directly from the disk without having to load it into system ram or does it always have to load it into ram first before the CPU can execute it?

This question also applies to the PS4 as well. Thanks in advance for anyone who responds to my post.

 

[godreborn]

from my understanding, ps4 games are encrypted with pfs (protected file system). I believe this protects all files. however, it's stripped of pfs when you boot the game, which I think is how we can run backups, minimizing the game after loading it. these files are placed in the sandbox folder: mnt/sandbox then I think pfsmnt which probably stands for protect file system mount.

--------------------- MERGED ---------------------------

further, pfs is bypassed (not defeated), because I think it would be too taxing to circumvent in real time, each file. that's my assumption anyway. we don't have the pfs keys, so we usually have to use the system to dump and decrypt content. @KiiWii or @Leeful may have a better explanation as this is based on my own experiences and my own understanding of how the system works. the vita also uses pfs and the ps5 probably does as well, since it's virtually impossible to crack. even though we've hacked the hell out of the vita, the pfs keys are still unknown afaik.

--------------------- MERGED ---------------------------

btw, you can dump games and patches via usb, but they're still decrypted in the same location. when you try dumping a game, it will give you a percentage of amount dumped, then it will say "decrypting selfs," then finally shutdown by default the system. themes and dlc are considered addcont, so they'll be in that folder. iirc, themes go in the ID000000 or something like that folder. it's an obvious folder. dlc will go in their own folder. when you dump say a theme, it will have an ac and an ac-nest folder. the ac-nest folder seems to be related to pfs and is not needed when making a fixed pkg or dumping the theme itself. dlc and themes are almost the same when it comes to making a fixed pkg. you just need to change the param.sfo (yeah, it's in lowercase this time compared with the ps3). there's only like six options for the param.sfo this time around using the orbis sfo editor. you can even put your name in the author, which will appear on the ps4 if you go to information for that theme. you may be able to do the same with dlc, but I haven't really messed with it.

 

[KiiWii]

Pretty much.. also:
https://www.psdevwiki.com/ps4/Talk:Keys#Portable_Keys_Dumps
https://www.psdevwiki.com/ps4/Talk:Keys#Portable_Keys_Dumps

“maxton of Maxton.xyz announced on Twitter that he updated the LibOrbisPkg PkgEditor adding support for EKPFS / XTS keys to access an encrypted PKG's file system alongside a keydb.json file to store the keys eliminating the need to paste them in every time a Package is opened.”

Tweak and data keys can be used to open PKG afaik, but idk about decryption, if the keys remove encryption... https://twitter.com/notzecoxao/status/963481474300370944?s=21

 

[tech3475]

Not a dev, but I’d imagine the read speed of the ssd matters because games can be designed with a certain expectation.

For example, if you know an asset will be loaded in x seconds you may not need to disguise it’s loading e.g. a long corridor, loading screen, etc.

 

[godreborn]

if you want a more technical explanation, it's beyond my personal understanding. btw, each pkg has a passphrase which I'm not sure how it's related. all 0s seem to be no passphrase like all 0s are no klicense on the ps3.

 

[Deleted User]

iirc, audio is not buffered in ram, but streamed directly from the hard drive as to improve performance, because of this, it may not be compressed, iirc Titanfall 2 was 35 gb in audio files alone

 

[tech3475]

iirc, audio is not buffered in ram, but streamed directly from the hard drive as to improve performance, because of this, it may not be compressed, iirc Titanfall 2 was 35 gb in audio files alone

That may be due to the Source Engine, last I checked, Source 1 only supports looped audio with wav files, at least on PC.

 

[Deleted User]

That may be due to the Source Engine, last I checked, Source 1 only supports looped audio with wav files, at least on PC.

the place i heard this was on a video talking about why newer games are so big, the two reasons were
-Repeated assets (necessary to speed up hdd loading)
-Lack of compression (decompression takes away cpu resources, iirc the same video said that if games played the same audio in mp3 320kbps, 5-15% of the cpu would just be used for pñaying audio)

 

[godreborn]

that's my understanding as well. games are only large due to lack of compression. I think someone posted a link showing that smash bros ultimate was actually over 60GBs uncompressed, but it's 12-15GBs using some ridiculous Nintendo compression. that could be one of the reasons why loading kinda sucks on the switch compared with the ps4.

 

[tech3475]

the place i heard this was on a video talking about why newer games are so big, the two reasons were
-Repeated assets (necessary to speed up hdd loading)
-Lack of compression (decompression takes away cpu resources, iirc the same video said that if games played the same audio in mp3 320kbps, 5-15% of the cpu would just be used for pñaying audio)

Source supports MP3, but not for looped audio.

That’s why HL2 uses MP3 (at 128kbps) for it’s BGM but Portal 2 used wav.

edit:

I don’t know if performance played a part in Valve’s decision, I only know from my experience messing around with the Source SDK.

 

[godreborn]

anyway, the technical side of things can be found on devwiki like @KiiWii mentioned in a previous post.

 

[biolizardshadow]

iirc, audio is not buffered in ram, but streamed directly from the hard drive as to improve performance, because of this, it may not be compressed, iirc Titanfall 2 was 35 gb in audio files alone

So audio is streamed from the disk/hard drive?

 

[Deleted User]

Source supports MP3, but not for looped audio.

That’s why HL2 uses MP3 (at 128kbps) for it’s BGM but Portal 2 used wav.

edit:

I don’t know if performance played a part in Valve’s decision, I only know from my experience messing around with the Source SDK.

the video wasnt about source, it was about games in general though

that's my understanding as well. games are only large due to lack of compression. I think someone posted a link showing that smash bros ultimate was actually over 60GBs uncompressed, but it's 12-15GBs using some ridiculous Nintendo compression. that could be one of the reasons why loading kinda sucks on the switch compared with the ps4.

yeah, sometimes due to space limitation devs had to take the performance hit of compression, i guess nintendo did this to be able to sell it on a 16gb cart, idk the difference in price but i'd imagine they saved a lot of money doing this

--------------------- MERGED ---------------------------

So audio is streamed from the disk/hard drive?

to my understanding, yes

 

[biolizardshadow]

So how fast is does it take for information to get from the disk to the CPU exactly? I was always under the impression that streaming from the disk was really slow and that's why loading screens really became a thing during the 32bit generation of consoles when home consoles switched to optical media.

 

[Tom Bombadildo]

https://www.tweaktown.com/news/7134...ep-dive-into-next-gen-storage-tech/index.html < Give this a read, it will give you some general information on how the PS5's custom flash controller can manage to push assets quickly from storage->RAM->CPU.

To tl;dr it though, basically the custom flash controller used in the PS5 can push around 5gb/s-9gb/s, meaning 5-9GBs worth of assets can be loaded to the RAM in each second, and that data can then be loaded by the CPU/GPU basically instantly (since the RAM's bandwidth is around 450gb/s, and they're using a custom chip that can decompress data quickly if the data has been compressed). As Cerny mentions, you can essentially load 4GB worth of data from storage->RAM->CPU in a single second, that's basically a good "average case scenario". The storage speeds aren't quite fast enough to load data directly to the CPU, but it is fast enough to load lots of data to the RAM which can then push lots of data to the CPU in basically an instant when it needs to.

Because games (at this time, anyways) don't use a huge amount of assets, and because the system can load so much data in just one second, load times can be virtually eliminated...if the developer uses the hardware properly, of course (which a few launch titles didn't, AssCreed Valhalla being one of them which has equivalent load times to PCs). And, for games that utilize the hardware, this is basically true. Demon's Souls Remake, for example, can go from the main menu to the game world in something like 5-6 seconds, and then from game world to game world it's basically 1-3 seconds.

The question about an off the shelf NVMe being fast enough is a valid concern, because if that NVMe can't push at least 4GB/s for a decent amount of time, it could very well have performance issues with games that are designed to take full advantage of the PS5's internal storage speeds. I suspect this is why Sony doesn't have that M.2 slot enabled at launch, because there aren't enough PCIe 4 NVMes currently available to buy that can support those kinds of speeds (and the ones that do exist are expensive).


However, I 100% expect that you can't just shove any off the shelf NVMe in there and call it good, Sony will either create a hard whitelist that won't let you use a slower drive at all, or they'll implement some kind of benchmark that must be passed before a drive can be used anyways to avoid those performance issues and so it's not something the end user will have to worry about so long as they purchase a proper drive.

 

[Codemastershock]

Nintendo games in general are a lot easier to compress because the lack of prerendered cutscenes, audio files in multiple languages and their "cartoonish" style, which use a lot of flat shaded textures. Compression is way more effective if the data have a lot of "patterns"(lossless compression), or you reduce the quality of the asset to reduce its size (lossy compression), there is no magical compression.

One example of lossless compression, if I create two text files, one with pear and one with banana written, obviously the pear file will be smaller because there is less characters, but if I use compression, the banana file wil be smaller because I can reduce the word to b [an x2] a, and I cant reduce the pear word anymore because all letters are unique and there is no pattern between them. This is a very, very simplified example, characters have different binary values and so on....

And even then, using lossless compression in Smash Bros is way easier than in a open world game, for example. Smash Bros have lenghty loadings, but the game is structured in levels, so as soon the decompression ends, the console allocate resources to that stage alone and the decompression algorythm go away, when you change the stage everything is cleared to make room for another stage to be decompressed... On an open world game everything has to be done very quickly and lossless decompression techniques waste a lot of hardware resources to be running constantly, nobody would want to play a game that stutters and slow down everytime, the only option to reduce the file size is to crank the quality of the assets down so the file shrinks too. One game that I remember pirates used lossless compression to make it work on a smaller media is Skies of Arcadia for the Dreamcast because SEGA implemented some integrity checks on the game and if the files were changed, the game froze, so the only alternative was lossless compression in real time. It is impressive, yes, but the user experience is very miserable.

What makes modern games so bloated in general are audio and video files because sometimes it is not worth it to change the design aesthetic entirely to save data. Everytime you install any game that is not Nintendo, every language is installed regardless if you use them or not, additional languages should and especially now, need to be optional downloads.

 

[biolizardshadow]

https://www.tweaktown.com/news/7134...ep-dive-into-next-gen-storage-tech/index.html < Give this a read, it will give you some general information on how the PS5's custom flash controller can manage to push assets quickly from storage->RAM->CPU.

To tl;dr it though, basically the custom flash controller used in the PS5 can push around 5gb/s-9gb/s, meaning 5-9GBs worth of assets can be loaded to the RAM in each second, and that data can then be loaded by the CPU/GPU basically instantly (since the RAM's bandwidth is around 450gb/s, and they're using a custom chip that can decompress data quickly if the data has been compressed). As Cerny mentions, you can essentially load 4GB worth of data from storage->RAM->CPU in a single second, that's basically a good "average case scenario". The storage speeds aren't quite fast enough to load data directly to the CPU, but it is fast enough to load lots of data to the RAM which can then push lots of data to the CPU in basically an instant when it needs to.

Because games (at this time, anyways) don't use a huge amount of assets, and because the system can load so much data in just one second, load times can be virtually eliminated...if the developer uses the hardware properly, of course (which a few launch titles didn't, AssCreed Valhalla being one of them which has equivalent load times to PCs). And, for games that utilize the hardware, this is basically true. Demon's Souls Remake, for example, can go from the main menu to the game world in something like 5-6 seconds, and then from game world to game world it's basically 1-3 seconds.

The question about an off the shelf NVMe being fast enough is a valid concern, because if that NVMe can't push at least 4GB/s for a decent amount of time, it could very well have performance issues with games that are designed to take full advantage of the PS5's internal storage speeds. I suspect this is why Sony doesn't have that M.2 slot enabled at launch, because there aren't enough PCIe 4 NVMes currently available to buy that can support those kinds of speeds (and the ones that do exist are expensive).


However, I 100% expect that you can't just shove any off the shelf NVMe in there and call it good, Sony will either create a hard whitelist that won't let you use a slower drive at all, or they'll implement some kind of benchmark that must be passed before a drive can be used anyways to avoid those performance issues and so it's not something the end user will have to worry about so long as they purchase a proper drive.

Interesting! So I assume that most physical games on the PS5 will have a download of some type? I just don't see the disk reader being able to load 5.5 gb/s when it comes to loading data off of the disk and into memory to then display in game. It also, sounds like the NVME slot kind of acts as an expandable ram for the PS5 kind of like the how the GBA slot on the DS could be used to expand its internal ram.

Also for data decompression (at least for audio) devs could always use sequenced audio for music instead of streamed audio. Whatever happened to sequenced audio in the first place? It seems like games don't use that method of playing music anymore?

 

[Tom Bombadildo]

Correct, the PS5 will not run any PS5 game from a disc itself as Blu-ray is much too slow, it'll need to be installed to the disc in order to be played. Most PS4 games later in the consoles life also required data be installed to the HDD for the same reason, Blu-ray is just too slow nowadays for running games off them.

 

[Deleted User]

Correct, the PS5 will not run any PS5 game from a disc itself as Blu-ray is much too slow, it'll need to be installed to the disc in order to be played. Most PS4 games later in the consoles life also required data be installed to the HDD for the same reason, Blu-ray is just too slow nowadays for running games off them.

iirc it was the 7th gen that did it the way you said, with 8th gen requiring installing for every game (i could be wrong though, i never had a ps4 or Xbox One)

 

[biolizardshadow]

Sorry for the late reply XD! So is data on the disc loaded into memory or is it downloaded from some server somewhere? If the data is downloaded from a server somewhere then what would be the point of having physical media?