1. biolizardshadow

    OP biolizardshadow Advanced Member
    Newcomer

    Joined:
    May 4, 2020
    Messages:
    68
    Country:
    United States
    My understanding is that from the 32bit generation onward that data was loaded from the disk into system ram where the CPU would run everything instead of trying to have the CPU read data directly from the disk which was a lot slower and not really possible. Now the PS5 has the nvme slot where you can plugin an external SSD to increase the amount of storage on the system. I watch Reviewtechusa on youtube and he seems to think that if you buy some cheap SSD that doesn't meet recommended read write speeds that the game will not run properly or something along those lines. So can the CPU load data directly from the disk without having to load it into system ram or does it always have to load it into ram first before the CPU can execute it?

    This question also applies to the PS4 as well. Thanks in advance for anyone who responds to my post.
     
  2. godreborn

    godreborn Retired
    Member

    Joined:
    Oct 10, 2009
    Messages:
    12,927
    Country:
    United States
    from my understanding, ps4 games are encrypted with pfs (protected file system). I believe this protects all files. however, it's stripped of pfs when you boot the game, which I think is how we can run backups, minimizing the game after loading it. these files are placed in the sandbox folder: mnt/sandbox then I think pfsmnt which probably stands for protect file system mount.

    — Posts automatically merged - Please don't double post! —

    further, pfs is bypassed (not defeated), because I think it would be too taxing to circumvent in real time, each file. that's my assumption anyway. we don't have the pfs keys, so we usually have to use the system to dump and decrypt content. @KiiWii or @Leeful may have a better explanation as this is based on my own experiences and my own understanding of how the system works. the vita also uses pfs and the ps5 probably does as well, since it's virtually impossible to crack. even though we've hacked the hell out of the vita, the pfs keys are still unknown afaik.

    — Posts automatically merged - Please don't double post! —

    btw, you can dump games and patches via usb, but they're still decrypted in the same location. when you try dumping a game, it will give you a percentage of amount dumped, then it will say "decrypting selfs," then finally shutdown by default the system. themes and dlc are considered addcont, so they'll be in that folder. iirc, themes go in the ID000000 or something like that folder. it's an obvious folder. dlc will go in their own folder. when you dump say a theme, it will have an ac and an ac-nest folder. the ac-nest folder seems to be related to pfs and is not needed when making a fixed pkg or dumping the theme itself. dlc and themes are almost the same when it comes to making a fixed pkg. you just need to change the param.sfo (yeah, it's in lowercase this time compared with the ps3). there's only like six options for the param.sfo this time around using the orbis sfo editor. you can even put your name in the author, which will appear on the ps4 if you go to information for that theme. you may be able to do the same with dlc, but I haven't really messed with it.
     
    E1ite007 and KiiWii like this.
  3. KiiWii

    KiiWii Reporter
    Reviewer

    Joined:
    Nov 17, 2008
    Messages:
    11,005
    Country:
    United Kingdom
    Pretty much.. also:

    https://www.psdevwiki.com/ps4/Talk:Keys#Portable_Keys_Dumps

    maxton of Maxton.xyz announced on Twitter that he updated the LibOrbisPkg PkgEditor adding support for EKPFS / XTS keys to access an encrypted PKG's file system alongside a keydb.json file to store the keys eliminating the need to paste them in every time a Package is opened.”

    Tweak and data keys can be used to open PKG afaik, but idk about decryption, if the keys remove encryption... https://twitter.com/notzecoxao/status/963481474300370944?s=21
     
    godreborn likes this.
  4. tech3475

    tech3475 GBAtemp Addict
    Member

    Joined:
    Jun 12, 2009
    Messages:
    2,004
    Country:
    Not a dev, but I’d imagine the read speed of the ssd matters because games can be designed with a certain expectation.

    For example, if you know an asset will be loaded in x seconds you may not need to disguise it’s loading e.g. a long corridor, loading screen, etc.
     
  5. godreborn

    godreborn Retired
    Member

    Joined:
    Oct 10, 2009
    Messages:
    12,927
    Country:
    United States
    if you want a more technical explanation, it's beyond my personal understanding. btw, each pkg has a passphrase which I'm not sure how it's related. all 0s seem to be no passphrase like all 0s are no klicense on the ps3.
     
    KiiWii likes this.
  6. Azerus_Kun

    Azerus_Kun Pro Plasher
    Member

    Joined:
    Dec 5, 2019
    Messages:
    572
    Country:
    Uruguay
    iirc, audio is not buffered in ram, but streamed directly from the hard drive as to improve performance, because of this, it may not be compressed, iirc Titanfall 2 was 35 gb in audio files alone
     
  7. tech3475

    tech3475 GBAtemp Addict
    Member

    Joined:
    Jun 12, 2009
    Messages:
    2,004
    Country:
    That may be due to the Source Engine, last I checked, Source 1 only supports looped audio with wav files, at least on PC.
     
    Azerus_Kun likes this.
  8. Azerus_Kun

    Azerus_Kun Pro Plasher
    Member

    Joined:
    Dec 5, 2019
    Messages:
    572
    Country:
    Uruguay
    the place i heard this was on a video talking about why newer games are so big, the two reasons were
    -Repeated assets (necessary to speed up hdd loading)
    -Lack of compression (decompression takes away cpu resources, iirc the same video said that if games played the same audio in mp3 320kbps, 5-15% of the cpu would just be used for pñaying audio)
     
  9. godreborn

    godreborn Retired
    Member

    Joined:
    Oct 10, 2009
    Messages:
    12,927
    Country:
    United States
    that's my understanding as well. games are only large due to lack of compression. I think someone posted a link showing that smash bros ultimate was actually over 60GBs uncompressed, but it's 12-15GBs using some ridiculous Nintendo compression. that could be one of the reasons why loading kinda sucks on the switch compared with the ps4.
     
    Azerus_Kun likes this.
  10. tech3475

    tech3475 GBAtemp Addict
    Member

    Joined:
    Jun 12, 2009
    Messages:
    2,004
    Country:
    Source supports MP3, but not for looped audio.

    That’s why HL2 uses MP3 (at 128kbps) for it’s BGM but Portal 2 used wav.

    edit:

    I don’t know if performance played a part in Valve’s decision, I only know from my experience messing around with the Source SDK.
     
    Last edited by tech3475, Nov 21, 2020 at 9:58 PM
    Azerus_Kun likes this.
  11. godreborn

    godreborn Retired
    Member

    Joined:
    Oct 10, 2009
    Messages:
    12,927
    Country:
    United States
    anyway, the technical side of things can be found on devwiki like @KiiWii mentioned in a previous post.
     
    Azerus_Kun and KiiWii like this.
  12. biolizardshadow

    OP biolizardshadow Advanced Member
    Newcomer

    Joined:
    May 4, 2020
    Messages:
    68
    Country:
    United States
    So audio is streamed from the disk/hard drive?
     
    Azerus_Kun likes this.
  13. Azerus_Kun

    Azerus_Kun Pro Plasher
    Member

    Joined:
    Dec 5, 2019
    Messages:
    572
    Country:
    Uruguay
    the video wasnt about source, it was about games in general though
    yeah, sometimes due to space limitation devs had to take the performance hit of compression, i guess nintendo did this to be able to sell it on a 16gb cart, idk the difference in price but i'd imagine they saved a lot of money doing this

    — Posts automatically merged - Please don't double post! —

    to my understanding, yes
     
  14. biolizardshadow

    OP biolizardshadow Advanced Member
    Newcomer

    Joined:
    May 4, 2020
    Messages:
    68
    Country:
    United States
    So how fast is does it take for information to get from the disk to the CPU exactly? I was always under the impression that streaming from the disk was really slow and that's why loading screens really became a thing during the 32bit generation of consoles when home consoles switched to optical media.
     
  15. Tom Bombadildo

    Tom Bombadildo Dick, With Balls
    Reviewer

    Joined:
    Jul 11, 2009
    Messages:
    13,617
    Country:
    United States
    https://www.tweaktown.com/news/7134...ep-dive-into-next-gen-storage-tech/index.html < Give this a read, it will give you some general information on how the PS5's custom flash controller can manage to push assets quickly from storage->RAM->CPU.

    To tl;dr it though, basically the custom flash controller used in the PS5 can push around 5gb/s-9gb/s, meaning 5-9GBs worth of assets can be loaded to the RAM in each second, and that data can then be loaded by the CPU/GPU basically instantly (since the RAM's bandwidth is around 450gb/s, and they're using a custom chip that can decompress data quickly if the data has been compressed). As Cerny mentions, you can essentially load 4GB worth of data from storage->RAM->CPU in a single second, that's basically a good "average case scenario". The storage speeds aren't quite fast enough to load data directly to the CPU, but it is fast enough to load lots of data to the RAM which can then push lots of data to the CPU in basically an instant when it needs to.

    Because games (at this time, anyways) don't use a huge amount of assets, and because the system can load so much data in just one second, load times can be virtually eliminated...if the developer uses the hardware properly, of course (which a few launch titles didn't, AssCreed Valhalla being one of them which has equivalent load times to PCs). And, for games that utilize the hardware, this is basically true. Demon's Souls Remake, for example, can go from the main menu to the game world in something like 5-6 seconds, and then from game world to game world it's basically 1-3 seconds.

    The question about an off the shelf NVMe being fast enough is a valid concern, because if that NVMe can't push at least 4GB/s for a decent amount of time, it could very well have performance issues with games that are designed to take full advantage of the PS5's internal storage speeds. I suspect this is why Sony doesn't have that M.2 slot enabled at launch, because there aren't enough PCIe 4 NVMes currently available to buy that can support those kinds of speeds (and the ones that do exist are expensive).


    However, I 100% expect that you can't just shove any off the shelf NVMe in there and call it good, Sony will either create a hard whitelist that won't let you use a slower drive at all, or they'll implement some kind of benchmark that must be passed before a drive can be used anyways to avoid those performance issues and so it's not something the end user will have to worry about so long as they purchase a proper drive.
     
  16. Codemastershock

    Codemastershock GBAtemp Regular
    Member

    Joined:
    Oct 14, 2013
    Messages:
    174
    Country:
    Brazil
    Nintendo games in general are a lot easier to compress because the lack of prerendered cutscenes, audio files in multiple languages and their "cartoonish" style, which use a lot of flat shaded textures. Compression is way more effective if the data have a lot of "patterns"(lossless compression), or you reduce the quality of the asset to reduce its size (lossy compression), there is no magical compression.

    One example of lossless compression, if I create two text files, one with pear and one with banana written, obviously the pear file will be smaller because there is less characters, but if I use compression, the banana file wil be smaller because I can reduce the word to b [an x2] a, and I cant reduce the pear word anymore because all letters are unique and there is no pattern between them. This is a very, very simplified example, characters have different binary values and so on....

    And even then, using lossless compression in Smash Bros is way easier than in a open world game, for example. Smash Bros have lenghty loadings, but the game is structured in levels, so as soon the decompression ends, the console allocate resources to that stage alone and the decompression algorythm go away, when you change the stage everything is cleared to make room for another stage to be decompressed... On an open world game everything has to be done very quickly and lossless decompression techniques waste a lot of hardware resources to be running constantly, nobody would want to play a game that stutters and slow down everytime, the only option to reduce the file size is to crank the quality of the assets down so the file shrinks too. One game that I remember pirates used lossless compression to make it work on a smaller media is Skies of Arcadia for the Dreamcast because SEGA implemented some integrity checks on the game and if the files were changed, the game froze, so the only alternative was lossless compression in real time. It is impressive, yes, but the user experience is very miserable.

    What makes modern games so bloated in general are audio and video files because sometimes it is not worth it to change the design aesthetic entirely to save data. Everytime you install any game that is not Nintendo, every language is installed regardless if you use them or not, additional languages should and especially now, need to be optional downloads.
     
    Last edited by Codemastershock, Nov 23, 2020 at 1:47 AM
  17. biolizardshadow

    OP biolizardshadow Advanced Member
    Newcomer

    Joined:
    May 4, 2020
    Messages:
    68
    Country:
    United States
    Interesting! So I assume that most physical games on the PS5 will have a download of some type? I just don't see the disk reader being able to load 5.5 gb/s when it comes to loading data off of the disk and into memory to then display in game. It also, sounds like the NVME slot kind of acts as an expandable ram for the PS5 kind of like the how the GBA slot on the DS could be used to expand its internal ram.

    Also for data decompression (at least for audio) devs could always use sequenced audio for music instead of streamed audio. Whatever happened to sequenced audio in the first place? It seems like games don't use that method of playing music anymore?
     
    Azerus_Kun likes this.
  18. Tom Bombadildo

    Tom Bombadildo Dick, With Balls
    Reviewer

    Joined:
    Jul 11, 2009
    Messages:
    13,617
    Country:
    United States
    Correct, the PS5 will not run any PS5 game from a disc itself as Blu-ray is much too slow, it'll need to be installed to the disc in order to be played. Most PS4 games later in the consoles life also required data be installed to the HDD for the same reason, Blu-ray is just too slow nowadays for running games off them.
     
    Codemastershock and Azerus_Kun like this.
  19. Azerus_Kun

    Azerus_Kun Pro Plasher
    Member

    Joined:
    Dec 5, 2019
    Messages:
    572
    Country:
    Uruguay
    iirc it was the 7th gen that did it the way you said, with 8th gen requiring installing for every game (i could be wrong though, i never had a ps4 or Xbox One)