Capacity and Filesystem affecting Performance?

[XFoxPrower]

It seems bigger isn't always better. Many guides say 'The bigger cards are always faster', period. But maybe there's a reason that Nintendo officially says 32GB is the max for the 3DS, or why Microsoft limits FAT32 to 32GB.

But filesystem overhead seems to choke available resources, and homebrew is more prone to see issues due to pushing the limits. Devs often take shortcuts, don't know the 'proper' approach, or simply don't have access to the official libraries (3rd party gamepads not recogninzed by some games, etc).

I've noticed that on Vita, it seems performance gets worse the more full a card gets. Copying a single 256 KB file can take minutes as you approach 1TB (only half full!). And Vitashell can't handle FS latency with Adhoc transfers, requiring a hard shutdown (Hold Power 30 secs) if something falls out of sync. It's known that Final Fantasy 5 crashes after every other battle. But if you have a bunch of apps on your home screen, a crash is fatal and requires another hard shutdown. Each improper shutdown (particularly during file transfer crashes) has a chance of corrupting the filesystem further...

That's my oversvation and rant. Anyone have thoughts or issues to cope with this situation? (Some of these cases cited are Vita-specific but I'm posting in General Computing as the concept is broad).

 

[Latiodile]

most of these symptoms primarily seem like console edge cases where they just weren't made to handle such large storage, the ps vita released n 2011 and 2tb flash storage didn't exist at the time, much less vita memory sticks, typically when i experience slowdown it's when an sd card gets full which is natural for flash storage to slow down based on how full it is

bigger card always faster is purely a myth, you can have a 2tb SD card but it'll be slower than a 256gb A2 rated card, especially on the 3ds where the bigger the card the slower it is to boot with it

 

[bonkmaykr]

I don't know the details of Sony's TexFAT implementation but I have a hunch that their very volatile database indexing is at least a little bit responsible for the poor filesystem performance. Also YAMT's gonna YAMT.

 

[Necron]

But maybe there's a reason that Nintendo officially says 32GB is the max for the 3DS, or why Microsoft limits FAT32 to 32GB.

The creator of the windows formatting tool mentioned not too long ago that 32GB is artificial, as that was made back in the late 90s and very quickly. The max supported is 2TB for FAT32 if I'm not mistaken.

As for the 3DS, I think those were the max cheap sd cards you can get or maybe because of its system memory, but it has nothing to do with "yeah they limit to this because this is the optimal".

Also I wonder what kind of filesystem the vita uses. For example, NTFS can get either very slow or very fast due to file fragmentation. I use linux and the ext4 filesystem has some measures against this, but on the other hand copying files can feel slower, but it actually just the time it takes (when using ext* filesystems). Plus take into consideration bandwidth of their chips, cpus, etc. Everything plays a part and not just the disk used.

 

[Exnor]

It seems bigger isn't always better. Many guides say 'The bigger cards are always faster', period. But maybe there's a reason that Nintendo officially says 32GB is the max for the 3DS, or why Microsoft limits FAT32 to 32GB.

But filesystem overhead seems to choke available resources, and homebrew is more prone to see issues due to pushing the limits. Devs often take shortcuts, don't know the 'proper' approach, or simply don't have access to the official libraries (3rd party gamepads not recogninzed by some games, etc).

I've noticed that on Vita, it seems performance gets worse the more full a card gets. Copying a single 256 KB file can take minutes as you approach 1TB (only half full!). And Vitashell can't handle FS latency with Adhoc transfers, requiring a hard shutdown (Hold Power 30 secs) if something falls out of sync. It's known that Final Fantasy 5 crashes after every other battle. But if you have a bunch of apps on your home screen, a crash is fatal and requires another hard shutdown. Each improper shutdown (particularly during file transfer crashes) has a chance of corrupting the filesystem further...

That's my oversvation and rant. Anyone have thoughts or issues to cope with this situation? (Some of these cases cited are Vita-specific but I'm posting in General Computing as the concept is broad).

It appears to be a problem with flash storage itself.

I had (and still have) similar problems with USB flash drives and some SDcards, on many different systems including Windows and Linux.

I think the problem resides in the way flash memory controllers do the job on SDcards and usb stick. Unlike SSD's, there is no garbage collection AFAIK, so every time it needs to write on a cell (or group off cells) that was occupied, 1st it needs to delete (takes a lot more time than write on flash) that cell to be able to write.
Since flash controllers always copy before deletion (they copy the data that is not marked for deletion to a new cell, and only then they erase those cells. Note that these flash storage uses MLC, TLC and even QLC, that means that to erase a cell it needs to erase a bunch of other cells too, so i needs to copy the data that is no marked for deletion), it means a bunch of erase and write is needed just to write a few bytes.

This makes the write process a lot slower, expecially if the sdcard is already near full, since there are, most likely, no never written before cells. So it needs to copy and delete a lot before writing the final data.

This is my theory based on my limited knoledge of flash technology.

Has for FS overhead. I dont think that is an issue, unless addressing that much data somehow requires a lot of register (CPU registers) "manipulation" that on itselft would consume more compute cycles (again my knoledge here is limited).

 

[XFoxPrower]

Thanks for the replies. I didn't know Vita's format was proprietary as it seemed to be PC-compatible. 3DS' slow boot is because it reads the entire table at boot (or kernel reboot) to calc free space. But these general symptoms present themselves on my PSTV too (though that's only at 1 TB) so I'm thinking it's just a (bad) design thing. Sure hope my storage isn't failing. The flash write & delete thing isn't just for SSDs? Also to mention, PS3's Managuns filemanager slows down to like 30 seconds per directory jump on local HDD just by connecting my external 5TB drive, so filesystem stuff must be a real RAM weight.

 

[Exnor]

Thanks for the replies. I didn't know Vita's format was proprietary as it seemed to be PC-compatible. 3DS' slow boot is because it reads the entire table at boot (or kernel reboot) to calc free space. But these general symptoms present themselves on my PSTV too (though that's only at 1 TB) so I'm thinking it's just a (bad) design thing. Sure hope my storage isn't failing. The flash write & delete thing isn't just for SSDs? Also to mention, PS3's Managuns filemanager slows down to like 30 seconds per directory jump on local HDD just by connecting my external 5TB drive, so filesystem stuff must be a real RAM weight.

I think Vita uses exfat for FS. Not sure.

As for the time it takes to enumerate the FS my theory is that it requires more compute cycles, not necessarily more RAM.

 

[bonkmaykr]

For example, NTFS can get either very slow or very fast due to file fragmentation

This is a physical property of mechanical drives and not of the filesystem.

but it has nothing to do with "yeah they limit to this because this is the optimal".

Well, depending on the desired sector size it can be, because what the filesystem is addressing is sectors, not bytes. Else the total limit for FAT32 would be 4 GB. The maximum filesystem size is a reflection of that, it's the size of each sector multiplied by the maximum number of sectors. (Technically sector refers to the physical sectors on the disk, I am not sure what the correct terminology would be: cluster, chunk, also sector?)

This sector limit is not 32-bit.

There is an optimal sector size for different hardware combinations, depending on the RAM available for the driver to buffer sector contents into, the maximum bandwidth over the bus, the size and alignment of the physical sectors that the logical sectors rest within, and whatever else.

I think Vita uses exfat for FS. Not sure.

Vita uses TexFAT on memory cards, at least when YAMT is running.

 

[Exnor]

This is a physical property of mechanical drives and not of the filesystem.


Well, depending on the desired sector size it can be, because what the filesystem is addressing is sectors, not bytes. Else the total limit for FAT32 would be 4 GB. The maximum filesystem size is a reflection of that, it's the size of each sector multiplied by the maximum number of sectors. (Technically sector refers to the physical sectors on the disk, I am not sure what the correct terminology would be: cluster, chunk, also sector?)

I think its called Clusters. Again not sure and i'm too lazy to search

 

[XFoxPrower]

Clusters are the set minimum block size of the filelsystem. With a 32KB cluster, a 4 KB file takes up 32KB. Then a 37KB file takes up 64KB. Probably ideal to align with sector size in mechanical drives. Picture handling several tiny packages vs larger packages physiclly: Bigger is easier but can be wasteful. Vita: I formatted a card in VitaDeploy and it said tEXFAT.
Still no idea how that's different, but it's beyond YAMT.

Well I have a greater appreciation for ISO files over say PS3's extracted JB Folder alternative now.