Hacking One of a kind bug

[Relican]

It requires a a little backstory. Ill write it with bullet points for clearer message.
-Have an offline console with x-project
-Leave the room while a friend plays PSVR game. He says something crashed but console didn't shut off i guess because he was still playing the game when i came back. turn console off for the day.
-Boot again next day. Console greets me with database rebuild. All the fpkg are gone.
-Google around, find and successfully use ps4_DB_rebuilder. Everything goes back to normal.
-A week later im trying to delete a game but "Delete" option is gone.
Here's what i figured out. Game is delete'able unless i run it. After i boot a game, the delete option disappears. From "context" menu and from "Applications" in mass storage setting menu(they are completely gone from a list). Also notice the 359,6GB size of the game. Other games have different but also ridiculously big sizes like Bastion having 230GB. Rebooting without and/or with hen still doesn't show delete option anymore.


Is there any fix for that?

Update time. I rebuilt database from safe menu again. Fpkg were all gone as expected. Used ps4_DB_rebuilder again. Now i can delete games again unless i boot them.
Before this event i had quite a few different crashes and freezes, (system integrity check included!) but never did it caused me problems with deleting fpkgs.

 

[Leeful]

I've just done some tests and the exact same problem hapened to me.

Rebuilt database > fake pkgs were gone > Used ps4_DB_rebuilder > fake pkgs were back
The delete option is there but once you have run a game the delete option dissapears!

I also noticed that the UnDeletable games do not show up in 'system storage' either.
Also the UnDeletable games have 'parental controls' appearing as - in the game information.

I'll look more into it but this is definatly a problem.

 

[Leeful]

Right, did some more testing and you can delete the games by manually editing the app.db

Change the 'canRemove' entry for the game from 0 to 1 in the tbl_appbrowse table of the db.

I edited the database for one of the problem games and the delete option was back.
I then ran the game and the delete option was still there.
Before I deleted the game I checked my system storage status and after the game was deleted the system storage changed as it should. So all is good.

while I was checking the database I also noticed that the following entries were also wrong for the problem games.
canRemove = 0 (should be 1)
parental level = 0 (usually 5)
content size = 0 (should be the game size)
install date = null (should be a date)

I'll have a look at the ps4_DB_rebuilder python script but I dont know if I'll be able to fix anything. I'm not very good with python.


EDIT:
As I thought, I cant do anything with it.
All the db entries look fine, but we already knew that because all the rebuilt games look fine at first when the db has been rebuilt.
The problem with the db entries only happen after you run the game and I have no idea what causes that.

 

[Relican]

Thanks for giving it a try. It looks like rebuilding database does something, somewhere to fpkg games. I Installed a new game and "delete" option stays visible after rebooting or using hen. Im not gonna bother checking if rebuilding database + ps4_DB_rebuilder will break my new game, as im sure it will. So now i can only ask kindly for a program like ps4_DB_rebuilder but which adds again "deletable" value. For convenience's sake.
About ps4_DB_rebuilder. Shouldn't it print at the end

print("Completed")
else :
print("Skipped")

instead of finishing at print("Processing table: tbl_appinfo")?

 

[Leeful]

At the moment the best thing to do is to regularly back up your working database.
That way if your PS4 ever forces a database rebuild in the future you can easily restore it.

The good thing about using this method instead of DB_Rebuilder is that ALL your pkgs will be restored.
Another problem with the DB_Rebuilder script is that it will not rebuild any homebrew apps like emulators, Remote PKG Installer or Lappy's PS4 Explorer.

When you restore a backup that you have made then you will have everything back and working properly.

--------------------- MERGED ---------------------------

I was thinking about making a modded version of HEN 2.1.1 that also makes a backup of your app.db on the PS4 HDD every time you run it.

That way you would always have an up to date version oy your database backed up on your PS4 and if a forced database rebuild ever happened all you would need to do is run DB_Restore and everything would be back to normal.

 

[Relican]

After an evening nap i decided to test if "db sg backup" and "db restore" by Stooged from x-project works. It doesn't. Even after trying for the third time. all my games are still gone after official DB Rebuild and "db restore". Is "checking system's storage memory" window with progress bar after a reboot a normal thing? Now ps4_DB_rebuilder gives a different output in the console. Instad of just listing everything without a new line key, its all printed more neatly and tbl_appbrowse and tble_appinfo actually does some work. Second run of "ps4_DB_rebuilder" gives the usual output. Now the weird part. The new game/shovel knight i wrote about in the previous post is now gone from browser but listed in system storage, size on the list says 137mb but in the detail its 296gb, "application starting" is "ineligible" but that's probably because i didn't activate hen. Im trying to restore stooged backupp through ftp. Location of the contents of Db_backup is explained but where do i place stuff from "UserData" folder? Luckily after copying backup through ftp shovel knight is visible again! And deletable too. Old games lose "delete" option as usual.
My nomination for a culprit is still database rebuild and app.db. If i knew how, I'd check differences in app.db between different states- original/after system's db restore/after ps4_DB_rebuilder script. Perhaps there's something else that needs changing values in order to keep games deletable.

 

[MostlyUnharmful]

My nomination for a culprit is still database rebuild and app.db. If i knew how, I'd check differences in app.db between different states- original/after system's db restore/after ps4_DB_rebuilder script.

I looked at the PS4_db_rebuilder script, the DB used it's sqlite3 so you'll find plenty of utilities to play with it even if you don't know SQL.

To compare two instances of the same DB I was going to suggest dumping the DBs (better, dumping the tables) in CSV or SQL format and diff the two text files, but SQLite is distributed with an *sqldiff* utility https://www.sqlite.org/sqldiff.html the output is in SQL syntax tho.

If you aren't at ease with SQL I would suggest diffing two CSV files, without external utilities and using only the sqlite3 client, you simply set the ".mode=csv" and ".output" to a file-path, something like ".output=/path/to/db-table.csv", and then "select * from table;", the output will be dumped to that file above...

Or use a frontend.

 

[Leeful]

@Relican db_backup and db_restore work fine.

The reason why they didnt work for you is because you have to create a backup BEFORE your database messes up.
If you didn't make a backup before (while your PS4 was working perfetly) then they will not work.

I have a WORKING backup of my databases and I restore it all the time when I am testing things and wipe my database and it always works fine.

That is why I came up with the idea of making a modded version of Hen 2.1.1 that creates a backup of your databases every time you run it.
That way you will always have an up to date version of your database on your PS4 HDD.

Although i think this is a good idea I have found a massive problem with it.

Imagine this scenario:
There is a problem with your PS4 and your DB is rebuilt and all your fake pkgs are gone.
If you run Hen, it would automatically make a backup of the BAD DB and your GOOD DB backup would be overwritten and lost!
The only way to avoid this would be to NOT run Hen first but to instantly run db_restore instead.
That would work, BUT...
Because we are all in the habbit of running Hen first when our PS4 boots it is more than likely that we would run Hen first by accident and that would overwrite the GOOD backup.

Anyone have any ideas?

 

[smf]

Anyone have any ideas?

An integrity check of the database vs the contents of the hard disk before backing up and if it fails then restore it?

It's kinda weird though that the database can't be rebuilt. I'd have thought focusing on what is wrong with that would be the first step.

Backup your working database, rebuild the database, compare the two & figure out why they are different.

 

[Leeful]

I thought of this:
if the filesize of app.db is less than the filesize of app.db_backup > do not make a backup.

That would work if the db was rebuilt because the size of the db would be smaller but what if you deleted some games? that would also make the db smaller and that would cause it to not make a backup also so that idea would not work.


I agree, trying to find out why the db_rebuilder script is not working should be the first step.
When the db is rebuilt with db_rebuilder it all looks fine. the problem only occurs when you run a game, then the database entry for that game is changed.

I'm gonna make some backups of my db in various states.
1 working backup
2 after official rebuild
3 after db_rebuilder
4 after a game has run

I'll post my db backups here when Ive done it.

 

[smf]

it might be worth doing

1a run a game, take a backup

In case there is something that is always updated when you run a game

 

[Leeful]

Here are all my app.db files in various states:

1 working backup app.db (database with everything working fine)
2 after official rebuild app.db (database after PS4 rebuilt the database - all fake pkgs gone)
3 after db_rebuilder app.db (games are back but not homebrew 'PS4 Explorer' and 'Remote PKG Installer')
4 after Worms WMD run app.db (database after ony running Worms WMD)
5 after all games have been run app.db (database after running all the games once)



EDIT: after all tests I restored my working database backup and everything is back working fine.

 

[Relican]

Here are all my app.db files

I did some searching for a software with capabilities of file comparison but it seems there are none or are for server databases only.

 

[Padremayi]

Here are all my app.db files in various states:

1 working backup app.db (database with everything working fine)
2 after official rebuild app.db (database after PS4 rebuilt the database - all fake pkgs gone)
3 after db_rebuilder app.db (games are back but not homebrew 'PS4 Explorer' and 'Remote PKG Installer')
4 after Worms WMD run app.db (database after ony running Worms WMD)
5 after all games have been run app.db (database after running all the games once)



EDIT: after all tests I restored my working database backup and everything is back working fine.

Hi Leeful, in your working DB, the number 1, did you have some PSN game or all digital games are fPKG? I'm trying to restore my original PSN games, I have the bakcup of all .rif and .idx files but my original app.db is lost. I would like to find the correlation between the values on the tables for a PSN game. Do you have this type of backup? Need only one entry

 

[KiiWii]

Hi Leeful, in your working DB, the number 1, did you have some PSN game or all digital games are fPKG? I'm trying to restore my original PSN games, I have the bakcup of all .rif and .idx files but my original app.db is lost. I would like to find the correlation between the values on the tables for a PSN game. Do you have this type of backup? Need only one entry

@Leeful

 

[godreborn]

I've asked in the dev discord where the act.dat is. it might be in system_data/priv/drm , but my folder is empty.

 

[Padremayi]

I've asked in the dev discord where the act.dat is. it might be in system_data/priv/drm , but my folder is empty.

Do you have a current working PSN game installed in your PS4? If YES, could you please share your app.db, not totally if you don't want, only with the row that reference to the working PSN game? I have the rif and idx files, with the correct reference, using your working values as a base example, maybe I can start my game again

EDIT: with PSN game I mean a retail PKG, not fPKG

 

[godreborn]

I do not. my old psn account got banned due to the ps3, so I reformatted the system. I haven't gotten a response about the location as of yet. something must happen when you remove the hdd, because I've heard that games become locked again if you do that.

 

[Padremayi]

I do not. my old psn account got banned due to the ps3, so I reformatted the system. I haven't gotten a response about the location as of yet. something must happen when you remove the hdd, because I've heard that games become locked again if you do that.

Yes, this is why I need a database with almost one PSN game working

 

[Leeful]

Hi Leeful, in your working DB, the number 1, did you have some PSN game or all digital games are fPKG? I'm trying to restore my original PSN games, I have the bakcup of all .rif and .idx files but my original app.db is lost. I would like to find the correlation between the values on the tables for a PSN game. Do you have this type of backup? Need only one entry

Just checked it and unfortunately the games in the db were all fake pkgs.

At some point on my 5.05 PS4 I had some officially downloaded PSN games on it. I ended up loosing them when I changed the HDD (official backup did not transfer the licences to the new HDD etc)

It's a long shot but I might have a backed up app.db from before the hdd upgrade.
I have a load of app.db's from various PS4's from various stages so it might take me a wile to search through them but I'll have a look for you.