All the information found below is outdated and has links to old/not working files etc.
The project and it's usage is now on my github here: https://github.com/dojafoja/Restricted-Webkit-Bug-Finder
ORIGINAL THREAD FROM 2015, THIS IS OBSOLETE NOW, VISIT MY GITHUB LINK ABOVE :
We all know that MarioNumber1 and NWPlayer123 are too busy with the kernel exploit to help do anything else. We also know that our best way in to the WiiU is the browser and that we need to find a new browser bug that will lead to an exploit.. How can the community help? We need to collectively find browser bugs that crash the browser. So I decided to pitch in and I wrote this handy little gui application in python. I pulled a svn log from WebKit repo, manually trimmed the log to only include commits entered after October 16, 2012 and ran it through my log parser. This produced 40,811 individual commit entries. My script attempts to extract all the urls for the bug. Unfortunately it isn't perfect and doesn't handle bug urls with leading or trailing characters or commits with multilple bug urls. Then the script scans every single url and if the bug is restricted it is marked as such and all of this is stored neatly into a sqlite3 database. This found 427 restricted bugs and took several hours to scrape every url. My gui presents this to the user. If the log parser were fixed or rewritten to handle all urls properly there may be more potential restricted bugs.
Things you should know:
1. I am not a great programmer. I'm more like a hobbyist. I taught myself everything I know and have very little time to do this stuff because I have a job, wife, 2 kids, other obligations, etc.
2. I was planning on this being python 2.7 and python 3 compatible but I have not fully implemented it yet, only partially.
3. I DO realize that obtaining a svn log in xml format would be much easier to parse but thats not how I did it. It must be a simple txt output.
4. I do NOT have much time to help answer too many questions.
5.I just noticed that I didn't put a scroll bar on the left box in the results tab, I will fix it later. Use your mouse wheel or keyboard arrow.
6.It does not resize properly because I manually set the width and height of the text and list boxes and I also didn't set the column and row weight properties.
7. The database file MUST be named 'commits.db' because it is hardcoded into the script.
8.The database contains ALL commit logs, not just ones with restricted bugs. Restricted ones are marked 1 in the 'restricted' field. Performance in the Results view could be dramatically increased if the db were stripped to erase all entries that are not restricted
9.The program will crash if you click the Parser log button and and invalid or blank path is given. This is easy to fix.
10. It is sometimes not the fastest program in the world. I have some redundant database queries all over the place, duplicate code in spots and is very procedural because it was thrown together very quickly
11. There is no context menu on right click. To copy/paste just use the keyboard Ctrl-C/Ctrl-V
12. It is buggy and not finished. Please feel free to make it better if it bothers you.
12. I AM NOT A GREAT PROGRAMMER
The community as a whole can pledge to scour the output and test all bugs which contain layout tests.
Do not publicly talk about which bugs crash the browser, we don't want to give Nintendo a nice list of all bugs they need to patch.
INSTRUCTIONS:
Requires Python 2.7, the only external libs you might need are beautifulsoup4 and possibly the tkinter packages if they arent included with your python dist. This also uses sqlite3 but that is usually shipped with python. Everything else should just be shipped with python. If you are going to use the Windows exe file then you will not need to satisfy the above mentioned dependencies.
You MUST have a local copy of the webkit repo so the program can host the proper files.
get it here: http://www.webkit.org/building/checkout.html
DOWNLOAD:
There are two versions available for download, a regular python script as well as a windows executable.
1. You can download the regular python script here: http://www.mediafire.com/download/c1mvzc0fsoi55cf/wbf_v0.4.rar
2. You can download the Windows exe file from here: http://www.mediafire.com/download/4vox8l896vo145u/wbf_v0.3_exe.rar
The Windows exe was created with cx_freeze for python. You will not need to satisfy any dependencies. Just extract the rar and run test_parser.exe This was tested on Windows7 with older hardware(1Ghz Pentum4) and performance was Ok.
OPTION A:
1. Download and extract the rar file found above.
2. To simply use the output I have provided, run the test_parser.py script or test_parser.exe, switch to the Results tab and click on Refresh list.
2. ALL commits with bugs marked as restricted in the db are listed on the left. Click on commit title and the entire commit entry will be displayed on the right.
3. Some of the commits will have layout tests listed. Copy the location of the layout test html file found in the results view.
4. Switch to the hosting tab, click Get my IP button or manually provide your local IP address. Next, click Browse and provide the location to your LayoutTests directory. Now paste the html file location into the bottom entry and then click the host file button.
5. Point the console browser to your_local_ip:8000, no need to provide a file name to the browser. The file is being hosted as index.html so the browser will automatically load it. If no crash or buggy behaviour occurs, find another layout test, host it and repoint the browser to local_ip:8000. I recommend creating a bookmark and select it over and over each time you host a new file. It will load the new file being hosted each time you do this!
OPTION B:
1. If you would like to parse your own log and generate your own database instead of using the one provided,
be sure that you have svn and obtain a webkit svn log as a plain txt file.
2. Download and extract the rar file found above.
3. Run the script test_parser.py or test_parser.exe
4. To parse your own log and generate a commits database, click the Browse button or provide a path to the svn log txt file. Click Parse log. It will parse the log into a database and automatically stop to only include entries prior to 10/16/2012. I have provide a svn_log.txt that is trimmed. This log was pulled on 1/07/2015.
5. To scan all urls, attempt to find all restricted bugs and mark them as restricted in the database, click the Scan for Restricted bugs button. This will take a VERY Long time!! It took several hours to complete on my machine(6 or more).
6. Optionally, you can strip the database of all entries that do not contain restricted bugs by clicking the Strip database button. This will dramatically increase performance in the Results view window.
7. To view the output of the commits.db database click the Results tab at the top of the program, click Refresh list and ALL commits with bugs marked as restricted in the db are listed on the left. Click on commit title and the entire commit entry will be displayed on the right.
8. Some of the commits will have layout tests listed. Copy the location of the layout test html file found in the results view.
9. Switch to the hosting tab, click Get my IP button or manually provide your local IP address. Next, click Browse and provide the location to your LayoutTests directory. Now paste the html file location into the bottom entry and then click the host file button.
10. Point the console browser to your_local_ip:8000, no need to provide a file name to the browser. The file is being hosted as index.html so the browser will automatically load it. If no crash or buggy behaviour occurs, find another layout test, host it and repoint the browser to local_ip:8000. I recommend creating a bookmark and select it over and over each time you host a new file. It will load the new file being hosted each time you do this!
Tested on Ubuntu 14.04 and Windows7
The generated db file is a standard sqlite3 db and can be viewed with any sqlite3 compatible viewer as well. The db has field named 'restricted' and restricted bugs will have a value of 1.
I encourage anybody with more time and knowledge to fix the log parser to handle some urls better. Also, if you can make the code more efficient please do so. I would love to learn from other peoples changes to the code. Just please be nice if you think my code sucks, when I don't quite know how to do stuff, I just dirty hack the hell out of it until it works :-P
The project and it's usage is now on my github here: https://github.com/dojafoja/Restricted-Webkit-Bug-Finder
ORIGINAL THREAD FROM 2015, THIS IS OBSOLETE NOW, VISIT MY GITHUB LINK ABOVE :
We all know that MarioNumber1 and NWPlayer123 are too busy with the kernel exploit to help do anything else. We also know that our best way in to the WiiU is the browser and that we need to find a new browser bug that will lead to an exploit.. How can the community help? We need to collectively find browser bugs that crash the browser. So I decided to pitch in and I wrote this handy little gui application in python. I pulled a svn log from WebKit repo, manually trimmed the log to only include commits entered after October 16, 2012 and ran it through my log parser. This produced 40,811 individual commit entries. My script attempts to extract all the urls for the bug. Unfortunately it isn't perfect and doesn't handle bug urls with leading or trailing characters or commits with multilple bug urls. Then the script scans every single url and if the bug is restricted it is marked as such and all of this is stored neatly into a sqlite3 database. This found 427 restricted bugs and took several hours to scrape every url. My gui presents this to the user. If the log parser were fixed or rewritten to handle all urls properly there may be more potential restricted bugs.
Things you should know:
1. I am not a great programmer. I'm more like a hobbyist. I taught myself everything I know and have very little time to do this stuff because I have a job, wife, 2 kids, other obligations, etc.
2. I was planning on this being python 2.7 and python 3 compatible but I have not fully implemented it yet, only partially.
3. I DO realize that obtaining a svn log in xml format would be much easier to parse but thats not how I did it. It must be a simple txt output.
4. I do NOT have much time to help answer too many questions.
5.
6.
7. The database file MUST be named 'commits.db' because it is hardcoded into the script.
8.
9.
10. It is sometimes not the fastest program in the world. I have some redundant database queries all over the place, duplicate code in spots and is very procedural because it was thrown together very quickly
11. There is no context menu on right click. To copy/paste just use the keyboard Ctrl-C/Ctrl-V
12. It is buggy and not finished. Please feel free to make it better if it bothers you.
12. I AM NOT A GREAT PROGRAMMER
The community as a whole can pledge to scour the output and test all bugs which contain layout tests.
Do not publicly talk about which bugs crash the browser, we don't want to give Nintendo a nice list of all bugs they need to patch.
INSTRUCTIONS:
Requires Python 2.7, the only external libs you might need are beautifulsoup4 and possibly the tkinter packages if they arent included with your python dist. This also uses sqlite3 but that is usually shipped with python. Everything else should just be shipped with python. If you are going to use the Windows exe file then you will not need to satisfy the above mentioned dependencies.
You MUST have a local copy of the webkit repo so the program can host the proper files.
get it here: http://www.webkit.org/building/checkout.html
DOWNLOAD:
There are two versions available for download, a regular python script as well as a windows executable.
1. You can download the regular python script here: http://www.mediafire.com/download/c1mvzc0fsoi55cf/wbf_v0.4.rar
2. You can download the Windows exe file from here: http://www.mediafire.com/download/4vox8l896vo145u/wbf_v0.3_exe.rar
The Windows exe was created with cx_freeze for python. You will not need to satisfy any dependencies. Just extract the rar and run test_parser.exe This was tested on Windows7 with older hardware(1Ghz Pentum4) and performance was Ok.
OPTION A:
1. Download and extract the rar file found above.
2. To simply use the output I have provided, run the test_parser.py script or test_parser.exe, switch to the Results tab and click on Refresh list.
2. ALL commits with bugs marked as restricted in the db are listed on the left. Click on commit title and the entire commit entry will be displayed on the right.
3. Some of the commits will have layout tests listed. Copy the location of the layout test html file found in the results view.
4. Switch to the hosting tab, click Get my IP button or manually provide your local IP address. Next, click Browse and provide the location to your LayoutTests directory. Now paste the html file location into the bottom entry and then click the host file button.
5. Point the console browser to your_local_ip:8000, no need to provide a file name to the browser. The file is being hosted as index.html so the browser will automatically load it. If no crash or buggy behaviour occurs, find another layout test, host it and repoint the browser to local_ip:8000. I recommend creating a bookmark and select it over and over each time you host a new file. It will load the new file being hosted each time you do this!
OPTION B:
1. If you would like to parse your own log and generate your own database instead of using the one provided,
be sure that you have svn and obtain a webkit svn log as a plain txt file.
2. Download and extract the rar file found above.
3. Run the script test_parser.py or test_parser.exe
4. To parse your own log and generate a commits database, click the Browse button or provide a path to the svn log txt file. Click Parse log. It will parse the log into a database and automatically stop to only include entries prior to 10/16/2012. I have provide a svn_log.txt that is trimmed. This log was pulled on 1/07/2015.
5. To scan all urls, attempt to find all restricted bugs and mark them as restricted in the database, click the Scan for Restricted bugs button. This will take a VERY Long time!! It took several hours to complete on my machine(6 or more).
6. Optionally, you can strip the database of all entries that do not contain restricted bugs by clicking the Strip database button. This will dramatically increase performance in the Results view window.
7. To view the output of the commits.db database click the Results tab at the top of the program, click Refresh list and ALL commits with bugs marked as restricted in the db are listed on the left. Click on commit title and the entire commit entry will be displayed on the right.
8. Some of the commits will have layout tests listed. Copy the location of the layout test html file found in the results view.
9. Switch to the hosting tab, click Get my IP button or manually provide your local IP address. Next, click Browse and provide the location to your LayoutTests directory. Now paste the html file location into the bottom entry and then click the host file button.
10. Point the console browser to your_local_ip:8000, no need to provide a file name to the browser. The file is being hosted as index.html so the browser will automatically load it. If no crash or buggy behaviour occurs, find another layout test, host it and repoint the browser to local_ip:8000. I recommend creating a bookmark and select it over and over each time you host a new file. It will load the new file being hosted each time you do this!
Tested on Ubuntu 14.04 and Windows7
The generated db file is a standard sqlite3 db and can be viewed with any sqlite3 compatible viewer as well. The db has field named 'restricted' and restricted bugs will have a value of 1.
I encourage anybody with more time and knowledge to fix the log parser to handle some urls better. Also, if you can make the code more efficient please do so. I would love to learn from other peoples changes to the code. Just please be nice if you think my code sucks, when I don't quite know how to do stuff, I just dirty hack the hell out of it until it works :-P
Attachments
Last edited by dojafoja,