RELEASE Mario Maker level browser for tinfoil

Discussion in 'Switch - Emulation, Homebrew & Software Projects' started by quicksilver501, Oct 13, 2019.

  1. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    Hi all, some time ago I wanted to download courses from tinfoi.io, but kept downloading the same levels because there is no easy way to distinguish downloaded courses from new ones. So I made this little python gui to keep track of downloaded courses and batch download them, showing a preview of the level.

    Just run "gui. py" and follow the instructions in the readme.


    https://github.com/mercury501/ductile_mm_lv_browser/releases



    Please keep in mind that I made this tool for myself primarily, then thought somebody else could use it. There may be bugs and shortcomings, maybe they'll be fixed one day, maybe they won't. Open an issue on github or, if you can, fork it.
     
    Last edited by quicksilver501, Oct 14, 2019
  2. blawar

    blawar Developer

    pip Developer
    11
    Nov 21, 2016
    United States
    While I appreciate the effort, you should not be scraping the site, let alone publishing a public scraper. I will have to kill those endpoints now.
     
  3. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    Well, if it bothers you I won't go publicly against your will. I'll take it down anyway.
     
  4. hippy dave

    hippy dave BBMB

    Member
    15
    Apr 30, 2012
    United Kingdom
    So it's cool for you to build this database from other people's work without their consent, but not for others to access it?
     
    Techjunky90 likes this.
  5. Mthodmn101

    Mthodmn101 GBAtemp Advanced Fan

    Member
    6
    Jan 31, 2008
    United States
    dont people upload courses in mario maker so others will play them?
     
  6. hippy dave

    hippy dave BBMB

    Member
    15
    Apr 30, 2012
    United Kingdom
    Yes, not sure how that affects my point.
    E: more specifically, I'm not complaining about him building his database from other people's work, I'm saying that leaves him in no position to be prissy about how people access his database.
     
    Last edited by hippy dave, Oct 13, 2019
  7. OrGoN3

    OrGoN3 GBAtemp Addict

    Member
    9
    Apr 23, 2007
    United States
    The uploader isn't necessarily the IP owner. Seeing as you need to hack your console to download your course, I'd assume that <10% are from the content creator, if any.
     
  8. blawar

    blawar Developer

    pip Developer
    11
    Nov 21, 2016
    United States
    Just scrape Cloudflare. My main rub is that you scraped the main site which is already under high load. You also do not need to scrape the html, that just drives up load. there is an api endpoint for that info: https://tinfoil.media/Api/mml

    replace all of the tinfoil.io references to tinfoil.media which is the caching server.

    — Posts automatically merged - Please don't double post! —

    The data is public for all to access it. The problem is running a scraper on the main server (not the caching server).

    edit: that website serves ~3 million requests per day, some of them scrapers which makes no sense since I publish most of the data on github. If users really want the mario marker levels, i can publish that data to github as well, someone just needs to ask :)

    edit2: those zip files are not stored on the disk, they are created on the fly because that url was only meant to be hit by cloudflare to refresh cache. Scraping that endpoint alone drives up load significantly.
     
    Last edited by blawar, Oct 13, 2019
  9. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    Well,
    Well, in my research I never came across this tinfoil.media caching server, or I'd have used that.

    As for the level data, I'd be glad if you posted it on github as well, maybe I can change the program to use that data instead of scraping (if anybody cares).

    I indeed asked for a way to store downloaded courses, but not for the raw data.

    If there's demand for something like this i'd be glad to whip up another quick gui (maybe using tinfoil.media/Api/mml or the github), but if there's not I'll pass momentarily. I just wanted a feature, and implemented it myself.

    So, to be clear, as a temporary solution I can reup the repo replacing tinfoil.io with .media?
     
    Last edited by quicksilver501, Oct 13, 2019
  10. blawar

    blawar Developer

    pip Developer
    11
    Nov 21, 2016
    United States
    yeah just replace all tinfoil.io references with tinfoil.media and it should be good enough for now, though the web pages wont be cached. Ideally you want to update it to not scrape any html, and use that mml endpoint I gave you above.

    if you go to the website and hover over the asset links, you will see they all link to tinfoil.media not tinfoil.io
     
  11. Datahead

    Datahead Newbie

    Newcomer
    1
    Oct 12, 2019
    United States
    Sounds like a great feature to me!
     
    Andalitez likes this.
  12. Sonikkua

    Sonikkua GBAtemp Regular

    Member
    5
    Mar 26, 2011
    United States
    I just wish people would upload weekly top courses as a save file like they used to :-/

    would love getting to my day off every week and grabbing a save with that weeks top levels in all categories. Was damned convenient
     
  13. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    Very well then, in a few hours I will try it with tinfoil.media,and in a few days I'll implement it using the api. Stay tuned!
     
    Datahead likes this.
  14. hippy dave

    hippy dave BBMB

    Member
    15
    Apr 30, 2012
    United Kingdom
    Glad to see a solution reached :)
     
    lordelan and ELY_M like this.
  15. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    Update:

    I'm now working on using the api directly, and a solution to cache just the right amount of icons or displaying them directly from the web, thus removing the need to download ALL of them at the start.
     
  16. quicksilver501
    OP

    quicksilver501 Member

    Newcomer
    1
    Jul 1, 2019
    Italy
    v0.1 Now it works using blawar's api link, downloads icons as needed, while browsing, and deletes them if you download/delete the level from list. Feedback appreciated! This is my first "Release" so be gentle.


    (Don't know if I can "UP" the thread, but felt a pre-release warranted one. Sorry if not allowed!)
     
    Datahead and hippy dave like this.
Quick Reply
Draft saved Draft deleted
Loading...