Front-page Updated

[Update] RetroArch servers and repositories have been hacked

unnamed.png

Just a few hours ago, RetroArch/Libretro's servers and main GitHub repositories have been targeted by a yet-unknown attacker.
The attack begun with the buildbot server being crippled, which means any subsequent automatic buildbot builds, and netplay won't be available until a new server is setup for this very purpose.




After that, and a few moments later, the hacker moved on to attack Libretro's repositories at GitHub.
This attack removed the entirety of codes for certain cores, like Mame, Mame 2003, DosBox and many others, and only left a dummy ReadMe with a vague description of the core.




GitHub hasn't given any reply regarding what could be done in regards to the hacking to the GitHub repositories, but we'll keep updating this post as things go along.
The full overview of the attack and what was compromised on Libretro's side can be seen on their main Libretro.com page.

Hacker vandalised our buildbot and Github organization said:
Approximately 5 hours ago, we were the target of a premeditated cybercrime attack on our key infrastructure.

The hacker did the following damage:

  • He accessed our buildbot server and crippled the nightly/stable buildbot services, and the netplay lobby service. Right now, the Core Updater and Netplay Lobbies won’t work. The websites for these have also been rendered inaccessible for the moment
  • He gained access to our Libretro organization on Github impersonating a very trusted member of the team and force-pushed a blank initial commit to a fair percentage of our repositories, effectively wiping them. He managed to do damage to 3 out of 9 pages of repositories. RetroArch and everything preceding it on page 3 has been left intact before his access got curtailed.
We are still awaiting any sort of response or support from Github. We hope they will be able to help us restore some of these vandalised Github repos to their proper state, and also to help us narrow down the attacker’s identity.

We wanted to clear up some confusion that may have arisen in the wake of this news breaking:

  • No cores or RetroArch installations should be considered compromised. The attacker simply wiped our buildbot server clean, there is nothing being distributed that could be considered malicious to your system. Nothing has happened here and there is no need for any concern.
  • For the current time being, the Core Installer is non-functional until further notice. The same goes for ‘Update Assets’, ‘Update Overlays’, ‘Update Shaders’, and all the other online services that RetroArch users normally have access to (such as the netplay lobby services).
The IP he was using while doing this was ‘54.167.104.253’, which seems to lead back to AWS.

We’re still assessing the situation but moving forward, we think that it’s probably best not to go forward with the buildbot server that was compromised earlier today. We had some long-term migration plans for a move to a new server, but this was always pushed back because we felt that we weren’t ready migration-wise. It might indeed be the case this is the catalyst for just starting all from scratch with a new server instead of trying to migrate the old one over. This would mean that the more commonplace builds for Linux/Windows/Android would be immediately available, but all the specialized systems like consoles, old MSVC builds and whatnot would have to wait for later until we have adapted this properly to the new system.

Lack of automated backups
This brings us onto another key issue – the lack of backups. We last performed a backup of our buildbot server about a couple of months ago. The truth is that while we pay a hefty amount for the servers on a monthly basis already, there is simply not enough money to pile on automated backups as well. We could really use your support on Patreon to help lighten our financial burden here, especially since this now-pretty-much-mandatory server switch will likely cost us an insubstantial amount of money upfront while we keep the current server running for a month longer.

How will we restore things
So, how are we going to restore things? We hope that Github will be able to restore the affected repositories. If they are unable to do so, we could rely on the goodwill of users to source us with git repositories with the full history intact.

As for the buildbot? No idea to be quite frank. If we make the switch to the new server, you’ll get Android/Windows/Linux up and running early again but all other platforms will have to be added as we go along.

It’s a shame what is happening to the emulation and homebrew community. When it isn’t developers leaving for greener pastures deciding it’s no longer worth it, prestigious developers like byuu are being forced to early retirement because of unsavory online gang-stalkers. In our situation, we can’t rule out the possibility that some of these attacks come from some of the same usual suspects (it isn’t the first time we’ve seen them abuse AWS for some of these attacks, we encountered them a year ago earlier targeting our lobby services). Whatever their aim may be, while they will not deter our will to continue working on this project, they have definitely increased our maintenance and cost burden for the time being. And for this we ask for your understanding and support as we attempt to come up with a plan to address these problems moving forward. Supporting us through Patreon is a great way of helping out, especially if we can reach the $1300 goal which means we can spend a bit more each month to make sure our stuff is properly backed up.

As if the complications with Android’s new store policies that requires us to coordinate with new contributors to come up with a workable solution was not enough of a headache, this comes along. With your help and support, we will overcome this and come out stronger than before.

Regarding the Android / Core Installer situation
While we’re on this subject briefly, while it’s off-topic, we felt the need to address this real quick. We will likely be making a version of RetroArch Android that is neutered ONLY for Google Play. It will mean that the Core Installer will not be available for this, and cores will come packaged in additional APKs that can be installed. Apparently there is a 50-core extra APK limit on this until it starts requiring a version of Android over version 8.0. So while trying not to artificially bump the Android OS system requirements, we’re deciding on a 50 core-APK limit for now. Hopefully we can fit nearly most of the cores within such narrow constraints.

On our download site (and on F-Droid), we will have a RetroArch Android version that will work as before – with the Core Installer feature completely left intact. We feel this is a much superior version to what will be available on the Play Store, but unfortunately Google will force our hand here.
Click to expand...

UPDATE:
GitHub has replied back to Libretro, with the sad news that they don't have a way to restore or have a backup of the repositories Libretro had before the hack. It seems the restoration of the repositories will have to be done (alongside the help of other users) through full commit pushes that hold the entire history of the repositories:


UPDATE #2:
Libretro has restored the vast majority of the repositories back to shape. The only downside has been the loss of recent Pull Request:



UPDATE #3:
The buildbot is now back online, though not at the same extend as it was before the attack
http://buildbot.libretro.com/

Right now some builds for multiple platforms have been compiled, but some of those builds might be prone to failure.
I recommend trying them out only if you have your previous build backed up in case some compilation went bad.




:arrow: Source
 
Last edited by ShadowOne333,
  • Like
Reactions: Julie_Pilgrim, SuzieJoeBob, Red771 and 16 others
Click to expand...
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
Joom said:
You're not wrong. Backing up code is actually easier. Redundancy and retention are also trivial. It's not like making a Git archive is hard, either. It's an available feature on the main page of every repo. Usually everyone involved also has the repo available locally.

@m4xw, maybe you guys could look into renting a Kimsufi box. They're an OVH reseller with very fair prices for the specs offered. If all you really need is storage, they're perfect. I've ran a seedbox with them for several years with no issues.
Click to expand...
Git isnt the issue, the non virtualized server is, if you wan't a proper backup its a full 2TB disk image.
I only took over this server, so for me its just tons of legacy scripts and we are in the process of creating a reproduceable pipeline by virtualizing and containerizing the buildbot anyway.
Also we will move to Hetzner, currently already using their storage solution, but as you can think a 2TB backup is easily days of downtime, the current bot just doesnt allow it without missing out some things (and the some part will by universal law, the most crucial thing at the time)
 
Last edited by m4xw,
  • Like
Reactions: ChronoTrig, ShadowOne333, TheCasualties and 1 other person
Joom

Joom

 ❤❤❤
Member
Level 16
Joined
Jan 8, 2016
Messages
6,067
Trophies
1
Location
US
Website
mogbox.net
XP
6,076
Country
United States
m4xw said:
Git isnt the issue, the non virtualized server is, if you wan't a proper backup its a full 2TB disk image.
Click to expand...
I know Git isn't the issue here. This is also why I recommended a Kimsufi box, because I kind of assumed everything en masse was a decently sized collection. I rent a 2 TB VDS for $25 a month. If you need more, the scale in price for storage is very affordable. But if you already have a solution, don't worry about it.
 
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
Joom said:
I know Git isn't the issue here. This is also why I recommended a Kimsufi box, because I kind of assumed everything en masse was a decently sized collection. I rent a 2 TB VDS for $25 a month. If you need more, the scale in price for storage is very affordable. But if you already have a solution, don't worry about it.
Click to expand...
We get 5TB for that price.
 
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
The backup storage funds are really less important than actually the buffer for the transition to the new infra due to setup costs (200+ bucks) as well as having both systems run in parallel for a transition period (thus doubling our server costs for that period) ,so we can ensure undisrupted service and still quick access to the files instead of a month of downtime altogether (we planned this for months already but then covid happened and servers werent in restock).
I think the blog post puts the wrong emphasis out. The current bot would be useless with automatic backups, it produces more data than we could even save away without basically DOS'ing ourselves the way its currently structured and this is a way more urgent matter (on top of breaking the consistency would throw you back to the same state like only doing a backup every half a year because so much volatile data and nobody knows which glue scripts got added over the years).
I was literally prepared for a fallout anytime tho since the HDD's run since 8years or so and are not even secured by a raid (which is ridiculous at the cost of the 100tb server, I literally get 2 servers with nearly simliar/better spec for that price in a different datacenter).

At least I can tell you we will put the money to good use to fix that legacy mess, but the 100$ extra allocated for a backup budget wont be much of a concern till all that is resolved (we plan on handling 4-8TB on data)

Suddenly 1k gone :P, btw little sidenote I was already planning to pay the setup costs of my private money, but seems we might just be able to cover the costs!
So your help is def. appreciated.
 
Last edited by m4xw,
  • Like
Reactions: ChronoTrig, ShadowOne333, TheCasualties and 1 other person
Tatsuna

Tatsuna

Well-Known Member
Newcomer
Level 5
Joined
Apr 8, 2020
Messages
64
Trophies
0
Age
26
XP
707
Country
United States
Silent_Gunner said:
Is it just me, or does it seem like hackers have been more active than usual this year? Because it seems like a lot of high profile hacks/leaks/and breaches have occurred regarding stuff that's kind of trivial, all things considered.

Like, I remember when hacking groups would target corporations because said corporations were corrupt and/or in the moral wrong, but now, Nintendo's gotten three/four different leaks this year, and now we have RetroArch getting hacked.
Click to expand...
Why, is RetroArch bad? I'm wondering out of curiosity -- I don't know anything about that.
 
Joom

Joom

 ❤❤❤
Member
Level 16
Joined
Jan 8, 2016
Messages
6,067
Trophies
1
Location
US
Website
mogbox.net
XP
6,076
Country
United States
m4xw said:
The current bot would be useless with automatic backups, it produces more data than we could even save away without basically DOS'ing ourselves the way its currently structured
Click to expand...
So why not split the load between it and rsync, and decrease the bot's workload as to not cause unnecessary overhead? Surely someone has the resources to solely dedicate to the bot, and another could use rsync to only sync on a set time interval. I feel like resource management itself is the real problem here. Then again, I only have the mind frame for projects that are considerably smaller than RetroArch. I more have a proclivity for data retention itself, too, so I probably don't have a dog in this race due to sheer ignorance.
Tatsuna said:
Why, is RetroArch bad? I'm wondering out of curiosity -- I don't know anything about that.
Click to expand...
It itself isn't. I guess uppity fan boys are gonna blame them for swagger jacking (which is dumb because emulator core devs support front-end development since it makes it easier on their users), but from what I can tell, the problem lies within the belief system of the lead dev who I guess flaunts a sense of arrogance.
 
Last edited by Joom,
  • Like
Reactions: Silent_Gunner
D

Deleted member 532471

Well-Known Member
Member
Level 4
Joined
Jul 6, 2020
Messages
233
Trophies
0
Age
36
XP
532
Country
United States
Is it that surprising
Some time ago I set up a password protected netplay session to play with a friend and someone hacked into our room almost immediately.
They made it in faster than my friend, who I was in a call with
Retroarch is not safe
 
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
Joom said:
So why not split the load between it and rsync, and decrease the bot's workload as to not cause unnecessary overhead? Surely someone has the resources to solely dedicate to the bot, and another could use rsync to only sync on a set time interval. I feel like resource management itself is the real problem here. Then again, I only have the mind frame for projects that are considerably smaller than RetroArch. I more have a proclivity for data retention itself, so I probably don't have a dog in this race due to sheer ignorance.
Click to expand...
This would add more costs, all this will come free of charge with the new infra.
Its a legacy mess
 
  • Like
Reactions: ChronoTrig and Joom
Joom

Joom

 ❤❤❤
Member
Level 16
Joined
Jan 8, 2016
Messages
6,067
Trophies
1
Location
US
Website
mogbox.net
XP
6,076
Country
United States
zupi said:
Is it that surprising
Some time ago I set up a password protected netplay session to play with a friend and someone hacked into our room almost immediately.
They made it in faster than my friend, who I was in a call with
Retroarch is not safe
Click to expand...
Now this intrigues me. Does RetroArch not use encryption, thus making MitM attacks possible? It's also quite possible that you had a keylogger at the time without knowing. This said, what does one benefit from joining a private netplay lobby?
 
Last edited by Joom,
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
Joom said:
Now this intrigues me. Does RetroArch not use encryption, thus making MitM attacks possible?
Click to expand...
Lobby is unsafe by design, a malicious actor can pause lock any game, this requires chcanges in the code and lobby system.
Mitm isnt a concern here, dunno about password protection tho.
Certainly not enterprise grade ;)
 
  • Like
Reactions: ChronoTrig
Joom

Joom

 ❤❤❤
Member
Level 16
Joined
Jan 8, 2016
Messages
6,067
Trophies
1
Location
US
Website
mogbox.net
XP
6,076
Country
United States
m4xw said:
Lobby is unsafe by design, a malicious actor can pause lock any game, this requires chcanges in the code and lobby system.
Mitm isnt a concern here, dunno about password protection tho.
Certainly not enterprise grade ;)
Click to expand...
Well, it's understandable. I edited my post to reflect a stupid epiphany; what could one benefit from circumventing their way into a protected lobby? I have to imagine that it doesn't offer a wide open attack vector, so it's more trolly than anything.
 
m4xw

m4xw

Ancient Deity
Developer
Level 17
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,958
Country
Germany
Joom said:
Well, it's understandable. I edited my post to reflect a stupid epiphany; what could one benefit from circumventing their way into a protected lobby? I have to imagine that it doesn't offer a wide open attack vector, so it's more trolly than anything.
Click to expand...
Pure troll reason
No RCE afaik, but maybe in one ver over the years
 

Site & Scene News

Go to forum More news

Popular threads in this forum

Zippo says Sonic heroes remake might happed

Hacking Gaming Homebrew Homebrew app  SUYU Discord get deleted

Rumor  Switch 2 Official Name Could Have Been Revealed by Youtube

Hero Shooter GUNDAM EVOLUTION revived in private server project following End of Service

"Apple has lifted its restriction on retro game emulators on the iOS App Store"

Mario Maker 1 Deemed 100% after Final Level revealed to be hacked

Pokemon Fan Game Community Site, Relic Castle, has been DMCA'd

Mig switch flashcard clones are coming

General chit-chat
Help Users
  • No one is chatting at the moment.
    Xdqwerty @ Xdqwerty: good night