[Update] RetroArch servers and repositories have been hacked

unnamed.png

Just a few hours ago, RetroArch/Libretro's servers and main GitHub repositories have been targeted by a yet-unknown attacker.
The attack begun with the buildbot server being crippled, which means any subsequent automatic buildbot builds, and netplay won't be available until a new server is setup for this very purpose.




After that, and a few moments later, the hacker moved on to attack Libretro's repositories at GitHub.
This attack removed the entirety of codes for certain cores, like Mame, Mame 2003, DosBox and many others, and only left a dummy ReadMe with a vague description of the core.




GitHub hasn't given any reply regarding what could be done in regards to the hacking to the GitHub repositories, but we'll keep updating this post as things go along.
The full overview of the attack and what was compromised on Libretro's side can be seen on their main Libretro.com page.

Hacker vandalised our buildbot and Github organization said:
Approximately 5 hours ago, we were the target of a premeditated cybercrime attack on our key infrastructure.

The hacker did the following damage:

  • He accessed our buildbot server and crippled the nightly/stable buildbot services, and the netplay lobby service. Right now, the Core Updater and Netplay Lobbies won’t work. The websites for these have also been rendered inaccessible for the moment
  • He gained access to our Libretro organization on Github impersonating a very trusted member of the team and force-pushed a blank initial commit to a fair percentage of our repositories, effectively wiping them. He managed to do damage to 3 out of 9 pages of repositories. RetroArch and everything preceding it on page 3 has been left intact before his access got curtailed.
We are still awaiting any sort of response or support from Github. We hope they will be able to help us restore some of these vandalised Github repos to their proper state, and also to help us narrow down the attacker’s identity.

We wanted to clear up some confusion that may have arisen in the wake of this news breaking:

  • No cores or RetroArch installations should be considered compromised. The attacker simply wiped our buildbot server clean, there is nothing being distributed that could be considered malicious to your system. Nothing has happened here and there is no need for any concern.
  • For the current time being, the Core Installer is non-functional until further notice. The same goes for ‘Update Assets’, ‘Update Overlays’, ‘Update Shaders’, and all the other online services that RetroArch users normally have access to (such as the netplay lobby services).
The IP he was using while doing this was ‘54.167.104.253’, which seems to lead back to AWS.

We’re still assessing the situation but moving forward, we think that it’s probably best not to go forward with the buildbot server that was compromised earlier today. We had some long-term migration plans for a move to a new server, but this was always pushed back because we felt that we weren’t ready migration-wise. It might indeed be the case this is the catalyst for just starting all from scratch with a new server instead of trying to migrate the old one over. This would mean that the more commonplace builds for Linux/Windows/Android would be immediately available, but all the specialized systems like consoles, old MSVC builds and whatnot would have to wait for later until we have adapted this properly to the new system.

Lack of automated backups
This brings us onto another key issue – the lack of backups. We last performed a backup of our buildbot server about a couple of months ago. The truth is that while we pay a hefty amount for the servers on a monthly basis already, there is simply not enough money to pile on automated backups as well. We could really use your support on Patreon to help lighten our financial burden here, especially since this now-pretty-much-mandatory server switch will likely cost us an insubstantial amount of money upfront while we keep the current server running for a month longer.

How will we restore things
So, how are we going to restore things? We hope that Github will be able to restore the affected repositories. If they are unable to do so, we could rely on the goodwill of users to source us with git repositories with the full history intact.

As for the buildbot? No idea to be quite frank. If we make the switch to the new server, you’ll get Android/Windows/Linux up and running early again but all other platforms will have to be added as we go along.

It’s a shame what is happening to the emulation and homebrew community. When it isn’t developers leaving for greener pastures deciding it’s no longer worth it, prestigious developers like byuu are being forced to early retirement because of unsavory online gang-stalkers. In our situation, we can’t rule out the possibility that some of these attacks come from some of the same usual suspects (it isn’t the first time we’ve seen them abuse AWS for some of these attacks, we encountered them a year ago earlier targeting our lobby services). Whatever their aim may be, while they will not deter our will to continue working on this project, they have definitely increased our maintenance and cost burden for the time being. And for this we ask for your understanding and support as we attempt to come up with a plan to address these problems moving forward. Supporting us through Patreon is a great way of helping out, especially if we can reach the $1300 goal which means we can spend a bit more each month to make sure our stuff is properly backed up.

As if the complications with Android’s new store policies that requires us to coordinate with new contributors to come up with a workable solution was not enough of a headache, this comes along. With your help and support, we will overcome this and come out stronger than before.

Regarding the Android / Core Installer situation
While we’re on this subject briefly, while it’s off-topic, we felt the need to address this real quick. We will likely be making a version of RetroArch Android that is neutered ONLY for Google Play. It will mean that the Core Installer will not be available for this, and cores will come packaged in additional APKs that can be installed. Apparently there is a 50-core extra APK limit on this until it starts requiring a version of Android over version 8.0. So while trying not to artificially bump the Android OS system requirements, we’re deciding on a 50 core-APK limit for now. Hopefully we can fit nearly most of the cores within such narrow constraints.

On our download site (and on F-Droid), we will have a RetroArch Android version that will work as before – with the Core Installer feature completely left intact. We feel this is a much superior version to what will be available on the Play Store, but unfortunately Google will force our hand here.

UPDATE:
GitHub has replied back to Libretro, with the sad news that they don't have a way to restore or have a backup of the repositories Libretro had before the hack. It seems the restoration of the repositories will have to be done (alongside the help of other users) through full commit pushes that hold the entire history of the repositories:


UPDATE #2:
Libretro has restored the vast majority of the repositories back to shape. The only downside has been the loss of recent Pull Request:



UPDATE #3:
The buildbot is now back online, though not at the same extend as it was before the attack
http://buildbot.libretro.com/

Right now some builds for multiple platforms have been compiled, but some of those builds might be prone to failure.
I recommend trying them out only if you have your previous build backed up in case some compilation went bad.




:arrow: Source
 
Last edited by ShadowOne333,

Obveron

Well-Known Member
Member
Joined
Jul 5, 2010
Messages
503
Trophies
1
XP
1,397
Country
Canada
2TB on current, 3TB projected on new infra.
Ok but thats a full server image right? If you just wanted to backup the actual database, like source code of all branches of all platforms, plus documentation, etc surely that would compress down to far less? This is the stuff you want to backup in real time, or atleast daily.

Things like a server core image if you exclude the database, shouldn't change as often and in theory could be rebuilt from scratch if you lost the image.
 

Joom

 ❤❤❤
Member
Joined
Jan 8, 2016
Messages
6,067
Trophies
1
Location
US
Website
mogbox.net
XP
6,075
Country
United States
Ok but thats a full server image right? If you just wanted to backup the actual database, like source code of all branches of all platforms, plus documentation, etc surely that would compress down to far less? This is the stuff you want to backup in real time, or atleast daily.

Things like a server core image if you exclude the database, shouldn't change as often and in theory could be rebuilt from scratch if you lost the image.
You have to take into consideration that the image also contains the build environment. For production, you need this kind of image.
So... was it wrong of me to update my nightly Retroarch app?
No, you're fine.
No RCE afaik, but maybe in one ver over the years
That's good. That would require something much more than just poor authorization handling anyway, and again, this isn't a priority or pertinent to emulation itself, so it's understandable that netplay is just "meh" status. I mean, it's just a UDP connection, so there might be a way to turn it into a DoS at the very least, but I kinda doubt it.
 
Last edited by Joom,

Silent_Gunner

Crazy Cool Cyclops
Banned
Joined
Feb 16, 2017
Messages
2,696
Trophies
0
Age
29
XP
4,727
Country
United States
Is it that surprising
Some time ago I set up a password protected netplay session to play with a friend and someone hacked into our room almost immediately.
They made it in faster than my friend, who I was in a call with
Retroarch is not safe

Solution is quite simple: use Parsec. I tried it and, when the connection is good, outside of maybe some compression here and there, it's like you're playing the game in the same room with the person in terms of lag and the usual problems that are associated with streaming games.

Or does Parsec have "hacker-jumping-into-private-lobbies" problems as well?
 

mituzora

Well-Known Member
Member
Joined
Aug 23, 2016
Messages
315
Trophies
0
Age
31
XP
1,039
Country
United States
At least some of the cores are hosted on AUR, so they can at least get a base of the cores from there. So that's good news at least. It's stupid that GitHub said they can't do anything. they're owned by big-pocket Microsoft, and you can't tell me they don't do something akin to VSS copies on every repository.
 

tigersaman

Well-Known Member
Newcomer
Joined
Jul 10, 2018
Messages
78
Trophies
0
Age
31
XP
1,059
Country
Iran
Tenfold the amount of data and add 7 day retention time.
You also want a redundant copy offsite in a different datacenter as well (relevant for new infra).
so ... you have 3 terabyte of code and stuff on server ? how ? what do you store there ? and 7 day retention with is like 10 or 20 percent more than original so even if there is realy 3 terabytes of data it cost around 60 bucks. and 60 more bucks for another one on another datacenter. still i can't believe there is 3 terabytes of data there.

ps: I just did a quick search and it's way cheaper the more you go up in storage. your argue is totally invalid.

edit: I just read your other answers and it seems money was not a major problem. there is a lesson to be learned in this situation. i'm a website developer and i learned the lesson the hard way. BACKUP IS THE MOST IMPORTANT THING ! :D
 
Last edited by tigersaman,
  • Like
Reactions: mathew77

m4xw

Ancient Deity
Developer
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,956
Country
Germany
so ... you have 3 terabyte of code and stuff on server ? how ? what do you store there ? and 7 day retention with is like 10 or 20 percent more than original so even if there is realy 3 terabytes of data it cost around 60 bucks. and 60 more bucks for another one on another datacenter. still i can't believe there is 3 terabytes of data there.

ps: I just did a quick search and it's way cheaper the more you go up in storage. your argue is totally invalid.

edit: I just read your other answers and it seems money was not a major problem. there is a lesson to be learned in this situation. i'm a website developer and i learned the lesson the hard way. BACKUP IS THE MOST IMPORTANT THING ! :D
I know that from my own experience as well, the server doesnt run a raid and didn't have a backup in 8 years!
Just wouldnt touch that shit without at least a copy if i cant do a snapshot, lol.
It will take a while to clean all that up if the funding is secured.
Talking about hundreds of Build rules for CI/CD for dozens of platforms..
 
  • Like
Reactions: ChronoTrig

Silent_Gunner

Crazy Cool Cyclops
Banned
Joined
Feb 16, 2017
Messages
2,696
Trophies
0
Age
29
XP
4,727
Country
United States
2TB on current, 3TB projected on new infra.

You have this backed up locally, I hope? Because I don't know about Germany, but over here in the US, getting even an external 5TB HDD can be pretty cheap. I know Best Buy had a sale recently on some easystore Western Digital external HDDs that's now done, but you could get a 5TB variant of it for less than $100!

That being said, after my previous main PC went up in smokes (not literally, thank god, the mobo just died), I usually try to use the internal SSD as little as possible other than for some games here or there due to it being a M.2 drive, and data being a little bit more of a hassle to recover in terms of acquiring an adapter from what I remember.
 

m4xw

Ancient Deity
Developer
Joined
May 25, 2018
Messages
2,442
Trophies
1
Age
119
XP
6,956
Country
Germany
You have this backed up locally, I hope? Because I don't know about Germany, but over here in the US, getting even an external 5TB HDD can be pretty cheap. I know Best Buy had a sale recently on some easystore Western Digital external HDDs that's now done, but you could get a 5TB variant of it for less than $100!

That being said, after my previous main PC went up in smokes (not literally, thank god, the mobo just died), I usually try to use the internal SSD as little as possible other than for some games here or there due to it being a M.2 drive, and data being a little bit more of a hassle to recover in terms of acquiring an adapter from what I remember.
Its backed up in a german datacenter right at the backbone ;)
Otherwise this would take a few weeks more, hah.
 
  • Like
Reactions: Silent_Gunner

the_randomizer

The Temp's official fox whisperer
Member
Joined
Apr 29, 2011
Messages
31,284
Trophies
2
Age
38
Location
Dr. Wahwee's castle
XP
18,967
Country
United States
GitHub customer service is fucking stupid, no wonder because Microsoft owns it.

And also to the hacker who did this, you can screw off. This is such BS >.>

lul imagine using retroarch

Lol imagine being a troll

If they'd done it without pissing so many people off and begging for $ and getting more than the emulator developers (who put way more work in than retroarch) then I might agree.

There are a lot of emulator developers who are not shedding a tear over the hack.

Yeah and they're all conceited jerks too, I'm sure
 
Last edited by the_randomizer,

Pickle_Rick

I'm a pickle Morty!
Member
Joined
Aug 28, 2017
Messages
694
Trophies
0
Age
27
Location
Garage
XP
1,469
Country
United States
If they'd done it without pissing so many people off and begging for $ and getting more than the emulator developers (who put way more work in than retroarch) then I might agree.

There are a lot of emulator developers who are not shedding a tear over the hack.
Do those devs plan on porting their emus to every platform RetroArch supports? Then maintaining it? NO? Then they can piss off.

Where is the native port of Gambatte to 3DS? Nestopia? Mame? FBN? Dosbox? Etc, etc...
They don't exist? That's what I thought. The devs who hate RA for these petty reasons piss ME off.
 
  • Like
Reactions: Silent_Gunner

Silent_Gunner

Crazy Cool Cyclops
Banned
Joined
Feb 16, 2017
Messages
2,696
Trophies
0
Age
29
XP
4,727
Country
United States
Do those devs plan on porting their emus to every platform RetroArch supports? Then maintaining it? NO? Then they can piss off.

Where is the native port of Gambatte to 3DS? Nestopia? Mame? FBN? Dosbox? Etc, etc...
They don't exist? That's what I thought. The devs who hate RA for these petty reasons piss ME off.

Final Burn was ported to other systems standalone before, was it not? Or have things changed over there due to the shenanigans that went down over the emulator being used in that one Capcom arcade stick that honestly kind of looks cool, but is more of a collector's item than anything.
 

Pickle_Rick

I'm a pickle Morty!
Member
Joined
Aug 28, 2017
Messages
694
Trophies
0
Age
27
Location
Garage
XP
1,469
Country
United States
Final Burn was ported to other systems standalone before, was it not? Or have things changed over there due to the shenanigans that went down over the emulator being used in that one Capcom arcade stick that honestly kind of looks cool, but is more of a collector's item than anything.
It was but not officially and it wasn't maintained. Regardless my point is that they don't support every platform RA does. There is a tangible benefit that RA gives people that they may want to pay for.
 
  • Like
Reactions: Silent_Gunner

64bitmodels

Professional Nintendo Hater
Member
Joined
Aug 1, 2019
Messages
1,451
Trophies
1
Age
18
XP
2,883
Country
United States
I would love to see you do better.
as much as i like retroarch...
you can just fuck off, this "id LOvE TO see YOu do BETTeR" attitude is complete cancer
you shouldnt have to be an expert in emulator/frontend development to criticize an emulator or not like it
 

Site & Scene News

Popular threads in this forum

General chit-chat
Help Users
  • BigOnYa @ BigOnYa:
    I remember seeing the very first episode back in the day, and have watched every episode since. I used to set my VCR to record them even, shows how long ago.
  • BigOnYa @ BigOnYa:
    I just like any comedies really, and cartoons have always been a favorite of mine. Family guy, American Dad, Futurama, Cleveland Show, Simpsons - I like them all.
    +1
  • BigOnYa @ BigOnYa:
    South Park is great cause they always touch on relavent issues going on today, and make something funny out of it.
    +3
  • S @ salazarcosplay:
    @BigOnYa were you always up to date on the current events and issues of the time or were there issues that you first found out thru south park
  • BigOnYa @ BigOnYa:
    Most of the time yea I knew, I watch and read the news regularly, but sometimes the Hollywood BS stuff, like concerning actors slip by me. I don't follow most Hollywood BS (example: the Kardasians)
    +2
  • S @ salazarcosplay:
    @BigOnYa there were relevant issues before south park was made, that's why i think a south park prequel/spinoff would be great. Randy and his friends in their child hood
    +1
  • BigOnYa @ BigOnYa:
    Yea, like them running in high school together, getting into stuff, and how they got hitched and had kids. And how the town of South Park was back then compared to now. That would be cool to see.
  • BakerMan @ BakerMan:
    yeah
  • The Real Jdbye @ The Real Jdbye:
    @salazarcosplay if they made a prequel, it would still be about current issues, cause it doesn't make sense to make it about stuff that happened 30 years ago that nobody cares about anymore
  • The Real Jdbye @ The Real Jdbye:
    it's too late
  • The Real Jdbye @ The Real Jdbye:
    the older south park episodes about particular issues usually age poorly since the topic is no longer relevant
  • The Real Jdbye @ The Real Jdbye:
    an exception is giant douche vs turd sandwich, that's always relevant :P
    +1
  • K3Nv2 @ K3Nv2:
    I was gone for like an hour and none of you thought to write or call pos
  • BigOnYa @ BigOnYa:
    We knew you were going to Sonic to get lunch.
  • K3Nv2 @ K3Nv2:
    Sonics fast I would've been home in 10 mins
  • BigOnYa @ BigOnYa:
    Meet and greet with AncientBoi then?
  • K3Nv2 @ K3Nv2:
    That would've gone slow he's old
    +1
  • ZeroT21 @ ZeroT21:
    sadly the person in question feels too young for his own good
  • K3Nv2 @ K3Nv2:
    We don't question people
  • ZeroT21 @ ZeroT21:
    me neither, i just bash them
  • K3Nv2 @ K3Nv2:
    We just question @AncientBoi
  • ZeroT21 @ ZeroT21:
    it wasn't a question, it was fact
  • BigOnYa @ BigOnYa:
    He said he had 3 different doctors apt this week, so he prob there. Something about gerbal extraction, I don't know.
    +1
  • ZeroT21 @ ZeroT21:
    bored, guess i'll spread more democracy
    ZeroT21 @ ZeroT21: bored, guess i'll spread more democracy