Just a few hours ago, RetroArch/Libretro's servers and main GitHub repositories have been targeted by a yet-unknown attacker.
The attack begun with the buildbot server being crippled, which means any subsequent automatic buildbot builds, and netplay won't be available until a new server is setup for this very purpose.

Our buildbot server got hacked just now. We were not anticipating this so we have no idea how long it will take us to restore the state of this. For now, we have to consider the server compromised and take the appropriate actions to ensure this does not happen again.

— libretro (@libretro) August 16, 2020

Buildbot server hacked - we have had a stretch goal for a long time for $1300 so we can actually have regular backups - we simply cannot afford this right now on top of the already enormous bills with our current finances. https://t.co/dNu6Fz5M8r

— libretro (@libretro) August 16, 2020

After that, and a few moments later, the hacker moved on to attack Libretro's repositories at GitHub.
This attack removed the entirety of codes for certain cores, like Mame, Mame 2003, DosBox and many others, and only left a dummy ReadMe with a vague description of the core.

Said hacker has moved to vandalizing our Libretro Github organization now, wiping every single repository. We're hoping @Github will be able to provide us with assistance here while this emergency unfolds.

— libretro (@libretro) August 16, 2020

Libretro/RetroArch - Hacker vandalised our buildbot and Github organization - what you should know https://t.co/faCsmYHs6z pic.twitter.com/wZKDjKxqGh

— libretro (@libretro) August 16, 2020

GitHub hasn't given any reply regarding what could be done in regards to the hacking to the GitHub repositories, but we'll keep updating this post as things go along.
The full overview of the attack and what was compromised on Libretro's side can be seen on their main Libretro.com page.

Approximately 5 hours ago, we were the target of a premeditated cybercrime attack on our key infrastructure.

The hacker did the following damage:

• He accessed our buildbot server and crippled the nightly/stable buildbot services, and the netplay lobby service. Right now, the Core Updater and Netplay Lobbies won’t work. The websites for these have also been rendered inaccessible for the moment
• He gained access to our Libretro organization on Github impersonating a very trusted member of the team and force-pushed a blank initial commit to a fair percentage of our repositories, effectively wiping them. He managed to do damage to 3 out of 9 pages of repositories. RetroArch and everything preceding it on page 3 has been left intact before his access got curtailed.

We are still awaiting any sort of response or support from Github. We hope they will be able to help us restore some of these vandalised Github repos to their proper state, and also to help us narrow down the attacker’s identity.

We wanted to clear up some confusion that may have arisen in the wake of this news breaking:

• No cores or RetroArch installations should be considered compromised. The attacker simply wiped our buildbot server clean, there is nothing being distributed that could be considered malicious to your system. Nothing has happened here and there is no need for any concern.
• For the current time being, the Core Installer is non-functional until further notice. The same goes for ‘Update Assets’, ‘Update Overlays’, ‘Update Shaders’, and all the other online services that RetroArch users normally have access to (such as the netplay lobby services).

The IP he was using while doing this was ‘54.167.104.253’, which seems to lead back to AWS.

We’re still assessing the situation but moving forward, we think that it’s probably best not to go forward with the buildbot server that was compromised earlier today. We had some long-term migration plans for a move to a new server, but this was always pushed back because we felt that we weren’t ready migration-wise. It might indeed be the case this is the catalyst for just starting all from scratch with a new server instead of trying to migrate the old one over. This would mean that the more commonplace builds for Linux/Windows/Android would be immediately available, but all the specialized systems like consoles, old MSVC builds and whatnot would have to wait for later until we have adapted this properly to the new system.

Lack of automated backups
This brings us onto another key issue – the lack of backups. We last performed a backup of our buildbot server about a couple of months ago. The truth is that while we pay a hefty amount for the servers on a monthly basis already, there is simply not enough money to pile on automated backups as well. We could really use your support on Patreon to help lighten our financial burden here, especially since this now-pretty-much-mandatory server switch will likely cost us an insubstantial amount of money upfront while we keep the current server running for a month longer.

How will we restore things
So, how are we going to restore things? We hope that Github will be able to restore the affected repositories. If they are unable to do so, we could rely on the goodwill of users to source us with git repositories with the full history intact.

As for the buildbot? No idea to be quite frank. If we make the switch to the new server, you’ll get Android/Windows/Linux up and running early again but all other platforms will have to be added as we go along.

It’s a shame what is happening to the emulation and homebrew community. When it isn’t developers leaving for greener pastures deciding it’s no longer worth it, prestigious developers like byuu are being forced to early retirement because of unsavory online gang-stalkers. In our situation, we can’t rule out the possibility that some of these attacks come from some of the same usual suspects (it isn’t the first time we’ve seen them abuse AWS for some of these attacks, we encountered them a year ago earlier targeting our lobby services). Whatever their aim may be, while they will not deter our will to continue working on this project, they have definitely increased our maintenance and cost burden for the time being. And for this we ask for your understanding and support as we attempt to come up with a plan to address these problems moving forward. Supporting us through Patreon is a great way of helping out, especially if we can reach the $1300 goal which means we can spend a bit more each month to make sure our stuff is properly backed up.

As if the complications with Android’s new store policies that requires us to coordinate with new contributors to come up with a workable solution was not enough of a headache, this comes along. With your help and support, we will overcome this and come out stronger than before.

Regarding the Android / Core Installer situation
While we’re on this subject briefly, while it’s off-topic, we felt the need to address this real quick. We will likely be making a version of RetroArch Android that is neutered ONLY for Google Play. It will mean that the Core Installer will not be available for this, and cores will come packaged in additional APKs that can be installed. Apparently there is a 50-core extra APK limit on this until it starts requiring a version of Android over version 8.0. So while trying not to artificially bump the Android OS system requirements, we’re deciding on a 50 core-APK limit for now. Hopefully we can fit nearly most of the cores within such narrow constraints.

On our download site (and on F-Droid), we will have a RetroArch Android version that will work as before – with the Core Installer feature completely left intact. We feel this is a much superior version to what will be available on the Play Store, but unfortunately Google will force our hand here.

UPDATE:
GitHub has replied back to Libretro, with the sad news that they don't have a way to restore or have a backup of the repositories Libretro had before the hack. It seems the restoration of the repositories will have to be done (alongside the help of other users) through full commit pushes that hold the entire history of the repositories:

Github has told us that they are not able to restore our repositories - "I'm sorry to say that we aren't able to restore a branch or repository to its previous state on our end." So we will have to do it manually.

— libretro (@libretro) August 17, 2020

UPDATE #2:
Libretro has restored the vast majority of the repositories back to shape. The only downside has been the loss of recent Pull Request:

We have now restored most of the Github repositories by hand. There might still be some PRs that were force closed during the vandalism, we will have to figure out a solution there still. Step 2 is restoring the buildbot for temporary use until we use the new server.

— libretro (@libretro) August 18, 2020

UPDATE #3:
The buildbot is now back online, though not at the same extend as it was before the attack
http://buildbot.libretro.com/

Right now some builds for multiple platforms have been compiled, but some of those builds might be prone to failure.
I recommend trying them out only if you have your previous build backed up in case some compilation went bad.

The @libretro buildbot is back alive and slowly starts building stuff again.
Meanwhile we got the new Servers and started our work on the new Infra!

— m4xw (@m4xwdev) August 21, 2020

Keep in mind, the downloader will still not work since I had to clean all old builds, otherwise lots of people would've downgraded by accident :P
Also expect the next Switch nightlies to be broken anyway, need to take a look and the toolchain on the bot isnt updated yet either.

— m4xw (@m4xwdev) August 21, 2020

Source

 

[Joom]

Is it just me, or does it seem like hackers have been more active than usual this year?

It's because of quarantine. This has been confirmed for a while now. More people are working from home who don't know how to implement security measures on their network, and hackers know this. There's also a lot of boredom in the world right now, and as the saying goes, if you put a thousand monkeys in a room with type writers one is bound to write a screenplay.

 

[duwen]

There's also a lot of boredom in the world right now, and as the saying goes, if you put a thousand monkeys in a room with type writers one is bound to write a screenplay.

That also explains all the shit on VOD streaming platforms right now too... except in that instance it seems like all thousand monkeys are having their screenplays produced.

 

[Solid One]

Damn, why hack RetroArch servers and repos? They're doing an excelent job maintaining a lot of emulators within a same interface, and even having a pretty decent netplay functionality. I use them a lot in RetroPie, in a Raspberry Pi.

I understand if some of the guys behind RetroArch were doing something wrong somehow, just like the guy from Final Burn Alpha that sold the emulator to Capcom without consulting other contributors, which culminated in a new fork called Final Burn Neo. But I haven't seen anything like this from RetroArch's side, which leads me to think the hackers that have done this are kinda the 'wrong guys' here.

Anyway, hope they can get RetroArch back. It's an awesome open source project that shouldn't die like this.

 

[m4xw]

Don't anyone of them have a local backup?

We do have backups of all things, restore takes like 30h+ still tho, from the time of writing.

 

[Raylight]

Probably Karma for the main dev and his shit ive heard about.

 

[atoxique]

Probably Karma for the main dev and his shit ive heard about.

explanation please? i

 

[tigersaman]

simple ftp backup space for 300gb costs like 5 bucks a month. are they really that desperate ?

 

[Clydefrosch]

fine, I'll chip in, I'll just hope this wasn't just a clever ploy to make it to that milestone.

 

[m4xw]

simple ftp backup space for 300gb costs like 5 bucks a month. are they really that desperate ?

Tenfold the amount of data and add 7 day retention time.
You also want a redundant copy offsite in a different datacenter as well (relevant for new infra).

 

[smf]

Damn, why hack RetroArch servers and repos? They're doing an excelent job maintaining a lot of emulators within a same interface,

If they'd done it without pissing so many people off and begging for $ and getting more than the emulator developers (who put way more work in than retroarch) then I might agree.

There are a lot of emulator developers who are not shedding a tear over the hack.

 

[PatrickD85]

Wow ... that sucks big time for a project as vig as Retroarch/Libretro.
Especially as they did so much good for gamers all around the globe.
Only people who might not like them ... are gaming companies themselves ... but they wouldn't ... or would they?

 

[Raylight]

explanation please? i

the main dev has been known to harass other devs and be a real dick to pl who dont share his beliefs that every emu should be open source and will smack talk about it to no end

 

[evertonstz]

No backups? I backup my old games at least in 3 places could and drives....

Conspiracy theory: Retroarch need some donations so they pretended to be hacked. ...jk

Oh yeah, kid, you backing your library in three places is just the same as backing up multiple git repositories that have literally years of commits and builds plus the redundancy and automation.

Jesus Christ, that's why nobody likes developing emulators/homebrew anymore, non coders think they know everything about everything because they coded "hello world" by seeing a youtube video and shitpost in internet forums.

 

[smf]

the main dev has been known to harass other devs and be a real dick to pl who dont share his beliefs that every emu should be open source and will smack talk about it to no end

Not just him, I heard byuu quit the scene after being hounded by mudlord for 6 years.

Conspiracy theory: Retroarch need some donations so they pretended to be hacked. ...jk

I wouldn't be surprised, but I think this just landed on them by surprise and just gave them an excuse to ask for more $$$

Either way, nobody should ever give them any money.

 

[Joom]

but they wouldn't ... or would they?

They wouldn't. Gaming companies don't give a single care about emulators unless they do something illegal. If nobody remembers, this also happened to Luma's Git because someone who had access used the same password on a site that was compromised. This kind of stuff isn't uncommon. Disgruntled developers who have access have done this in the past as well.

 

[PatrickD85]

They wouldn't. Gaming companies don't give a single care about emulators unless they do something illegal. If nobody remembers, this also happened to Luma's Git because someone who had access used the same password on a site that was compromised. This kind of stuff isn't uncommon. Disgruntled developers who have access have done this in the past as well.

I hope you are right when it comes to that. But they shouldnt indeed totally agree on that.

As for passwords etc. From my multiple years in the hosting space ... I learned to never re-use any password.
Otherwise you get a domino effect rather simple.

 

[FAST6191]

I am going to need some help understanding this build server scenario/general setup and how it could go so wrong here.

As I understand it
Retroarch itself is a frontend for emulators but I assume each of the "cores" (that being other existing emulators things boot into/load from and retroarch presumably handling some of the issues with save locations, controller settings, graphics setups and the like) all want their own special versions of compilers, packers, asset tweaking and whatever else in addition to the main project both as getting everything on the same page (or same page for what can be if there are very disparate languages) is a nightmare and the cores themselves are external projects that the devs are presumably happy enough with their own build setups and don't want to change it on account of this (see also the far from unjustified apathy and indifference, or worse, from various emulator devs towards this project).
Not a makefile/build batch file I would want to make from scratch, and I can well imagine a few of those tools conflicting so needing to be isolated after a fashion (whether other paravirtualisation, full virtualisation, sandboxie or custom to appear as another I don't know). All this multiplied again as various target architectures exist rather than just Windows X86/X64 PC though I am not sure how tied in they are (things vary somewhat).

Also good reason to have a remote version of the compiler setup so randoms, or a mainline dev playing on a tablet somewhere, just wanting to type a bit of C on an evening or whatever to squash a bug or tweak a feature don't have to install the complete build setup or figure out a piecemeal variant for their particular emulator/"core" of choice. Personally I would have probably gone for the piecemeal approach and precompiled versions some of the Linux "we like source code" distros operate with but I can see that being additional maintenance and not gaining the absolute most.

So anyway this many stacked containers/virtual machines added up to something notable it seems, above someone claiming hundreds of gigs (an impressive amount for a few compilers but I can see it, especially if full VM approaches are adopted/needed).
At the same time was development/alteration so fast paced that something in the order of a once a month (or few months) backup is so impossibly far behind the curve as to be useless? If it is container and VM based then that surely also lends itself to even more able to keep current (500 gigs is nothing special but can be a slight imposition, especially if you are somewhere backwards like the US and have bandwidth caps in addition to already "some dude with some flags" bandwidth) not entirely differently to how git works itself.
So now single point of failure and enough people with admin creds presumably that someone utilised a lesser guarded (or maybe forgotten) address with a nice bit social engineering to press the delete button, for which github has no restore option.

Guess then the usual lesson of never too big, never too small, never too virtuous and never too careful to not have backups. Could also be a good time to revisit a monster if it is going to be several hundred gigs and make it more robust.

Sucks I guess, but at the same time if I ask the "what has retroarch ever done for us?" question I can see why several are not particularly distraught and saying this is going to set back the state of emulation a while.

 

[Joom]

Oh yeah, kid, you backing your library in three places is just the same as backing up multiple git repositories that have literally years of commits and builds plus the redundancy and automation.

You're not wrong. Backing up code is actually easier. Redundancy and retention are also trivial. It's not like making a Git archive is hard, either. It's an available feature on the main page of every repo. Usually everyone involved also has the repo available locally.

@m4xw, maybe you guys could look into renting a Kimsufi box. They're an OVH reseller with very fair prices for the specs offered. If all you really need is storage, they're perfect. I've ran a seedbox with them for several years with no issues.

 

[Agusto101]

I don't really understand why would someone hack retroarch its not even a company or something just a normal app full of emulators, btw this are not going to stop them is just a little trouble, I'm a dev too and know that pain in the ass.

 

[Joom]

I don't really understand why would someone hack retroarch its not even a company or something just a normal app full of emulators, btw this are not going to stop them is just a little trouble, I'm a dev too and know that pain in the ass.

For teh lulz, of course. That's exactly what happened here.