In mid-September we briefly mentioned the spate of vulnerabilities exposed by AI. In the intervening weeks it has not gotten any better. Quite the opposite, actually. The Hackening they’re calling it. And while Synclinical engineers are more sanguine about the future than some, it’s safe to say that things will get worse before they get better.
Recall from the earlier write up that most of the vulnerabilities that AI is exposing are found by the good guys, who report them to the appropriate engineering team, who fix the vulnerabilities and release a patch. This is when the bad guys spring into action. With the patch pointing the way, it’s trivial for them to craft an exploit of the vulnerability and unleash it on the world, knowing that all un-patched systems are targets.
In response, here’s what Ready Room engineering has been doing over the past few weeks:
Ready Room already sits behind Cloudflare’s proxy servers. That is, when you connect to Ready Room, you are really connecting to Cloudflare, which then proxies the request on to us. In between, Cloudflare examines each request for known threats and drops them if they look suspicious. Last week, we upgraded our Cloudflare license and dialed up many of the security settings, essentially stopping most vulnerability scans and suspicious requests before they even reach us.
We have updated our CI/CD process to query our package providers for all recent security updates. If we are running a vulnerable package, Ready Room will not build, forcing us to upgrade immediately. Similarly, we have enabled our source code control system to request a code change to upgrade any server- or client-side package that has been updated for any reason.
On top of that, we have developed a tool that runs every night and attempts to merge those changes into our go-forward development branch. The merge process, in turn, is supported by our suite of almost 5,000 automated tests (2,000 more than just 6 weeks ago), and will not complete if any of them fail.
Finally, if an issue is particularly severe, we leverage our support for blue/green deployments, and release these patches in the middle of the day—without interrupting current users.
What does all that look like in the real world? Since the beginning of October, 60% of all Ready Room dependencies have seen updates, and we have upgraded them all. Every single component that Ready Room relies on, from the OS on up, is always at its most recent release.
Again, we encourage you to ask your other system vendors how they are responding to The Hackening. Anything short of aggressive patching, updating, and firewalling is cause for concern.
PS: Since a couple of exploits actually made the news during “Black October”, they deserve a little extra discussion.
PHP2Shell
PHP is a hugely popular programming language for building web applications. Facebook, Wikipedia, and WordPress are all built largely or entirely in PHP. A flaw was discovered in PHP that would allow an attacker to run arbitrary code on the website’s host.
Ready Room does not use PHP in any capacity and is not vulnerable to this exploit. The same is true for all Ready Room partners.
eMudhra Weak Keys
Public key cryptography is the heart of nearly all network security, utilizing paired public and private keys to enable encryption, authentication, and data integrity. Certificate Authorities (CAs) are companies entrusted with verifying the identity of websites and servers. They do so by issuing certificates (aka certs) “signed” with their private key. For this to work, private keys are meant to be extremely large, undiscoverable, random numbers.
eMudhra is one such CA. For reasons still unknown, they used criminally simple private keys (literally 4, 56, 66, 115, 155, 156, 230, and 237) to create eight of their own intermediary certs. Attackers could use these compromised keys to create trusted certificates of their own and thereby impersonate any website on the planet. Fortunately, in most cases, the attacker would also need a second exploit in order to intercept the network traffic.
No website or browser was immune to this flaw (which had nothing to do with AI, by the way). Fortunately, the compromised certs were withdrawn quickly. And the many, many places where these certificates were stored were updated.
On Ready Room’s part, we quickly updated the two (unused, but still required) certificate stores embedded in the system, as well as our monitoring software which embedded its own. In addition, our PaaS provider, Gigalixir, just as quickly provided an updated OS bundle, whose certificate store Ready Room actually uses. All of this was rolled out days ago.
