Security / Software Development
Optimizely, malware and the case for CSP

Your website runs code you've never seen
On the 16th of November, for around 17 hours, visitors to the websites of 178 organisations, including the BBC, Zoom, Salesforce and HSBC, were served malware - and none of those websites had actually been hacked. The malicious code came through Optimizely, an experimentation platform that a lot of large sites use for A/B testing, and it was delivered from Optimizely's own CDN, with Optimizely's own certificate, inside a script those sites had chosen to load on every page. It swapped cryptocurrency wallet addresses, skimmed card numbers typed into checkout forms and, on older iPhones, tried to exploit the device itself. A small number of those sites had a Content Security Policy that blocked it. Most didn't.
How did the Optimizely attack work?
To run its experiments, Optimizely asks each customer to add a script tag to their site, which loads a snippet from cdn.optimizely.com that's unique to their account. Inside that snippet is a field called Project JavaScript, where the customer can write their own code to run on every page, and that's the field the attacker wrote into. The attack was spotted by Report URI, the web security monitoring service founded by security researcher Scott Helme, who have published a detailed write-up. They don't yet know how the attacker got write access to so many customers' projects at once (Optimizely's own websites were affected too), but the result was that the file all of those sites trusted was still the genuine file, served from the genuine CDN, with something extra in it.
From the browser's point of view nothing was wrong, and as Scott Helme puts it, the CDN "served exactly what it was given". Most of the security advice we give, and follow ourselves - keep dependencies up to date, serve everything over HTTPS, only load scripts from providers you trust - wouldn't have helped here, because every one of those boxes was already ticked.
In the few days before, the same attacker had also been injecting code through scripts from Kissmetrics, StorifyMe and Minute Media. In our experience it's very common for a site to carry several third-party scripts like these, often added by a marketing team through Google Tag Manager without a developer ever seeing them, and each one is code you've agreed to run without being able to read it first.
What is a content security policy?
A Content Security Policy is a list your website sends to the browser alongside every page, saying where that page is allowed to load scripts from, which servers it's allowed to send data to, which other sites can show it in a frame, and so on. If anything on the page tries to step outside that list, the browser blocks it and, if you've asked it to, sends you a report of what it blocked.
It's essentially a firewall for your website, and we've always found it a valuable tool but quite a challenging one to work with. Like a firewall, it can block things you actually wanted, so a checkout stops loading or the analytics stop recording, and getting it right takes some care and time to work out exactly which traffic should be allowed.
We use a Content Security Policy on all of our live sites and applications, and part of what makes it challenging is that no two of them need the same configuration, because each one loads a different mix of payment providers, analytics and other third-party tools.
It also means everyone involved in a site has to work together. Adding a new tag in Google Tag Manager isn't the end of the story - it's one task out of several in putting new JavaScript onto a website or app, and the tag won't actually run until the Content Security Policy has been updated to allow it, which is usually a job for whoever looks after the code and hosting. We believe the marketing team, the developers and anyone else with a hand in the site need to be talking to each other if they're going to deliver the kind of web security that applications should have.
Of course every team has constraints on budget, time and deadlines, so each of those decisions is a balance between speed and thoroughness, and that's a balance every team has to choose for itself.
The Optimizely attack shows why that time and effort is worth it.
Would a content security policy have stopped the Optimizely attack?
For the sites with a carefully kept one, it did. Of the affected sites Report URI could assess, 68 had no policy at all and only 24 blocked the attack completely. Allowing cdn.optimizely.com never helped, since that's where the malicious code came from. What saved those 24 was a policy that only allowed scripts from a short list of named places, so when the snippet tried to fetch the rest of the malware from an unfamiliar domain, the browser refused.
HSBC's policy came close, blocking that domain, but a backup copy of the malware hidden in the snippet ran through a JavaScript feature called eval, which a policy only permits if it's been explicitly allowed - and Optimizely's documentation asks customers to allow it. Even where the malware did run, sites that limited which servers the page could talk to stopped the stolen card numbers and cookies from going anywhere.
Do you need a content security policy audit?
If your site loads tags through Google Tag Manager or any other third party, it's worth knowing exactly what your Content Security Policy allows, and whether anyone would find out when it blocks something. If you'd like a content security policy audit, or help getting one working alongside Google Tag Manager, drop us a line.
Questions or thoughts? Email us at hello@sinovi.uk.


