Lessons from building AI to Stop Cyberattacks

Successful Phishing attack on John Podesta that lead to the 2016 DNC email leaks. Released by WIkiLeaks.

On March 19th 2016 John Podesta, was tricked into revealing his Gmail credentials to a Russian-backed organization who then released emails regarding the Clinton campaign, effectively influencing the 2016 election.

Social engineering attacks like this (and much more sophisticated ones) are on the rise and getting more advanced every day. They prey on the trust we put in our business tools and social networks, especially when a message appears to be from someone on our contact list (but is not) or even more insidiously when the attack is actually from a contact whose account has been compromised. The FBI estimates that over the past few years over 75% of cyberattacks start with social engineering, usually through email.

Why is this a new problem?

Hasn’t email been around forever? Why hasn’t this problem been solved? Why is it a machine learning problem?

With more business being done online and connected to a single cloud login, as well as the increasing use of mobile devices where UX is often designed without security in mind, these forms of attacks have skyrocketed in number and are a threat to every organization that uses email. The sophistication and targeted nature of the attacks has, until recently, outpaced the existing email security.

The only way to effectively detect these attacks is by using modern machine learning techniques of deep natural language understanding combined with careful modeling of an organization’s typical communication patterns.

About a year and a half ago I left Twitter’s ML team to help found Abnormal Security where I lead the development of our AI system to detect and prevent these malicious emails.

I’m somewhat new to cybersecurity and to email security (though it has similarities to the problem of detecting abuse detection which I worked on at Twitter). It is a fascinating and scary world:

• There’s a multi-billion dollar economy of criminals using email-based attacks to steal money, account information, and trade secrets, but as many as 50% of these attacks are probably perpetrated by state-actors: either spy organizations or politically motivated organizations funded by states such as Russia, China, Iran, and North Korea. For example, Fancy Bear (APT28) was responsible for the successful attack on John Podesta, and is probably funded by the Russian security organization GRU
• Using zero-day exploits are not the only tools attackers have. It’s better for criminals to save those for a rainy day or sell them to a state actor. It’s much easier to convince someone to give up their credentials and use those to escalate privileges until you can get what you want, either money or trade secrets
• Existing email security has not been able to stop these attacks and the attacks are only getting more sophisticated.

Can we use modern AI to prevent attacks?

There’s a long history of using ML to stop email fraud, for example, Spam filters are taught in most introductory ML courses. However, attackers have far-outpaced existing email security. Nearly every organization from local governments, manufacturing, energy companies, technology companies, to individuals are targeted and many are successfully breached every day.

Why is it hard to prevent these attacks?

1. The text in email attacks is often indistinguishable from legitimate communication. It’s been carefully designed to fool both the recipient and security software.
2. Attackers are actively attempting to avoid ML systems, and using their own ML systems to thwart detection (for example, we found spear-phishing A/B testing software for sale on the dark web)
3. Attackers may launch attacks from compromised accounts making them even harder to identify from regular email.

Before Abnormal Security, I worked at Twitter Cortex bringing deep learning models to many areas of the Twitter product. One particular problem I worked on was abuse detection: How do we identify harassment, bullying, and hate speech in Tweet conversations? We found that a combination of modern NLP (embeddings and LSTMs etc.) alongside features of the communication graph (who is tweeting at who, what communities are they in, what is their communication history etc.) was the crucial combination to build successful detection models.

These lessons learned from Twitter were helpful when approaching the problem of identifying email attacks: (1)is the content of the message suspicious? (2) are identity of parties in the communication often targeted? (3) what is the past communication patterns of the recipient and sender?

Interestingly, Twitter and Email are two of the only large technologies where anyone can contact anyone else without a prior connection and are therefore both ripe for abuse But, email breaches are more insidious due to how much access is linked to email accounts.

In this age of cloud identity providers, so much information and access sis linked to email accounts on Office 365 and Gmail: documents, voice chat, video, possible even desktop access.

Why is this a hard ML problem?

A needle in a haystack — The first challenge is that the base rate is very low. Advanced attacks are rare in comparison to the overall volume of legitimate email:

• 1 in 100,000 emails is advanced spear-phishing
• less than 1 in 10,000,000 emails is advanced BEC (like invoice fraud) or lateral spear phishing (a compromised account phishing another employee)
• (compare to spam, which accounts for 65 in every 100 emails)

This means we have an extremely biased classification problem which raises all sorts of difficulties.

Enormous amounts of data — At the same time, the data we have is large (many terabytes), messy, multi-modal, and difficult to collect and serve at low latency for a realtime system. For example, features that a ML system would want to evaluate include:

• Text of the email
• Metadata and headers
• History of communication for parties involved
• Account sign-ins, mail filters, and other account activity
• Identity of parties involved (is it the CEO’s name ? an accountant? etc.)
• Attachment content
• Links in attachments
• Images in attachments
• Links and body
• Contents of linked landing page
• Images in linked landing pages
• Code in landing pages
• Malware in attachments
• …

Turning all this data into useful features for a detection system is a huge challenge from a data engineering as well as ML point of view.

Adversarial attackers — To make matters worse, attackers actively manipulate the data to make it hard on ML models:

• Attackers encode text with Unicode-lookalike text (e.g. 𝙼𝚒𝚌𝚛𝚘soft)
• Attackers insert distracting text hidden in non-displayed HTML (to confused NLP models)
• Attackers encode text in images (to prevent NLP without OCR)
• Phishing pages render all content with javascript and require a CAPTCHA to access (to prevent automated crawling)
• Attackers send innocuous emails to an organization for months to build up communication/reputation features
• Attackers include text in password-protected PDF files attached to an email (to prevent automatic parsing of attachments)
• Attackers sit on a compromised account for months waiting for the right moment. For example by inserting an illegitimate invoice into a conversation about payment at just the right time.

The precision must be very high — to build a product to prevent email attacks we must avoid false positives and disruption of legitimate business communications, but at the same time catch every attack. The false-positive rate needs to be as low as one in a million!

Successfully solving the problem

At Abnormal we’ve built an effective solution that relies on good data engineering, data science, and robust scalable systems underneath powerful ML models.

The key to detecting malicious emails is through good representations and discriminative featurization of that data. At the end of the day, our detection system relies on three dimensions of an email:

All this data is pulled together and made available on the receipt of each email. Our ensemble of detection models use these data sources in various ways, and we have redundant detectors for various classes of particularly damaging attacks. The core engine of detection is a multi-modal ML model.

Malicious Email Classifier

When we first started solving this problem, we used simple GBDT models on top of basic text and communication-graph based features, which I highly recommend to get started on this, or any other, ML problem. Begin with the simplest models you can before going onto sophisticated deep learning approaches.

Eventually, we outgrew simple models, and have found deep learning architectures particularly useful, not only for their predictive power but also for the convenience of including multiple modalities of data inside the same model. For example, one of our most powerful models needs to consume text data of various forms alongside tabular data:

The ability to easily combine and train models representing multiple dimensions of an email in this way has been extremely powerful and helps us generalize to never-before-seen attacks.

Thresholding the model output is one of the hardest problems. Since the precision must be so high, we must be very careful that the model performs well autonomously and can handle distributional shifts.

(I discuss difficulties around thresholding more in a previous post)

Some lessons

1. Study the attacks, trends, and false negatives. Build your data pipelines and features to represent what attackers might try next as well as variations on what you have seen.
2. Use human experts to create heuristics! Heuristics features can be some of the most powerful inputs into a model even if they are not sufficiently precise to be standalone rules.
3. Build a portfolio of detection models, heuristics, and signatures. There can never be too much redundancy in a detection stack.
4. Ensure each sub-model is representing its sub-problem well before trying to combine them into a larger network, or you will never make progress.
5. Always baseline and iterate: for example when building a text model, start simple and add complexity only when you can show it is better: (1) try heuristic phrases (2) try bag of words (3) try canned embeddings like fasttext (4) try fine-tuning your embeddings (5) try state-of-the-art techniques like BERT

Bringing it all together

For this whole detection system to work successfully, we must ingest terabytes of data a day — all the incoming emails and other signals we have for an organization — and maintain sophisticated aggregated feature stores to keep track of the communication patterns in realtime. Online we must process every incoming email at low latency, extract and join data, possibly even crawl links and process attachments, apply NLP models and apply featurization. Then we must pass the data through classification models, and combine results of all models for a final decision on whether the email is malicious. All this must be done at latencies of less than a second and maintaining an extremely low false-positive rate. This is a difficult engineering and data science problem.

We’ve built a powerful detection system at Abnormal Security that can prevent the most advanced targeting phishing, business email compromise, lateral phishing and account compromise. But the problem is ever-changing as attackers learn to thwart these systems, so our work is never done.

Some (of many) major challenges we are continually improving

• How can we detect invoice fraud better? To do so we must deeply understand the natural language and images in an invoice to identify Abnormal patterns.
• How can we detect phishing sites with higher accuracy? For an AI system to understand the content of the page we must render that page and use computer vision and NLP to identify malicious intent.
• How can we better identify account takeovers? This is a difficult anomaly detection problem with enormous volumes of data, what constitutes unusual usage behavior and emails from a compromised account?

Acknowledgments

All this work at Abnormal wouldn’t have been possible without the amazing team, especially those working closely on building this detection engine from the ground up: Dmitry Chechik, Kevin Lau, Sanny Liao, Yu Zhou Lee, Abhijit Bagri, Carlos Gasperi, James Yeh, Sanjay Jeyakumar, and the rest of the team.

And yes, we’re hiring! abnormalsecurity.com/careers