Embrace The Red

You probably heard of NFTs (non-fungible tokens). They are receiving a lot of interest over the last several months. I did some digging and realized that there are some bigger issues with the standards and various interpretations and implementations of it, and how centralized many offerings are.

What is an NFT

The idea behind it is simple, use a blockchain (and cryptography) to assign and be able to proof ownership over a specific piece of digital content.

Next week (on March, 9th 2021) I will be speaking at the Hong Kong Information Security Summit 2021.

歡迎, 你好!

I was invited to share my thoughts around protecting the modern (and remote) workplace. Of course, my talk is addressing this topic from a red teaming point of view. Conference details are here.

The adversary will come to your house

The name of the talk is “Red Team Strategies for Helping Protect the Modern Workplace” which might seem less creative, but there is some (hopefullly) good and interesting information in my talk.

The other day I read this blog post about “The Death of Manual Red Teams” and I thought I’d take a moment to comment on it to provide an alternative perspective.

In my opinion the premise of the blog post is backwards, highlighting a lack of understanding of what red teaming is about.

For instance the following sentence in the post seems quite incorrect: “Red teaming is the process of using existing, already known security bugs and vulnerabilities to hack a system.”.

The Kindle (ebook) edition of “Cybersecurity Attacks - Red Team Strategies” is currently free on Amazon.

Grab your copy while it lasts and spread the word!

The other day Microsoft published a great deep dive around the second stage payloads and hands-on hacking activities of the Solarwinds/Sunburst incidents that where uncovered late 2020.

One thing that is so interesting is the use of off the shelve Cobalt Strike tooling and templates for command & control. After doing all the hard work and customization to seamlessly backdoor binaries the adversaries sem to use Cobalt Strike.

Intel for Red Teamers

Although, they did add customizations with some interesting features, which are interesting for red teamers to be aware of. For instance each instance of the zombie would be unique in name, folder locations, etc. to make it more difficult to identify in environments.

Survivorship bias is an interesting thing that we can observe nearly daily. It is the success stories of exponentially growing startups, the motivational speaker who shares their insights on how to be successful and so forth.

What is it exactly?

What is survivorship bias?

Wikipedia defines it as “…the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.”

Security metrics are an interesting topic.

Over the years I used “scores” as a tool to identify and shine light on problematic areas or highlight lack of engineering and security quality of certain teams.

A security score should not seen as an objective or absoulte measure, but it allows to compare systems with each other at a relative scale, and by sharing the score it makes people ask questions.

Today FireEye shared that they were victim of a cyberattack and internal red teaming tooling was accessed by adversaries. More details in this NYT article.

This reminded me that I wanted to do a post on actively protecting pen testers and pen testing assets for a while.

Against persistent adversaries it is only a matter of time when they succeed, not if they will succeed. The big question is do you know when an adversary starts poking around, and when they succeed?

What a journey it has been. I wrote quite a bit about machine learning from a red teaming/security testing perspective this year. It was brought to my attention to provide a conveninent “index page” with all Husky AI and related blog posts. Here it is.

Machine Learning Basics and Building Husky AI

• Getting the hang of machine learning
• The machine learning pipeline and attacks
• Husky AI: Building a machine learning system
• MLOps - Operationalizing the machine learning model

Threat Modeling and Strategies

• Threat modeling a machine learning system
• Grayhat Red Team Village Video: Building and breaking a machine learning system
• Assume Bias and Responsible AI

Practical Attacks and Defenses

• Brute forcing images to find incorrect predictions
• Smart brute forcing
• Perturbations to misclassify existing images
• Adversarial Robustness Toolbox Basics
• Image Scaling Attacks
• Stealing a model file: Attacker gains read access to the model
• Backdooring models: Attacker modifies persisted model file
• Repudiation Threat and Auditing: Catching modifications and unauthorized access
• Attacker modifies Jupyter Notebook file to insert a backdoor
• CVE 2020-16977: VS Code Python Extension Remote Code Execution
• Using Generative Adversarial Networks (GANs) to create fake husky images
• Using Microsoft Counterfit to create adversarial examples
• Backdooring Pickle Files
• Backdooring Keras Model Files and How to Detect It

Miscellaneous

• Participating in the Microsoft Machine Learning Security Evasion Competition - Bypassing malware models by signing binaries
• Husky AI Github Repo

Conclusion

As you can see there are many machine learning specific attacks, but also a lot of “typical” red teaming techniques that put AI/ML systems at risk. For instance well known attacks such as SSH Agent Hijacking, weak access control and widely exposed credentials will likely help achieve objecives during red teaming operations.

In this post we will explore Generative Adversarial Networks (GANs) to create fake husky images. The goal is, of course, to have “Husky AI” misclassify them as real huskies.

If you want to learn more about Husky AI visit the Overview post.

Generative Adversarial Networks

One of the attacks I wanted to investigate for a while was the creation of fake images to trick Husky AI. The best approach seemed by using Generative Adversarial Networks (GANs). It happened that right then deeplearning.ai started offering a GAN course by Sharon Zhou.

There are plenty of examples of artificial intelligence and machine learning systems that made it into the news because of biased predictions and failures.

Here are a few examples on AI/ML gone wrong:

• Amazon had an AI recruiting tool which favored men over women for technical jobs
• The Microsoft chat bot named “Tay” which turned racist and sexist rather quickly
• A doctor at the Jupiter Hospital in Florida referred to IBM’s AI system for helping recommend cancer treatments as “a piece of sh*t”
• Facebook’s AI got someone arrested for incorrectly translating text

The list of AI failures goes on…

You might have heard about “NAT Slipstreaming” by Samy Kamkar. It’s an amazing technique that allows punching a hole in your routers firewall by just visiting a website.

The attack depends on the router having the Application Layer Gateway enabled. This gateway can be used by anyone inside your network to open a firewall port (totally by design). Protocols such as SIP (Session Initiation Protocol) use it.

What I will focus on in this post is the Application Layer Gateway (ALG) and SIP. In particular to understand how these can be abused by an adversary.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts.

• Overview: How Husky AI was built, threat modeled and operationalized
• Attacks: The attacks I want to investigate, learn about, and try out

In this post we are going to look at the “Repudiation Threat”, which is one of the threats often overlooked when performing threat modeling, and maybe something you would not even expect in a series about machine learning.

My GrayHat Red Team Village talk “Learning by doing: Building and breaking a machine learning system” is now live on YouTube.

Check it out: https://www.youtube.com/watch?v=-SV80sIBhqY and smash the Like button! :D

Question?

I thought of turning the content into a hands-on workshop. Let me know if that would be something that would you would attend? Trying to see if there is interest.

Cheers, Johann

Twitter: @wunderwuzzi23

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts.

• Overview: How Husky AI was built, threat modeled and operationalized
• Attacks: Some of the attacks I want to investigate, learn about, and try out

A few weeks ago while preparing demos for my GrayHat 2020 - Red Team Village presentation I ran across “Image Scaling Attacks” in Adversarial Preprocessing: Understanding and Preventing Image-Scaling Attacks in Machine Learning by Erwin Quiring, et al.

A few years back the Blue Team of a company asked to be targeted in a Red Team Operation.

That was a really fun, because Rules of Engagement commonly prevent targeting Blue Teams. Blue’s infrastructure, systems and team members are often out of scope, unfortunately.

Blue team infrastructure is a gold mine for credentials, recon but also for remote code execution!

Often companies do not have adequate protection, procedures (MFA, multi-person attestation), monitoring and auditing in place when it comes to accessing data from endpoint agents. There is also frequently a lack of knowledge on what Endpoint Agents are capable of doing.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts.

• Overview: How Husky AI was built, threat modeled and operationalized
• Attacks: Some of the attacks I want to investigate, learn about, and try out

I wanted to explore the “Adversarial Robustness Toolbox” (ART) for a while to understand how it can be used to create adversarial examples for Husky AI.

For GrayHat 2020 I was asked to create a short intro video for my Red Team Village talk “Learning by doing: Building and breaking a machine learning system”.

So I put my green screen to good use and recorded this short clip for Red Team Village.

Here is the link to the clip on Twitter:

Hope you like it. :)

The talk will be October, 31st 2020.

There is a lot of discussion around terms such as red team, attack team, pentest, adversarial engineering or offensive security team and similar ones.

I typically stay away from the (sometimes passionate) discussions that ensue whenever this topic comes up.

Personally, I think a good strategy is to define programs and teams who operate in this space by what services the team (or teams) provide(s) to the organization.

The business groups, blue team, developers, engineers, employees and clients are the customers.

While building “Husky AI” I started working a lot with Microsoft’s VS Code Python extension. It is a super convinient way to edit Jupyter Notebooks. I just use VS Code’s Remote SSH feature to get to my Linux host and work on modeling and testing there.

When threat modeling “Husky AI” I identified backdooring of third party libraries and development tools as a potential issue to be aware of.