Embrace The Red

There is a lot of discussion around terms such as red team, attack team, pentest, adversarial engineering or offensive security team and similar ones. I typically stay away from the (sometimes passionate) discussions that ensue whenever this topic comes up. Personally, I think a good strategy is to define programs and teams who operate in this space by what services the team (or teams) provide(s) to the organization. The business groups, blue team, developers, engineers, employees and clients are the customers.

While building “Husky AI” I started working a lot with Microsoft’s VS Code Python extension. It is a super convinient way to edit Jupyter Notebooks. I just use VS Code’s Remote SSH feature to get to my Linux host and work on modeling and testing there. When threat modeling “Husky AI” I identified backdooring of third party libraries and development tools as a potential issue to be aware of. So finding security issues in the tools is naturally something I am keeping an eye on.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out We talked about creating adversarial examples and “backdoor images” for Husky AI before. One thing that we noticed was that an adversary with model access can very efficiently come up with adversarial examples.

Excited to announce that I will be presenting at Grayhat - Red Team Village on October 31st 2020. The presentation is about my machine learning journey and how to build and break a machine learning system. If you follow my blog, you can guess that there will be lots of discussion around “Husky AI”. The bits and pieces that make up a machine learning pipeline, and how to threat model such a system.

This was also presented at BSides Singapore 2020. The slides are here and YouTube link is here. The origins of the Shadowbunny A few years ago, around 2016, I went on a relaxing two weeklong vacation. It was great to disconnect from work. I traveled to Austria, enjoying hiking in the mountains, and exploring Vienna. When I came back to the office, the team had placed a giant bunny teddy into my chair.

This year one of my goals was to learn about machine learning and artificial intelligence. I wrote about my journey before - including what classes I took and books I read, the models and systems I built and operationalized, threat modeling it to learn about practical attacks and defenses. My goal is to be knowledge enough in the AI/ML space enough to be able to help bridge the gap between research and operational red teaming - by doing practical things with life systems.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out Mitigations: Ways to prevent and detect the backdooring threat During threat modeling we identified that an adversary might tamper with model files. From a technical point of view this means an adversary gained access to the model file used in production and is able overwrite it.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out The previous post covered some neat smart fuzzing techniques to improve generation of fake husky images. The goal of this post is to take an existing image of the plush bunny below, modify it and have the model identify it as a husky.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. There are the two main sections of the series - more content will be added over time: Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out The previous post covered basic tests to trick the image recognition model.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. The previous four posts explained the architecture and how Husky AI was built, threat modeled and deployed. Now it’s time to start the attacks and build mitigations. The appendix in this post shows all the attacks I want to research and perform in this series over the next few weeks/months.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see all the posts, or visit the machine learning attack series overview section. In the previous post we walked through the steps required to gather training data, build and test a model to build “Husky AI”. This post is all about threat modeling the system to identify scenarios for attacks which we will perform in the upcoming posts.

This post is part of a series about machine learning and artificial intelligence. In the previous post we walked through the steps required to gather training data, build and test a model. In this post we dive into “Operationalizing” the model. The scenario is the creation of Husky AI and my experiences and learnings from that. Part 3 - Operationalizing the Husky AI model This actually took much longer than planned.

This post is part of a series about machine learning and artificial intelligence. In the previous post we described the overall machine learning pipeline. In this post we dive into the technical details on how I built and trained the machine learning model for Husky AI. After reading this you should have a good understanding around the technical steps involved in building a machine learning system, and also some thoughts around what can be attacked.

This post is part of a series about machine learning and artificial intelligence. In the previous post I talked about good resources for learning more about artificial intelligence and machine learning in general, and how I started my journey in this space. The next few posts will be about Husky AI. What is Husky AI? Husky AI allows a user to upload an image, and get an answer back if the image contains a husky or not.

This year I have spent a lot of time studying machine learning and artificial intelligence. To come up with good and useful attacks during operations, I figured it is time to learn the fundamentals and start using software, tools and algorithms. My goal was to build a couple of end to end machine learning systems from scratch, and then attack them. This post describes my studying approach, materials, courses, and learnings.

Excited to announce that I will be presenting at BSides Singapore this year. The topic is adversarial usage of virtual machines during lateral movement. And we will also cover threat hunting and detection ideas. I have been referring to this technique as the Shadowbunny over the years. :) The conferences is on September 24th-25th, it will be all virtual and free to attend. Check out the BSidesSG 2020 website and schedule for other talks and details.

Today I’m gonna talk about a class of application security issues I ran across a few times over the years. In particular, let’s discuss race conditions when it comes to files with sensitive content and permissions. Race conditions can allow an adversary to gain access to sensitive information on machines. Assume a system creates a file that contains sensitive information and afterwards applies permissions to lockdown that file. Understanding the race condition Let’s look at a practical example seen in the wild a few times.

These days business decisions and feature development often is influenced heavily by telemetry information. Telemetry is baked into the programs, services and applications we use. Companies are hungry for telemetry because with machine learning and Deep Neural Networks “data is the new oil”. Telemetry provides insights into how users use a particular system, what features they exercise, how they configure the system, what errors they trigger and what buttons they like clicking on.

Throughout my career I have been fascinated with quality assurance and testing, especially security testing and red teaming. One discussion that comes up frequently is how to measure the maturity of such programs and processes. My answer is straight forward as there are already existing frameworks that can be leveraged, adjusted and borrowed from to fit the needs of offensive security programs. You are likely familiar or have at least heard of the Capability Maturity Model Integration from Carnegie Mellon University.

In this post I will discuss some testing techniques for internal red teams to identify privacy issues in services and infrastructure, most importantly a simple three step approach that might uncover interesting results. Background story First, let me share a story from the past. When I did my master’s I built an app that performs end to end encryption of Facebook posts. This means that only the intended audience for which your posts were encrypted for can decipher the posts.