Embrace The Red

Today we have another post about OpenHands from All Hands AI. It is a popular agent, initially named “OpenDevin”, and recently the company also provides a cloud-based service. Which is all pretty cool and exciting. Prompt Injection to Full System Compromise However, as you know, LLM powered apps and agents are vulnerable to prompt injection. That also applies to OpenHands and it can be hijacked by untrusted data, e.g. from a website.

Another day, another AI data exfiltration exploit. Today we talk about OpenHands, formerly referred to as OpenDevin. It’s created by All-Hands AI. The OpenHands agent renders images in chat, which enables zero-click data exfiltration. Simon Willison recently gave this data exfiltration attack pattern a great name: Lethal Trifecta. We discuss this specific image based attack technique frequently. Sometimes a message must be repeated multiple times to raise awareness and become mainstream knowledge.

Today let’s explore Devin’s system prompt a bit more. Specifically, an interesing tool that I discovered when reading through it. Hidden in Devin’s capabilities is a tool that can open any local port to the public Internet. That means, with the right indirect prompt injection nudge, Devin can be tricked into publishing sensitive files or services for anyone to access. This tunneling feature can be invoked without a human in the loop.

In this post we show how an attacker can make Devin send sensitive information to third-party servers, via multiple means. This post assumes that you read the first post about Devin as well. But here is a quick recap: During an indirect prompt injection Devin can be tricked into download malware and extract sensitive information on the machine. But there is more… Let’s explore how Devin can leak sensitive information and send it to a third-party server.

Today we cover Devin AI from Cognition, the first AI Software Engineer. We will cover Devin proof-of-concept exploits in multiple posts over the next few days. In this first post, we show how a prompt injection payload hosted on a website leads to a full compromise of Devin’s DevBox. GitHub Issue To Remote Code Execution By planting instructions on a website or GitHub issue that Devin processes, it can be tricked to download malware and launch it.

Sandbox-escape-style attacks can happen when an AI is able to modify its own configuration settings, such as by writing to configuration files. That was the case with Amp, an agentic coding tool built by Sourcegraph. The AI coding agent could update its own configuration and: Allowlist bash commands or Add a malicious MCP server on the fly to run arbitrary code This could have been exploited by the model itself, or during an indirect prompt injection attack as we will demonstrate in this post.

Cursor is a popular AI code editor. In this post I want to share how I found an interesting data exfiltration issue, the demo exploits built and how it got fixed. When using Cursor I noticed that it can render Mermaid diagrams. Cursor Renders Mermaid Diagrams If you are not familiar with Mermaid, it has a simple syntax: graph TD User --> Computer This will create a diagram as follows:

A few months ago I was looking at the filesystem MCP server from Anthropic. The server allows to give an AI, like Claude Desktop, access to the local filesystem to read files or edit them and so forth. I was curious about access control and in the documentation there is a configuration setting to set allowedDirectories, which the AI should be allowed access to: As you can see the example shows two folders being allowlisted for access.

Today we cover ChatGPT Codex as part of the Month of AI Bugs series. ChatGPT Codex is a cloud-based software engineering agent that answers codebase questions, executes code, and drafts pull requests. In particular, this post will demonstrate how Codex is vulnerable to prompt injection, and how the use of the “Common Dependencies Allowlist” for Internet access enables an attacker to recruit ChatGPT Codex into a malware botnet. The ZombAI attack arrives at ChatGPT Codex today!

In this post we demonstrate how a bypass in OpenAI’s “safe URL” rendering feature allows ChatGPT to send personal information to a third-party server. This can be exploited by an adversary via a prompt injection via untrusted data. If you process untrusted content, like summarizing a website, or analyze a pdf document, the author of that document can exfiltrate any information present in the prompt context, including your past chat history.

This year I spent a lot of time reviewing, exploiting and working with vendors to fix vulnerabilities in agentic AI systems. As a result, I’m excited to announce the Month of AI Bugs 2025! Goal Of The Initiative The main purpose of the Month of AI Bugs is to raise awareness about novel security vulnerabilities in agentic systems, primarily focusing on AI coding agents. Posts will cover both simple and advanced, sometimes even mind-boggling exploits.

This is a security advisory for a data leakage and exfiltration vulnerability in a popular, but now deprecated and unmaintained, Slack MCP Server from Anthropic. If you are using this MCP server, or run an “MCP Store” that hosts it, it is advised that you analyze how this threat applies to your use case and apply a patch as needed. Anthropic’s Slack MCP Server When Anthropic introduced MCP they published reference server implementations on Github.

When the Model Context Protocol (MCP) came out it reminded me of the Common Object Model (COM) from Microsoft. COM has been around for decades and it’s used for programming, scripting, sharing of functionality at a binary/object level across languages and hosts. Via DCOM all of this can even be done remotely, and well, it’s also useful for red teaming. A lot of software on Windows was/is implemented as COM objects, including Microsoft Office.

Today we are going to discuss how real-world tactics, techniques, and procedures (TTPs) apply to computer-use systems, specifically, we’ll look at ClickFix attacks. This demo was part of my presentation at the SAGAI Workshop on May 15th, 2025 in San Francisco. It was a great workshop, with tons of interesting insights and discussions. So, let’s talk about ClickFix, and how it applies to AI systems! What is ClickFix? ClickFix is a social engineering technique that is being used by adversaries.

Recently OpenAI added an additional memory feature called “chat history”, which allows ChatGPT to reference past conversations. The details of the implementation are not known. The documentation highlights that: “It uses this to learn about your interests and preferences, helping make future chats more personalized and relevant.” I decided to spend some time to figure out how it works. Update: Video Tutorial Added Based on the interest in this post, I’ve also created a video tutorial.

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools. I have been calling it Model Control Protocol at times, because due to prompt injection, MCP tool servers control the client basically. This post will explain in detail why that is, and I will also share a novel exploit chain. Why MCP - How Is It Different? The main difference to other tool invocation setups, like OpenAPI is that MCP is dynamic.

GitHub Copilot has the capability to be augmented with custom instructions coming from the current repo, via the .github/copilot-instructions.md file. Pillar Security recently highlighted the risks associated with rules files. Their post discusses custom Cursor rules in ./cursor/rules ending in .mdc. If you watch the demos, you’ll notice that they also have a GitHub Copilot demo which uses the GitHub specific copilot-instructions.md file. Update: May 1, 2025 GitHub made a product change and is now highlighting invisible Unicode characters in the Web UI.

You are likely aware of ASCII Smuggling via Unicode Tags. It is unique and fascinating because many LLMs inherently interpret these as instructions when delivered as hidden prompt injection, and LLMs can also emit them. Then, a few weeks ago, a post on Hacker News demonstrated how Variant Selectors can be used to smuggle data. This inspired me to take this further and build Sneaky Bits, where we can encode any Unicode character (or sequence of bytes for that matter) with the usage of only two invisible characters.

ChatGPT Operator is a research preview agent from OpenAI that lets ChatGPT use a web browser. It uses vision and reasoning abilities to complete tasks like researching topics, booking travel, ordering groceries, or as this post will show, steal your data! Currently, it’s only available for ChatGPT Pro users. I decided to invest $200 for one month to try it out. Risks and Threats OpenAI highlights three risk categories in their Operator System Card:

Imagine your AI rewriting your personal history… A while ago Google added memories to Gemini. Memories allow Gemini to store user-related data across sessions, storing information in long-term memory. The feature is only available to users who subscribe to Gemini Advanced so far. So, in the fall of last year I chimed in and paid for the subscription for a month to check it out. As a user you can see what Gemini stored about you at https://gemini.