Streamlining Security Investigations with Agents

Slack’s Security Engineering team is responsible for safeguarding the company’s core infrastructure and services. Their daily operations involve managing a security event ingestion pipeline that processes billions of events per day from a wide range of data sources. During on-call shifts, the team’s primary responsibility is to review alerts generated by the security detection system. To optimize their working efficiency and enhance Slack’s security defenses, the team has been experimenting with AI agents. This article explores how they are using AI to streamline security investigations, marking the beginning of a series that will delve into the design choices made and the lessons learned along the way.

The development process began in May 2025, when the team created a rudimentary prototype of what would eventually become their AI-driven security investigation tool. Initially, the prototype was little more than a 300-word prompt, which was divided into five sections: Orientation, Manifest, Methodology, Formatting, and Classification. The Orientation section introduced the AI agent as a security analyst tasked with investigating security alerts. The Manifest outlined the data sources the agent had access to, including Slack’s internal databases, logs, and external threat intelligence feeds. The Methodology section outlined the steps the agent should follow during an investigation, emphasizing the need to cross-reference evidence across different data sources. The Formatting section specified that the agent should produce a markdown report detailing the investigation, while the Classification section required the agent to choose a response classification from a predefined list.

To safely expose a subset of Slack’s data sources through the tool call interface, the team implemented a simple “stdio” mode MCP server. They repurposed a coding agent CLI as an execution environment for their prototype. However, the performance of the prototype was highly variable. At times, the AI agent produced excellent, insightful results, demonstrating an impressive ability to cross-reference evidence across different data sources. On other occasions, the agent would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, the team needed consistent performance and greater control over the investigation process.

To address these challenges, the team spent time refining the prompt. They stressed the importance of questioning assumptions, verifying data from multiple sources, and ensuring that the agent’s conclusions were well-supported. They also experimented with different ways to structure the prompt, including adding more detailed instructions and examples. Over time, these refinements helped improve the agent’s performance and reliability.

As the team continued to develop the AI-driven security investigation tool, they faced several design challenges. One of the key issues was balancing the need for flexibility and control with the complexity of the investigation process. The team wanted the AI agent to be able to adapt to different types of alerts and investigate them effectively, but they also needed to ensure that the agent followed a structured approach to avoid missing critical information.

Another challenge was integrating the AI agent with Slack’s existing security infrastructure. The team had to ensure that the agent could access the necessary data sources and that it could communicate with other security tools and systems. This required careful planning and coordination with other teams within Slack.

Despite these challenges, the team was able to make significant progress in developing the AI-driven security investigation tool. By using AI agents, Slack’s Security Engineering team has been able to streamline their investigations, improve efficiency, and enhance the overall security of the company’s infrastructure and services. As the team continues to refine the tool and expand its capabilities, they look forward to sharing more insights and lessons learned in future articles.