Deploy Safety: Reducing customer impact from change

In mid-2023, Slack identified significant opportunities to enhance its reliability, a critical factor as the platform became increasingly mission-critical for customers. This realization marked the beginning of a transformative journey that would culminate in a dramatic reduction of customer impact hours by January 2025. The Deploy Safety Program, launched in 2023, aimed to improve deployment processes, strengthen safety culture, and maintain the pace of change necessary to meet business needs.

The problem Slack faced was multifaceted. As businesses evolved, so too did their system requirements, driven by changing customer expectations, load, and scale. Two critical trends emerged from the analysis: first, Slack's growing importance to customers heightened expectations for product reliability and specific feature availability. Second, the majority of customer-facing incidents (73%) were triggered by Slack-induced change, particularly code deployies.

Slack's incident management process was already robust, enabling rapid responses to issues of varying impact and severity. However, a significant portion of these incidents resulted in customer disruption, with impact varying depending on the specific features affected. Change-triggered incidents occurred across a wide range of systems and deployment processes, further complicating the challenge.

Additionally, customer feedback highlighted that interruptions became increasingly disruptive after about 10 minutes, a threshold they referred to as a "blip." This trend was expected to worsen until the introduction of Agentforce in 2025. The situation was compounded by Slack's complex software engineering environment, which included hundreds of internal services and diverse deployment systems and practices.

Prior to the Deploy Safety Program, Slack's reliability efforts often focused on individual deploy systems or services. This approach led to manual change processes that stifled innovation, consumed excessive time and effort, and negatively impacted engineering morale. The need for a more holistic and efficient strategy became apparent as the company sought to balance rapid innovation with the reliability required to meet customer expectations.

The Deploy Safety Program addressed these challenges by adopting a systemic approach to deployment. It introduced standardized practices, automated testing, and rigorous pre-deployment checks to minimize the risk of customer-impacting incidents. Additionally, the program emphasized a culture of safety, encouraging engineers to prioritize reliability and collaborate across teams to identify and mitigate potential issues.

By January 2025, the results of the Deploy Safety Program were remarkable. Customer impact hours had been reduced by 90% from their peak, with the trend continuing downward. This achievement was not only a testament to the effectiveness of the program but also a reflection of Slack's commitment to innovation and customer satisfaction.

The Deploy Safety Program's success underscored the importance of balancing agility with reliability in a fast-paced technological environment. By prioritizing safety and fostering a culture of collaboration, Slack was able to maintain its pace of change while ensuring that its customers experienced minimal disruption. This approach not only enhanced Slack's reputation as a reliable platform but also reinforced its position as a leader in the industry.

Looking ahead, Slack's Deploy Safety Program continues to evolve, adapting to new challenges and opportunities. As the company remains committed to innovation and reliability, the program serves as a blueprint for other organizations seeking to navigate the complexities of rapid change while prioritizing customer impact. The journey from mid-2023 to January 2025 represents a significant milestone in Slack's history, demonstrating the power of a unified, safety-first approach to deployment and the long-term benefits it can bring to both the company and its customers.