How Meta Used AI to Map Tribal Knowledge in Large-Scale Data Pipelines

At Meta, we've long recognized the potential of AI to enhance our development processes, but we quickly encountered a challenge: AI coding assistants are only as effective as their understanding of the codebase. When we directed AI agents to one of our large-scale data processing pipelines, which spanned four repositories, three languages, and over 4,100 files, we found that they were unable to make useful edits quickly enough. This was a significant problem, as our pipeline is config-as-code, involving Python configurations, C++ services, and Hack automation scripts that work together across multiple repositories. A single data field onboarding task touches configuration registries, routing logic, DAG composition, validation rules, C++ code generation, and automation scripts, all of which must stay in sync.

We had already built AI-powered systems for operational tasks, such as scanning dashboards, pattern-matching against historical incidents, and suggesting mitigations. However, when we tried to extend these capabilities to development tasks, the AI struggled to navigate the complex and fragmented codebase. It lacked a "map" that would allow it to understand the relationships between different components and make informed edits.

To address this issue, we developed a pre-compute engine: a swarm of 50+ specialized AI agents that systematically read every file in the pipeline. These agents produced 59 concise context files that encoded the "tribal knowledge" previously held only in engineers' heads. This tribal knowledge included not just the code itself but also the underlying design choices and relationships that were not immediately apparent.

The result was transformative. AI agents now have structured navigation guides for 100% of our code modules, up from just 5%. They can now cover all 4,100+ files across three repositories. Furthermore, we documented over 50 "non-obvious patterns," or underlying design choices and relationships that were not immediately apparent from the code. Preliminary tests have shown that AI agent tool calls per task have decreased by 40%.

The system is model-agnostic, meaning it works with most leading models. The knowledge layer it provides is not tied to any specific AI architecture, ensuring compatibility and flexibility. Additionally, the system maintains itself. Every few weeks, automated jobs periodically validate file paths, detect coverage gaps, re-run quality critics, and auto-fix stale references. This self-sustaining infrastructure ensures that the AI remains effective and up-to-date as the codebase evolves.

The AI isn't just a consumer of this infrastructure; it's the engine that runs it. By mapping the tribal knowledge of our large-scale data processing pipelines, we've enabled AI agents to make more informed decisions and contribute more effectively to our development processes. This breakthrough not only improves the speed and accuracy of our code edits but also ensures that our AI systems remain adaptable and reliable as our codebase continues to grow and change.

In essence, the challenge of navigating a complex, multi-repository codebase has been transformed into an opportunity for AI to thrive. By building a pre-compute engine that systematically maps tribal knowledge, we've created a foundation for more efficient and intelligent development processes at Meta. This approach not only benefits our own teams but also serves as a blueprint for other organizations looking to harness the power of AI in their codebases.