Inside Gen 13: how we built our most powerful server yet

Cloudflare's Gen 13 servers represent a significant leap forward in server technology, designed to meet the growing demands of the internet. These powerful machines are equipped with AMD EPYC™ Turin 9965 processors, 100 GbE networking, and a range of other advanced components. In this article, we delve into the engineering decisions that led to the creation of Gen 13, exploring the rationale behind each major component selection.

The journey towards Gen 13 began with a shift in Cloudflare's software stack. A few months ago, the company announced the transition to FL2, a Rust-based rewrite of its core request handling layer. This migration accelerated Cloudflare's ability to innovate and improve the internet, and it necessitated a refresh of the server hardware to align with the evolving demands of the network and software stack.

At the heart of Gen 13 is the AMD EPYC™ Turin 9965 processor, a 192-core marvel. This processor was chosen for its impressive performance and efficiency. Cloudflare evaluated several options, including the AMD EPYC™ 9965 Turin and the AMD EPYC™ 9684X Genoa-X 96-core processor. The Turin's superior core count and advanced architecture provided the necessary boost in throughput and efficiency, making it the ideal choice for Gen 13.

Memory capacity is another critical component of Gen 13. The servers are equipped with 768GB of DDR5-6400 memory, spread across twelve memory channels. This significant increase in memory capacity compared to the previous generation's 384GB of DDR5-4800 memory was driven by the need to support the growing demands of Cloudflare's network and software. The higher memory bandwidth and capacity enhance the server's ability to handle complex tasks and maintain performance under heavy load.

Storage in Gen 13 has also been upgraded. The servers feature x3 E1.S NVMe Samsung PM9D3a 7.68TB or Micron 7600 Pro 7.68TB drives, providing a substantial increase in storage capacity. This upgrade, along with the 1.5x storage capacity improvement, ensures that Gen 13 can efficiently manage the vast amounts of data it processes.

Networking is a key area where Gen 13 makes a significant leap forward. The servers are equipped with dual 100 GbE OCP 3.0 Intel Ethernet Network Adapter E830-CDA2 or NVIDIA Mellanox ConnectX-6 Dx. This transition from 25 GbE to 100 GbE networking provides a 4x increase in network bandwidth, enabling Gen 13 to handle the growing traffic demands of the internet. The dual 100 GbE ports also offer improved redundancy and reliability.

Gen 13 introduces PCIe encryption hardware support in addition to memory encryption. This feature enhances the security of the servers, ensuring that data is protected at every level. The inclusion of PCIe encryption hardware was a deliberate choice to address the growing concerns around data security and privacy.

The Gen 13 design also supports thermally demanding powerful drop-in PCIe accelerators. This feature allows Cloudflare to easily integrate additional hardware, such as graphics processing units (GPUs) or field-programmable gate arrays (FPGAs), to enhance performance and flexibility.

The system management components of Gen 13 include DC-SCM 2.0 ASPEED AST2600 (BMC) and AST1060 (HRoT). These components provide robust and efficient management of the servers, ensuring optimal performance and ease of maintenance.

Power supply is another area where Gen 13 has been optimized. The servers feature a 1300W power supply, built to withstand the demands of the high-performance components. The use of Titanium Grade 800W power supplies ensures reliability and longevity.

In conclusion, Cloudflare's Gen 13 servers represent a significant advancement in server technology. The combination of AMD EPYC™ Turin 9965 processors, 100 GbE networking, and other advanced components positions Gen 13 as one of the most powerful and efficient servers in the industry. The engineering decisions behind each major component selection reflect Cloudflare's commitment to innovation and its dedication to meeting the growing demands of the internet. With Gen 13, Cloudflare continues to push the boundaries of what is possible in server design, ensuring that it remains at the forefront of the internet's infrastructure.