Helios: hyperscale indexing for the cloud & edge (part II)

Helios: Hyperscale Indexing for the Cloud & Edge (Part II)

In our previous article, we explored the motivations behind the development of Helios, a new reference blueprint for large-scale data processing. Today, we delve deeper into the intricacies of Helios itself, a distributed, highly-scalable system utilized by Microsoft for flexible ingestion, indexing, and aggregation of massive streams of real-time data. Designed to integrate seamlessly with relational engines, Helios processes nearly a quadrillion events daily, indexing approximately 16 trillion search keys from hundreds of thousands of machines across tens of data centers worldwide.

At its core, Helios separates the ingestion and indexing processes, introducing a novel bottoms-up index construction algorithm. This approach allows the system to efficiently manage the vast amounts of data it handles. Helios exposes tables and secondary indices for use by relational query engines through standard access path selection mechanisms during query optimization. This design ensures that Helios can be easily integrated into existing data processing pipelines, providing a flexible and scalable solution for handling large-scale real-time data streams.

One of Helios' primary features is its ability to move computation to the edge, enabling efficient data processing closer to the source. This capability is particularly valuable in scenarios where latency is a critical factor, such as in real-time analytics or IoT applications. By leveraging edge computing, Helios can deliver faster response times and reduce the burden on centralized data centers, enhancing overall system performance and reliability.

Helios is designed to handle large streams of real-time data, with a capacity to process tens of petabytes a day. A prime example of its application is the log data generated by Azure Cosmos, a globally distributed database service provided by Microsoft. The system supports a variety of key use cases, including searching for records related to specific attributes (such as incident support), conducting impact and drill-down analyses, and monitoring and reporting performance metrics.

One particularly interesting use case for Helios is in supporting GDPR right to be forgotten requests. In such cases, it becomes necessary to search through tens of billions of streams to identify and remove any data containing a user's information. Helios' scalability and efficient indexing capabilities make it well-suited to tackle such complex and time-sensitive tasks.

Incoming streams processed by Helios can have data rates as high as 4TB per minute, with multiple columns requiring indexing (seven or more columns is common). Additionally, the system must handle high cardinality data, which poses unique challenges for indexing and query optimization. To address these challenges, Helios employs a stream table definition based on a loose schema that specifies the sources to be monitored and the indices to be created.

For instance, a CREATE STREAM statement might define a source list and the indices to be created. Based on this definition, Helios generates the necessary infrastructure to process and index the incoming data streams. This flexible schema design allows Helios to adapt to diverse data sources and indexing requirements, ensuring that it can be applied to a wide range of use cases and applications.

In conclusion, Helios represents a significant advancement in hyperscale indexing for the cloud and edge. Its ability to efficiently ingest, index, and aggregate massive streams of real-time data, coupled with its capacity to leverage edge computing, positions it as a powerful tool for organizations dealing with large-scale data processing needs. As we continue to explore the capabilities of Helios, it becomes clear that this system not only addresses current challenges but also sets a new benchmark for future large-scale data processing solutions.