Summary
Ayder is an advanced, HTTP-native event streaming platform designed to handle large-scale data processing with ease and efficiency. It operates as a single-binary system, eliminating the need for JVM, ZooKeeper, or complex client libraries. Ayder achieves remarkable durability and performance metrics through its use of a Raft consensus mechanism, ensuring data integrity across nodes. Its simplicity is highlighted in its quick setup and recovery times, significantly outperforming traditional systems like Kafka and Redis Streams in benchmarks. Ayder supports a variety of features including append-only logs, consumer groups, and built-in stream processing, making it a versatile tool for developers seeking powerful yet straightforward data management solutions.
Highlights:
- Single-binary with no external dependencies simplifies operation
- HTTP-native interface allows easy interaction using simple tools like curl
- Raft consensus for robust data durability and consistency
- Exceptional performance with low latency and high throughput
- Quick and reliable recovery from failures with minimal data loss
Ayder is a robust, HTTP-native durable event log and message bus system written in C, designed to provide high performance and reliability in data streaming. It operates as a single-binary event streaming system, where curl acts as the client, eliminating the need for JVM, ZooKeeper, or extensive client libraries. This simplicity enables rapid setup and deployment, with the entire system being operational in approximately 60 seconds. Ayder supports features such as consumer groups with committed offsets, high availability (HA) replication using a Raft consensus mechanism, and a key-value store with features like compare-and-swap and time-to-live settings.
In terms of performance, Ayder demonstrates superior metrics compared to traditional systems like Kafka and Redis Streams. It supports sustained messaging rates of approximately 50,000 messages per second with a client-side P99 latency of 3.46ms and a server-side P99.999 latency of 1.22ms. The system's design allows for rapid recovery following unplanned outages, with data recovery times of 40-50 seconds and zero data loss, significantly outperforming Kafka's recovery times. Ayder's performance has been validated under real network conditions and on various hardware including ARM64 platforms, ensuring reliability and efficiency.
Setting up Ayder involves minimal configuration, with support for Docker and standard compilation from source. It offers a comprehensive API for topic management, message production and consumption, and advanced stream processing capabilities like windowed joins and aggregations. Ayder's architecture facilitates easy scaling and maintenance, with no manual intervention required for tasks such as partition reassignment or in-sync replica management. This makes Ayder a practical solution for developers and enterprises looking for a modern, efficient, and easy-to-manage event streaming system.