Change is coming for users of Apache Kafka, the leading distributed event-streaming platform. Apache ZooKeeper, Kafka's metadata management tool, will soon be phased out in favor of internal technology.
Colin McCabe, a member of the Apache Kafka project management committee and an engineer at Confluent, which leverages Kafka, explained the reason for the change. ZooKeeper stores persistent cluster metadata and handles maintaining dynamic configurations and topics, as well as partitions within the topics, he said. But ZooKeeper adds an extra layer of management. Storing metadata internally within Kafka will make managing it easier and enable better guarantees around issues such as versioning, according to McCabe.
Kafka Raft, or KRaft, a protocol for internally managed metadata, will replace ZooKeeper. In KRaft mode, Kafka metadata will be stored in a distributed log. Scalability will be the main benefit, McCabe said, but management will also improve. Kafka users will no longer need to deploy a separate system to manage their Kafka clusters.
It's unclear exactly when ZooKeeper will be decommissioned, but a vote on the timeline is expected in the next week or two. The current proposal is to declare KRaft generally available in the upcoming Kafka 3.3 release. ZooKeeper would be deprecated in the release after that, and removed in Kafka 4.0.
Targeted for August, Kafka 3.3 would include options for both ZooKeeper and KRaft. The end-of-life date for ZooKeeper is undetermined. “KRaft mode is moving to production soon, very soon this year,” McCabe said. “It’s going to be a big step forward for the project.”
Presently, Kafka developers are working on full-feature parity between KRaft and ZooKeeper, which is said to be closing in. KRaft mode actually has been available since Kafka 2.8, released in April 2021, but not in production-ready status; Kafka 3.3 will be the first production-ready release. For developers accustomed to ZooKeeper, McCabe does not anticipate a steep learning curve in moving to KRaft. The same APIs are supported for developers, but operators may have a few things to learn. “Our hope is that new administrators will actually find this easier and that existing administrators will find it manageable to transition.”
Confluent views ZooKeeper's deprecation as an important move for the Kafka community, said Jun Rao, Kafka's co-creator and co-founder of Confluent. “It makes deployment/operation much simpler and improves the scalability by a factor of 10 because of more efficient handling of metadata. We are happy to work with the community on this work and hope that it will become production-ready soon."
More than 80% of Fortune 100 companies use Kafka, according to the Apache Software Foundation. Accessible from the Apache Kafka website, Kafka is a distributed system consisting of clients and servers communicating via a high-performance TCP protocol, with deployment on virtual machines, bare-metal hardware, and containers on-premises or in cloud environments.