AWS has announced the general availability of Amazon MSK, a fully managed service for Apache Kafka. The solution would make it easy for developers to build and run “highly available, secured, and scalable” applications based on Apache Kafka without having to worry about managing the underlying infrastructure.
Customers would use Apache Kafka to capture and analyze real-time data streams from a range of sources, including IoT devices, website clickstreams, financial systems, and database logs. While many customers chose to self-manage their Apache Kafka clusters, they would often spend a lot of time and expense securing, scaling, patching, and ensuring high availability for Apache Kafka and Apache ZooKeeper clusters, which Apache Kafka depends on for resource management.
Amazon MSK would give customers all of the attributes of Apache Kafka combined with the “availability, security, and scalability” of AWS Cloud. With just a few clicks, customers can create Apache Kafka clusters designed for high availability that span multiple Availability Zones (AZs). Amazon MSK continuously monitors underlying server health, automatically replacing servers when they fail and patching servers in need of updates. Amazon MSK also runs highly available Apache ZooKeeper nodes at no additional cost.
Multi-Level Security
Amazon MSK provides multiple levels of security for Apache Kafka clusters, including VPC network isolation, AWS Identity and Access Management (IAM) for AWS Application Programming Interface (API) authorization, encryption at rest, encryption in-transit, TLS based certificate authentication, and authorization using Apache Kafka access control lists (ACLs).
Customers can scale out cluster storage with a few clicks in the AWS management console to meet changes in demand. Amazon MSK is fully compatible with Apache Kafka, enabling customers to “easily” migrate their on-premises or Amazon Elastic Cloud Compute (Amazon EC2) clusters to Amazon MSK with no code changes. There are no commitments or upfront investment required to run Amazon MSK, and customers only pay for the capacity they use.
“Customers who are running Apache Kafka have told us they want to spend less time managing infrastructure and more time building applications based on real-time streaming data,” said Rajesh Sheth, General Manager of Amazon MSK, AWS. “Amazon MSK gives these customers the ability to run Apache Kafka without having to worry about managing the underlying hardware, and it gives them an easy way to integrate their Apache Kafka applications with other AWS services. With Amazon MSK, customers can stand up Apache Kafka clusters in minutes instead of weeks, so they can spend more time focusing on the applications that impact their businesses.”
CTO of Compass
Compass is a real estate technology company with a powerful end-to-end platform that supports the entire buying and selling workflow. Compass uses Apache Kafka to provide its agents with quick access to fresh and accurate data from hundreds of real estate data sources nationwide.
“Streaming technology is an important part of what we do as a business, but managing streaming technology involves a lot of heavy lifting by our software developers,” said Joseph Sirosh, CTO of Compass. “Amazon MSK has allowed us to eliminate a lot of this heavy lifting. By using Amazon MSK we no longer need to spend effort on ensuring data durability, cluster availability, and scalability, and can now focus on building applications. Amazon MSK relieves our team from the burden of operating and maintaining Kafka and at the same time, we experience less downtime as our clusters are fully-managed.”
