Case Study
Cloud & DevOps
Data Engg & Analytics

How Velotio Built an Asynchronous Messaging Solution by Deploying “Kafka-as-a-service” for a Premier Global Media & Entertainment Company

About This Project

The client’s primary requirement was to enable asynchronous communication between their existing messaging platform services. Setting up this communication was also required by their Observability team, which was building an in-house solution for monitoring and alerting. 

Velotio team compared various asynchronous solutions available in the market and narrowed down on using Kafka for the client use case. Moreover, different Kafka offerings were compared wrt features, performance, and pricing.

Velotio team designed and developed the “Kafka-as-a-service” (KaaS) platform for the client. This extensively built platform, with modernized infrastructure, offers interservice Communication with 80+ Services, 150 Topics, and 340M messages per day.

Services

Cloud & DevOps
Data Engg & Analytics

Technologies

About the Client

Our client, a premier global media and entertainment company, offers audiences the world’s most differentiated and complete portfolio of content, brands, and franchises across television, film, streaming, and gaming. The client’s existing customers include premium entertainment, sports, news assets, leading non-fiction and international entertainment, and sports businesses.

Understanding the Challenge

Our client used an existing messaging platform that offered minimal to zero asynchronous communication between services.

The existing app was built on a manual, non-serviceable architecture. Due to this, simple tasks like creating topics became a pain area. Over time, it became susceptible to downtime risks.   

The client reviewed the issue and recorded the following performance concerns that needed to be addressed immediately:

  • The platform required a redesign/release from scratch with a self-serviceable and scalable infrastructure.
  • The Observability team intended to use Kafka primarily to design a perfect HA solution.
  • The existing platform required to be migrated to a fully automated KaaS platform. So that the teams could simply write a yaml file with details like the topic name and list of publishers and consumers, and the KaaS platform automatically creates the topics.
  • The crucial part was boosting the platform's overall performance of slow-running processes.  

The client needed a strategic engineering partner who could effectively brainstorm a practical roadmap with their observability team and technologically lead them to a better platform with faster release cycles. 

“Kudos to team Velotio for the PROD Confluent set-up. It’s great to see how everyone on the team contributes to building the platform and unlocking services teams.”

CTO, Global Media and Entertainment Company

How We Made It Happen

Global media and entertainment joined hands with Velotio as its strategic engineering partner because of our expertise in Kafka Cluster in production at a large scale.

However, a big challenge was the critical timeline of the project. The client needed to develop and migrate to the new platform in less than 2 months. Velotio took on the challenge of automating the existing platform to spin up the infrastructure within the expected timeline.

Velotio team deployed an engineering team of 1 Tech Lead, TPM (Technical Project Manager), and 3 Engineers to start working on the project in the mid of September 2021.

Our team designed and built 2 KaaS platforms for the client:

  • One is to address the business data communication the service teams use.
  • The other KaaS platform set up for the Observability team was used to send metrics from logs and client telemetric data. This Kafka addresses a scale of 1 GB/s and peaks during major streaming events.

Our strategic approach and agile methodologies helped the client build an all-new platform to enhance performance. The team worked in two phases.

Phase 1: Managing the existing Kafka Clusters

  • Our team helped the client set up Kafka environments in the development and integration environment with the existing vendor AWS MSK.
  • Kafka would be the messaging bus that every service would use.
  • The service teams wanted the self-service feature of Creating topics and configuring the ACLs. The service teams would have to use this Kafka-as-a-Service platform for their async messaging requirements.
  • Velotio team built HA Kafka clusters on AWS with mirroring and cross-account replication. The setup was done using Infrastructure as Code (IaC) using Terraform and deployed on top of the Kubernetes cluster. Helm charts, Jenkins, GitHub Actions, and Flux, were used in the automation.
  • The Kafka clusters have 150 topics and are used by over 80 applications that are producing 340M events per day on average.

Phase 2: Expanding the Kafka Usage for inter-service communication as well as operational data load

  • In this phase, we could choose from multiple options for Kafka vendors.
  • Therefore, detailed research was done regarding features, cost, etc. After brainstorming for around 2 months, we finally chose - Confluent-based Kafka. This choice was going to impact their global team of 1000s of engineers.
  • Our team worked on improving the overall Kafka architecture. Build an all-new long-term Kafka Platform, using Confluent and migrating all the development/integration to Confluent, which was previously built on AWS MSK.
  • The team worked on all the automation required to spin up the infrastructure. It made all the teams get started with Kafka. It took 15 mins to get onboarded on Kafka.
  • In this phase, the team also set up a staging/production environment and a global and regional strategy for Kafka data replication.

How Velotio Made a Difference

Successfully delivered a fully automated and self-serviceable KaaS platform.

The all-new, extensively built platform offers interservice Communication with 80+ Services, 150 Topics, and 340M messages per day.

The platform has been enabled with an automated healing process, which eliminates dependency on support engineers to work 24x7.

With Velotio, achieve breakthroughs in your product development journey.

Over 90 global customers, including NASDAQ-listed enterprises, unicorn startups, and cutting-edge product companies have trusted us for our technology expertise to deliver delightful digital products.

Talk to us

Work with modern and scalable technologies

We leverage emerging technologies to build products that are designed for scalability and better usability.

Rated 4.6/5 on Clutch

325+ highly skilled engineers

With us as your tech partners, you get access to a pool of digital strategists, engineers, architects, project managers, UI/UX designers, Cloud & DevOps experts, product analysts and QA managers.

At Velotio, we hold ourselves to sky-high standards of excellence and expect the same from our customers.