1 Million Notifications in 60 Seconds : How We Engineered A Processing Engine

In this talk, we will share how we overcame multiple hurdles to be able to process at least 1 million notifications a minute. We encountered job processing delays, low throughput, and poor performance in notification delivery. As we go through this talk, we want to show you how we broke through these obstacles and how you can apply what we did to your job process system.

We will discuss the "Queueing theory" for large-scale job processing, how to ensure jobs are persisted in Redis, and worker isolation for dynamically scaling job processing. During this process, we rigorously tested our system with realistic scenarios and figured out meaningful metrics to ensure we had insights into our processing system.

We are very proud of this achievement, and we want to share our steps and insights with you as we set the future for notification development.

Key Takeaways:
  • What is the foundation to scale job processing to be able to handle a large amount of events?
  • Consider the pros and cons of using redis as a queue system.
  • Results of making thoughtful, contentious, and consistent improvements to your system.
  • Best practices to design a job processing system.Your photo: Attached to the email.
  • Difficulty Middle
  • Speech type Standart 40min

Zac Clifton
Systems and Application Architect at Novu, Israel
A Software Architect that specializes in high scale job processing, Zac is a developer, architect, and leader that thoroughly enjoys helping people solve their problems.

He works for Novu as their Systems and Application Architect where he leads the company in building and maintaining large scale infrastructure and deployments to serve all people that need to be notified.

Besides that, Zac is a speaker, and loves to contribute to the community by giving talks, training, writing articles or participating in open source projects.
Participation is necessary
But closed. Sold out already. See you next year.

Speakers / Agenda

The program is full of amazing talks about people, code, trends, and people again. All talks undergo a сareful selection process by our program committee. We don't accept marketing talks and don’t sell slots to sponsors. Only meaningful content, only hardcore.