If you’ve ever waited endlessly for a report to load during a busy financial year or experienced sluggishness on an app due to heavy data loads, you’re not alone. Handling massive databases efficiently, especially under peak loads, is a challenge that even the biggest companies face. Zerodha, India’s largest stockbroker, came up with an elegant solution to this problem: DungBeetle.
Let’s dive into how this lightweight tool is solving real-world issues in the most ingenious way.
What’s the Problem ?
Think about a banking app or a stock trading platform. On regular days, it hums along just fine. But during peak times, like the end of a financial year, millions of users log in to generate their annual reports. This flood of requests can overwhelm the database, slowing things down for everyone and creating a frustrating user experience.
Even more challenging is scalability. How do you handle millions of simultaneous queries without spending a fortune upgrading your infrastructure?
Entry of DungBettle
DungBeetle is Zerodha’s answer to this conundrum. It’s a small but mighty tool that queues SQL queries, processes them asynchronously, and writes the results to a separate database.
Why is this clever? It means the main database doesn’t bear the brunt of these heavy tasks, so it stays fast and responsive for everyone else. Meanwhile, users still get their data quickly—just in a smarter way.
How does it work ?
Tasks
Developers define reusable queries, called tasks, in.sqlfiles. These tasks specify what kind of data to fetch.Jobs
When a user makes a request, DungBeetle turns it into a job. For example, if five users want the same report, five jobs are queued.Results Database
Once a job is processed, its results are written to a new table in a separate results database. Each job gets its own table, which makes retrieval super fast.Data Retrieval
The app simply checks the results database for completed jobs and fetches the data with a quickSELECT *. No delays, no stress.
Why It Works So Well ?
PostgreSQL to the Rescue
Zerodha chose PostgreSQL as the results database, and it’s been a surprising hero. Even with millions of temporary tables being created and deleted, PostgreSQL handles it without breaking a sweat.
Why use a new table for every job? Because it’s clean, efficient, and surprisingly cost-effective. Each user’s report is isolated, so there’s no chance of one job slowing down another.
Lightweight and Frugal
DungBeetle isn’t bloated software. It’s just 1,700 lines of Go code. It runs as a single binary, making it easy to deploy and scale. You can even set up different queues for heavy and light jobs, ensuring resources are used wisely.
What Makes DungBeetle Special?
It’s Distributed: Multiple instances share the load.
Asynchronous Processing: Queries don’t compete for resources.
Instant Retrieval: Results are stored in a dedicated table for each job, making fetching lightning-fast.
Affordable Scaling: No need to invest in massive infrastructure upgrades.
A Real-World Example
At Zerodha, millions of reports are generated daily. With DungBeetle, each report creates a new table in PostgreSQL, tailored specifically for the user who requested it.
This system allows Zerodha to:
Keep its main database free from overload.
Deliver reports faster, even during peak times.
Scale operations without massive costs.
Why DungBeetle Deserves a Spotlight?
DungBeetle might not sound glamorous, but it’s an incredibly practical solution to a common problem. By thinking outside the box and leveraging PostgreSQL in a unique way, Zerodha has created a system that’s lightweight, scalable, and cost-effective.
It’s proof that sometimes the best solutions aren’t about throwing more hardware or money at a problem—they’re about rethinking how the problem is tackled in the first place.
So the next time your banking app feels lightning-fast during a busy season, just remember: it might have a little DungBeetle behind the scenes, doing the heavy lifting.