The Guide to Apache Druid Architectures
As we build out the Rill Data team, we often encounter folks who are new to Apache Druid and looking for ways to get up to speed quickly. For this reason we maintain this “Guide to Apache Druid.” It’s meant to be a balanced list of articles, customer stories, and architectural diagrams that best helped us get up to speed answering questions like:
- Who uses Apache Druid?
- What are they using it for?
- How does it fit in with the other pieces of the Modern Data Stack?
This is a living document so when particularly relevant pieces come up we’ll update this page.
The Apache Druid site
It’s always best to hear directly from users exactly how they’re using Druid inside their company. While the community-maintained list of companies above is fairly exhaustive, these stories below are some of our favorites (in no particular order):
- Salesforce, 2020: Delivering High-Quality Insights Interactively Using Apache Druid at Salesforce
- Netflix, 2020: How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience
- Walmart, 2017: Event Stream Analytics at Walmart with Druid
- eBay, 2019: Monitoring at eBay with Druid
- AirBnB, 2018: How Druid enables analytics at Airbnb – Airbnb Engineering & Data Science – Medium
- Lyft, 2018: Data modeling tradeoffs with Druid
- Lyft, 2018: Streaming SQL and Druid at Lyft
- Snap, 2018: Druid at Snap Meetup presentation from Charles Allen, formerly of Metamarkets
- Pinterest, 2020: Powering Pinterest ad analytics with Apache Druid
- Pinterest, 2019: Druid at Pinterest
- Naver, 2018: Web analytics at scale
What is the internal architecture of Apache Druid and how is it different from other OLAP databases?
- Apache Druid (part 1): A Scalable Timeseries OLAP Database System
- An introduction to Druid, your Interactive Analytics at (big) Scale (This is one of the best reviews)
- Comparison of the Open Source OLAP Systems for Big Data: ClickHouse, Druid and Pinot
- Druid | Introducing Druid: Real-Time Analytics at a Billion Rows Per Second
- Druid: A Real-Time Analytical Data Store
- The challenges of running Druid at large scale, and future directions, part 1
- The challenges of running Druid at large scale, and future directions, part 2
- The anatomy of a Druid segment file – Engineers @ Optimizely – Medium
Reference Architecture Diagrams
If a picture is worth 1000 words, an architecture slide is worth a 1000 lines of code. One of the most valuable learnings I take away from online talks, presentations, and blogs are the architecture slides from the leading pioneers of real-time data infrastructure. What technologies have Netflix, AirBnb, Lyft, Pinterest, and Snap used to assemble data stacks to process, store, and act upon the massive amounts of signal streaming from their platforms? How do batch and real-time systems interoperate in practice? What databases are being used? Which visualization tools?
Since founding Rill, I began collecting architecture diagrams from the leading companies that have adopted real-time data stacks. I focused on Apache Druid because that's the technology we're building on at Rill, but these insights hold true for other real-time databases like Clickhouse and Pinot. I intentionally chose slides from content that was not sponsored or associated with any commercial vendor (including Rill), so you can trust that is engineers telling their stories.
While "monoDBism" is a compelling philosophy in theory, and in a vendor's interest, evidence from these leading companies shows "polyDBism" is more widely practiced. Real-time databases have a complementary role to play alongside data lakes, warehouses, key-value stores, and graph DBs.
I hope you enjoy reading this tour through reference architectures for real-time data stacks "feat. Apache Druid", as much as I enjoyed collecting and redrawing these.
Did we miss something great?
We're always on the lookout for smart write-ups. Let us know if you found something that we overlooked.