The Guide to Apache Druid Architectures

Michael Driscoll

Author

October 4, 2021

Date

As we build out the Rill Data team, we often encounter folks who are new to Apache Druid and looking for ways to get up to speed quickly. For this reason we maintain this “Guide to Apache Druid.” It’s meant to be a balanced list of articles, customer stories, and architectural diagrams that best helped us get up to speed answering questions like:

Who uses Apache Druid?
What are they using it for?
How does it fit in with the other pieces of the Modern Data Stack?

This is a living document so when particularly relevant pieces come up we’ll update this page.

‍

The Apache Druid site

The open source Apache Druid project site itself is always a good place to start learning about Druid, and in particular, the exhaustive list of companies using Druid.

‍

Customer Stories

It’s always best to hear directly from users exactly how they’re using Druid inside their company. While the community-maintained list of companies above is fairly exhaustive, these stories below are some of our favorites (in no particular order):

Salesforce, 2020: Delivering High-Quality Insights Interactively Using Apache Druid at Salesforce
Netflix, 2020: How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience
Walmart, 2017: Event Stream Analytics at Walmart with Druid
eBay, 2019: Monitoring at eBay with Druid
AirBnB, 2018: How Druid enables analytics at Airbnb – Airbnb Engineering & Data Science – Medium
Lyft, 2018: Data modeling tradeoffs with Druid
Lyft, 2018: Streaming SQL and Druid at Lyft
Snap, 2018: Druid at Snap Meetup presentation from Charles Allen, formerly of Metamarkets
Pinterest, 2020: Powering Pinterest ad analytics with Apache Druid
Pinterest, 2019: Druid at Pinterest
Naver, 2018: Web analytics at scale

‍

Druid Architecture

What is the internal architecture of Apache Druid and how is it different from other OLAP databases?

‍

Reference Architecture Diagrams

If a picture is worth 1000 words, an architecture slide is worth a 1000 lines of code. One of the most valuable learnings I take away from online talks, presentations, and blogs are the architecture slides from the leading pioneers of real-time data infrastructure. What technologies have Netflix, AirBnb, Lyft, Pinterest, and Snap used to assemble data stacks to process, store, and act upon the massive amounts of signal streaming from their platforms? How do batch and real-time systems interoperate in practice? What databases are being used? Which visualization tools?

Since founding Rill, I began collecting architecture diagrams from the leading companies that have adopted real-time data stacks. I focused on Apache Druid because that's the technology we're building on at Rill, but these insights hold true for other real-time databases like Clickhouse and Pinot. I intentionally chose slides from content that was not sponsored or associated with any commercial vendor (including Rill), so you can trust that is engineers telling their stories.

While "monoDBism" is a compelling philosophy in theory, and in a vendor's interest, evidence from these leading companies shows "polyDBism" is more widely practiced. Real-time databases have a complementary role to play alongside data lakes, warehouses, key-value stores, and graph DBs.

I hope you enjoy reading this tour through reference architectures for real-time data stacks "feat. Apache Druid", as much as I enjoyed collecting and redrawing these.

‍