The Redis Paradox: my DelhiFOSS 2.0 talk at IIT Delhi, written up

Last Saturday I gave a talk called “The Redis Paradox: Beyond the Cache Horizon!” at DelhiFOSS 2.0, over at IIT Delhi. I had the first session of the day, in the 10:15 AM slot. It was a wonderful event with such a good turnout, and I absolutely loved the energy of the audience.

As promised, the slide deck is public and the demo code is up on GitHub. But slides without the person talking over them only tell half the story, so here’s the written version for folks who couldn’t make it (and for the ones who did but were still waking up at 10:15 AM on a Saturday :P).

And here is the whole deck, if you want to click through it first:

The cache-aside slide at DelhiFOSS 2.0. The clock on the wall says 10:47, so we were half an hour in and still on caching.

How I ended up giving this talk

This one had been building up for a while. Back in February 2023 I replied to a thread on X saying there’s still a misperception among a lot of devs that Redis is “just a cache”, mostly because it’s so popular as one. With RDB and AOF persistence turned on, Redis can also be a quick, throwaway, persistent NoSQL database for an app with low scale.

Then one night in August 2023 I sat down and explored the redis codebase on my ultra-wide monitor.

August 2023: the README, the hellodict module example, expire.c and server.c open side by side.

In December I spoke at a FOSS United Delhi meetup, and in February 2024 this talk got selected for DelhiFOSS 2.0, with a promise to unveil the lesser-known capabilities of Redis, far beyond its typical use as a cache.

The speaker card FOSS United made for the talk.

The night before, I tweeted “So, it’s time to build the slides! :P”. Just kidding, I was all set. 🧑‍💻

$ whoami

I opened with the usual intro: a career timeline from intern to principal engineer (it really started with Logo turtle programming when I was 10), and the line I use to describe my job: “I write words that control computers to tell other computers to build fake computers that run on different computers.” The speaker note under it says I don’t really know what I’m doing, but it seems to be working so far.

How I describe my job, and the road to it.

The next slide was life outside work: too much homelab tinkering on weekends, journaling (hence the fountain pens and three mechanical keyboards), chess again after a decade off, photography and motorbiking. Fun fact from my speaker notes: for someone who writes this much, my handwriting is terrible. I’m a southpaw, if that’s an excuse.

The typical view vs the paradox

I started with a show of hands: what is Redis?

My next slide already had the typical answers: caching, in-memory, fast, ephemeral, open source, simple. All true. The slide after that listed the paradox, the same tool doing geospatial, transactions, modules, clustering, sharding, HyperLogLog and streams.

The typical view, and the paradox.

The promise I made to the room was that everyone would walk out with a new appreciation for how versatile Redis is.

The system design interview version

A lot of times when I’m taking system design interviews, the design goes the same way. Clients talk to a server, the server talks to a database. Then we scale up and someone asks about concurrency, the database starts taking locks, catches fire, and everyone has a bad day.

How every design starts, and how it ends once the traffic shows up.

And then, boom, a box labelled Redis appears between the server and the database. First check if the response is in Redis. If yes, return it from the cache. If not, query the DB, store the result in the cache and return it. That’s the slide in the photo above.

Everyone reaches for the same button.

Good! But what else? 🤔

“Can’t we just use a hashmap?”

This is where I asked the room what’s special about Redis here. Can’t we get the same thing with a simple hashmap or dictionary in the app server?

The question: why not a plain hashmap inside the app server?

Short answer: no. Long answer: also no.

I didn’t give the reasoning away on stage. I told everyone to work it out and slide into my DMs, because it’s a genuinely interesting thought experiment. Hint: think about more than one app server, a process restart, and two requests updating the same key at once.

A short history of caching

Before getting to Redis itself we took a quick trip back in time, with a Back to the Future slide (fun fact: Rick and Morty is based on it).

• 1960s: locality of reference, the idea behind every cache. Programs keep touching the same or nearby data in short bursts.
• The mainframe era: the IBM System/360 Model 85 was one of the first commercial computers with a hardware cache, a small fast memory between the CPU and main memory, and through the 70s caches spread across mainframes.
• 1980s: CPUs start getting small caches of their own, and multiple levels (L1, L2, later L3) become the norm.
• 1990s: browsers cache pages and images to survive dial-up, and HTTP caching headers show up to manage it.

The point of that detour was one slide: for most of history, the constraint was on the client side, not the server side. My speaker note there just says “MTNL dial-up to 128 kbps broadband”, which anyone in Delhi of a certain age will understand. Today it’s hard to even use all of your bandwidth.

Then today’s caches, from CDNs to distributed ones. Memcached arrived in 2003 as one of the first popular distributed in-memory key-value stores, and Redis followed in 2009 with its data structures.

Beyond the cache: Redis

Quiz time. Two chances for the full form of Redis. It’s REmote DIctionary Server.

The Wikipedia definition calls it an in-memory key-value database, cache and message broker, “with optional durability”. People skip right past that last bit. On the DB-Engines ranking for key-value stores, Redis is at the top, and that’s a ranking of databases. My speaker note for that slide says “Yes DB.”

The DB-Engines key-value ranking, March 2024.

The data structures are what make it more than a cache. The core ones are strings, lists, sets, sorted sets, hashes, streams, geospatial indexes, bitmaps, bitfields and HyperLogLog, and Redis Stack adds things like JSON, search and time series on top (the dotted boxes on my slide). The Redis docs on data types are a very good starting point.

The data structures, with the two I picked highlighted.

There was no way to cover all of them in one slot, so I picked two use cases and asked everyone to go home and try the rest.

Demo 1: matching riders with drivers

The first use case was real-time driver tracking and geospatial matching, the kind of problem a ride-hailing app has at Uber’s scale: match riders with nearby drivers in real time, through a huge volume of location updates, with low latency, room for peak demand and a fast “find the nearest” query.

Drivers around you, then the search: widen the radius, skip the ones already taken, match with the nearest free driver.

Before the demo there was a “Light mode alert in 3… 2… 1” countdown for a couple of bright slides, then “Okay, we are back to the Dark side!”, then “Demo Time” with “fingers crossed!” in small print.

The simulator is a small Jupyter notebook. It drops 10 drivers at random points within about 2 km of 28.54°N, 77.19°E, which happens to be right around IIT Delhi, then sends 10 riders looking for a driver within a 1 km radius. Each driver check-in is a single GEOADD:

redis_client.geoadd("available_drivers", (longitude, latitude, driver_id))

And each rider query is a single radius search, sorted by distance:

nearby_drivers = redis_client.georadius(
    "available_drivers", user_longitude, user_latitude, radius, unit='km', withdist=True
)
nearby_drivers.sort(key=lambda x: x[1])

Under the hood, a geo set is just a sorted set scored by a geohash of the coordinates.

Where Redis sits in the matching service.

The matching is deliberately naive, set in an ideal world where the nearest driver always accepts your ride. Each rider takes the closest driver who isn’t already matched. Folium draws the map with taxi markers for drivers and a dashed green line for every match, and if every nearby driver is taken, the rider gets a red 1 km circle instead.

In the run that’s saved in the repo, the first three riders got a driver within a kilometre. The fourth rider’s only nearby driver had just been taken by the third, so: “No driver found for rider3!” and a red circle. Pretty honest picture of Delhi on a Friday evening. :P

If you run it on a newer Redis, note that GEORADIUS has been deprecated since Redis 6.2 in favour of GEOSEARCH (with FROMLONLAT and BYRADIUS), which does the same job.

And yes, both live demos worked on the day. Murphy’s law sat this one out.

Demo 2: Pub/Sub

After a time check, the second section was messaging. On the slide I called it “Streams aka PubSub”, and the demo itself was plain Pub/Sub: two terminals, one publishing and one subscribed.

Publishers, channels and subscribers.

# terminal 1
SUBSCRIBE mychannel

# terminal 2
PUBLISH mychannel "Hello, world!"

Messages from the second terminal show up in the first in real time. A third client subscribed to a pattern gets every matching channel:

PSUBSCRIBE news:*

PUBLISH news:sports "Latest sports update!"
PUBLISH news:weather "Today's weather forecast."

It isn’t one-to-one either. You can have many publishers and many subscribers.

The bigger idea was messaging queues: applications talking asynchronously, with senders and receivers decoupled. The use cases on the slide went from real-time chat and notifications to distributed task processing and real-time analytics, and my note there says each of those lines has a startup trying to wrap an API around it (Confluent and Kafka came to mind). Do your research, it’s fascinating.

A message queue, and one of the use cases: a real-time analytics pipeline.

Since the slide lumped them together, I’ll be precise here. Pub/Sub is fire-and-forget: if a subscriber isn’t connected when a message is published, that message is gone for it. Streams (XADD, XREADGROUP, XACK) are an append-only log stored in Redis, with consumer groups and acknowledgements, so a consumer that was down can catch up. If losing a message would hurt, use Streams.

Yes, a database (with persistence)

Back to that 2023 tweet. The deck only had “Yes DB.” on this, so here’s the longer version. Redis keeps everything in memory, but it has two ways to put data on disk, and you can use both:

• RDB takes point-in-time snapshots on a schedule. The files are compact and restarts are fast, but you can lose whatever was written after the last snapshot.
• AOF logs every write operation and replays the log on restart. With the default appendfsync everysec you lose at most about a second of writes, and you pay for it with a bigger file that Redis rewrites in the background.

With both turned on, Redis uses the AOF on restart because it’s the more complete one. The persistence docs cover the trade-offs well.

So for a side project or a prototype, where the data fits in RAM and access is by key, Redis is a fine primary store, which is what I meant in that tweet. Once the data outgrows memory, you need joins, or you can’t lose even a second of writes, use a real database and put Redis in front of it.

Must reads

I closed with a few engineering blog posts where Redis does a lot more than caching:

• Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework (Uber)
• Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System (Netflix)
• Building a Scalable ML Feature Store with Redis (DoorDash)
• How Paytm Redis Clusters is Managing High Workloads with Growing Customer Base (Paytm)

Wrapping up

I’d asked people to find me after the talk and say hi, and a few did. A few more posted about it on X later that day. Thank you, all of you, it really made my weekend! 🙌

If you were there and want to play with it, clone the repo, run Redis locally, crank NUM_OF_DRIVERS and NUM_OF_RIDERS up and see how fast the matching stays. And if you’ve worked out the hashmap question, my DMs are open.

If you like talk write-ups, here’s an older one: the time I walked colleagues through my Lord of the Rings homelab back in 2020.

‹ 1 / 78 › ⤢

jatinkrmalik/DelhiFOSS-2.0 Demo code from my DelhiFOSS 2.0 talk, The Redis Paradox: Beyond the Cache Horizon. Jupyter Notebook ★ 3 View on GitHub →