Two system design problems from my 2022 interviews, worked with real numbers

Hello folks,

I’ve spent most of June and July in interview loops, and I kept an Obsidian note for almost every round. Most of these notes are scrappy: half-finished code, text pasted out of a shared Google Doc with all its formatting, a “Feedback” heading with half a sentence under it. Two of them are worth cleaning up though. Both are really good problems, and in both I got to put actual numbers on the design!

The first is about getting street-level photos from a fleet of camera cars into a maps product. The second starts as a split-testing (A/B) service and then keeps pushing on the data side: a backfill, deduplication, and a migration worker that has to survive failures.

I’m keeping both generic, so no company or interviewer names here. Back in 2016 I wrote about prepping for software engineering interviews as a fresher. This is the other end of that, a few jobs later.

What follows is roughly what I said in the room, cleaned up from the notes. There’s also a section at the end on what I’d tighten, because I redid the math while writing this and, well, not everything holds up.

How I run a system design round

Over the last few weeks I’ve settled on a ten-step cheatsheet for these:

1. Functional requirements.
2. Non-functional constraints.
3. Capacity estimation: traffic, storage, bandwidth, and cache sizing with the 80/20 rule.
4. The APIs.
5. A top level diagram.
6. The database and schema.
7. An MVP that works for a single user.
8. Scaling: bottlenecks, single points of failure, load balancers, caches, replication, queues, async workers.
9. Observability.
10. Testing and reviewing the design.

The last step has a note next to it: treat it like a coding round. Walk the design against every requirement, and walk the failure cases too, not just the happy path.

Problem 1: street-level photos from a fleet of camera cars

A maps product wants street-level imagery, collected by 360-degree cameras mounted on taxis.

Requirements

On the car:

• The capture interval has to be configurable. I modelled it as a function of location and speed that returns an interval, with 1 second as the default. A taxi crawling through a city and one doing highway speed in the countryside shouldn’t shoot at the same rate.
• Each photo needs a position and a direction, so GPS for lat/lng plus a gyroscope or compass for heading.
• Photos are stored on the car and uploaded later for processing.

On the backend, the photos get processed for privacy and consistency, then stored and served to users.

The non-functional side shapes everything else. It’s write heavy and network heavy, because every car is pushing large files over an onboard 4G modem. It needs high durability: a photo that never makes it off the car is gone, so there are retries on failure and an onboard buffer. Eventual consistency is fine, since nobody needs a street photo on the map a second after it was taken. And the server side has to be highly available, because the cars upload whenever they can.

Capacity estimation

• One 360-degree image is about 10 MB (high shutter speed, good dynamic range). Each one carries metadata: car ID (a UUID), lat/lng, direction, timestamp, and camera settings like shutter speed and ISO.
• A car covers about 150 miles a day at an average of 15 miles an hour, so about 10 hours of driving.
• At 1 photo a second, that’s 10 x 60 x 60 = 36,000 photos a day, or about 360 GB per car per day.
• On the car, 4 to 8 GB of RAM acts as a small buffer and flushes to a 1 TB SSD with fast reads and writes.
• For upload, assume 4G LTE at about 10 MB/s, sending batches of 60 photos (one minute of driving) with a batch_id.

Then processing. I assumed 1 second per image per thread. A 32-thread machine does about 30 images a second, so one car’s day (I rounded it to 30,000 images) takes about 1,000 seconds, roughly 16 minutes. On 16 threads it’s about 32 minutes, and on 8 it’s 64.

Assume 1,000 taxis in a city. 1,000 x 16 minutes is 16,000 minutes, about 266 machine-hours a day. Spread over 24 hours, that’s about 11 machines with 32 threads, or about 5 with 64 threads. The fleet can also scale with load, as a function of queue size and worker count.

API

The car only needs one endpoint:

POST /upload
Authorization: <auth token>        (header)
body: taxi_id, a batch of images (gzip/tar)

201 Created  -> mark the batch DONE
4xx / 5xx    -> keep the batch, retry later

The design

Here’s the diagram from the round.

The whole pipeline as I drew it. “geo-shared” means geo-sharded.

The car runs a tiny durable queue of its own. Image capture writes to the SSD, and every batch gets a row in a local BatchLog table with an id, the file path, a status of PENDING, DONE or FAILED, the taxi_id and created_at. A worker pulls pending batches off a local task queue and uploads them. A 201 marks the batch done, and a 4xx or 5xx puts it back for a retry. When the network is bad, the car degrades gradually: it keeps shooting and keeps buffering, and nothing is lost until the SSD fills up.

On the car: capture, a batch log, a retry queue and the upload worker.

On the server, uploads go through a load balancer with auth to an ingestion service sharded by geography: megaregion, then region, then subregion, then by how dense the area is. It pulls the metadata out into a metadata store and queues each image for processing.

The image processing service reads the metadata (car ID, lat/lng) and then does three jobs. For privacy, it blurs faces, number plates, local billboards and sensitive sites. It renders each image at a few resolutions for different clients. And it dedupes: lat/lng plus direction hash into a UUID, so the hundredth photo of the same corner from the same angle doesn’t become the hundredth copy. The known UUIDs sit in a Bloom filter replicated across megaregions (APAC, ANZ, USNC and EMEA, as I labelled them).

Processing, dedup and serving.

Processed images land in a store, grouped into clusters: a map_image_cluster_id holds a list of images with their relative orientation, so a viewer can stitch the 360-degree view back together. Serving goes through a CDN sharded the same way, megaregion down to subregion, and all the way to local caching at the ISP. A user on the mobile or web app asks for a tile by hash(lat/lng, direction) and gets it from the nearest cache.

Problem 2: an A/B split-testing service

A product has users and features, each feature can have several variations, and each variation gets a configurable weight. The examples were the usual ones:

• Button colour: blue 10%, red 80%, purple 10%. Then the weights ramp. Blue goes to 20% and then 90%, red drops to 60% and then 5%, purple goes to 20% and then 5%.
• Pricing: $10 a month for 90% of users and $20 a month for 10%, moving later to 80/20.

There were mock screens of the product, with a few assumptions: about 400 tests, 3 to 10 variants each, and a list page showing 10 tests at a time. For scale, 100 million to a billion users, about 100 million daily actives, geo-distributed.

Data model

The hierarchy is simple. The platform has tests, and tests have variants. This is what made it into my notes:

Test                 id (uuid, primary key), name
User                 id, country, is_sharing_password_with_family
UserTestAssignment   id, user_id, variant_id

The variants themselves, each with its weight, hang off a test. I never wrote that table out.

The client side is one call. The frontend asks for the variant and renders it:

color = getVariantForUser(userId, testId)
<button color={color}>

In my notes the body of getVariantForUser is literally ???. Yup. I never filled it in, more on that below.

The backfill

Then the follow-up: move every US user who shares their password with family onto the $20 variant. Out of 100 million users, call it 10 million who match.

My answer was a batched backfill. Index the columns the filter uses (user ID, country and the password-sharing flag). Walk the user table in chunks of 5,000, pick out the matching users in each chunk, and push each batch onto a task queue. Workers then write the new assignments at about 4,000 to 5,000 requests a second.

At 5,000 a second, 10 million users is 2,000 seconds, so the whole backfill takes about half an hour.

Deduplication

Next: the user table has 100 million to a billion rows, and some of them are the same person with their name typed in different ways. Design a worker that finds the duplicates. The only reliable unique key comes from a slow external identity API that takes a few seconds per call and maps each real person to exactly one ID.

I went with MapReduce. Map each user to (external_id, user_id) by calling the API, then reduce by key:

{
  external_id_1: [user_id_1, user_id_2, ...],
  ...
}

Any key with more than one user ID is a duplicate group.

Make the worker idempotent

The last part was a snippet that walks a subset of people and does person.age = person.age + 1, with a save that fails 0.0001% of the time. It’s neither interruptible nor idempotent. If it dies halfway and you rerun it, some people age twice.

My fix in the round was to keep a set of IDs that already succeeded and skip anything in it, retry on failure, and only add an ID to the set after its save goes through. I also noted that the set could be a Bloom filter.

What I’d tighten

Writing this up, I re-ran the numbers and reread the notes, and a few things jump out.

The camera car math. I used 30,000 photos for processing but 36,000 for storage. With 36,000, one car’s day is 20 minutes on 32 threads, the fleet needs about 333 machine-hours a day, and that’s about 14 machines, not 11. The 11 also assumes processing runs evenly around the clock. The upload is the bigger problem. 1 photo a second at 10 MB is 10 MB/s, exactly the LTE bandwidth I assumed, so there’s zero headroom while driving. The 1 TB SSD (almost 3 days of photos) is what keeps the design working, and uploading over depot Wi-Fi at the end of a shift might be a better plan than LTE. A city of 1,000 cars is also pushing about 360 TB a day before dedup, and that number should have driven the storage discussion.

getVariantForUser. The ??? has a standard answer: hash user_id + test_id into one of 100 buckets and map the buckets onto the cumulative weights. There’s no lookup on the hot path, and the same user always lands in the same bucket. The ramps are the catch. Moving blue from 10% to 20% re-buckets some users, so anyone who has already seen a variant needs a stored assignment to keep their experience sticky. That’s what UserTestAssignment is really for.

The backfill. My notes have both an ID range and offset += 5000. The range on the primary key is the one to keep, because OFFSET gets slower with every page.

The Bloom filter. A Bloom filter never gives false negatives, but it does give false positives. In a “skip if already done” check, a false positive means skipping someone who was never migrated, which is exactly the mistake you can’t afford here. I’d make the write itself idempotent instead: store a migration version on each row and only update rows still on the old version, so a rerun does nothing.

The dedup worker. At, say, 3 seconds a call, a billion calls is about 95 years of serial work. The API is the bottleneck, and MapReduce doesn’t change that. Most of the design is really about how wide you can fan out under the API’s rate limit, and caching every answer so a rerun never calls it again.