The map bill that was paying for answers we threw away

Every delivery app is, under the hood, a maps company that happens to sell food. We ask a maps provider where an address is, how far a store is from you, how long a rider will take, and which road they should use. Each of those questions costs a little money. At our size, a little money per question is fine. A little money times a mistake is not.

This is the story of a mistake that had been quietly running for a year, and the rules we wrote afterwards.

How the maps bill works#

Most of our maps questions go to Google Maps Platform. Different questions are billed differently, but the one that matters here is the Distance Matrix. You give it a list of starting points and a list of destinations, and it returns the travel time from every start to every destination.

The key word is every. You are billed per element, and the number of elements is:

elements = origins x destinations

Two starts and two destinations is four elements. Twenty starts and twenty destinations is four hundred. That multiplication is the whole story.

Finding it#

When we sat down with a year of billing, one line stood out. Distance Matrix was by far the biggest maps cost, much bigger than geocoding addresses or drawing routes, which surprised us. The customer app does most of our maps work, and we already knew its calls were cached aggressively. Something else was asking a lot of questions.

We searched our code for every place that called that API. There was exactly one server-side caller left: the rider app’s My Run screen. When a rider carries several orders, My Run shows the order of their stops and how long each leg will take: from where they are now to the first stop, from the first stop to the second, and so on.

The bug#

To get those legs, the server built a list of origins (the rider, stop 1, stop 2, ...) and a list of destinations (stop 1, stop 2, stop 3, ...) and sent them in one request. What it needed was the first origin to the first destination, the second to the second, and so on. The diagonal.

What it paid for was every cell. For a run of N legs it was billed N times N elements and read N of them. A run with twenty legs paid for four hundred travel times and used twenty. The rest were thrown away the moment the response arrived.

Nobody would have noticed from the app. The screen was correct, and it was fast. Only the bill knew.

The fix#

The fix was simple once we could see it. Instead of one big matrix, the server now sends one small request per leg, each with exactly one origin and one destination, all fired at the same time. The rider sees the same numbers in the same time. We pay for N elements instead of N squared.

We made two more changes while we were in there:

• Ask once per leg. If two legs in the same run share the same start and end, they are now one request, not two.
• Remember longer. Only the first leg really changes as the rider moves. The leg from stop 1 to stop 2 is the same road it was a minute ago. My Run refreshes every half minute, and it used to re-ask about every leg every minute. Those answers are now kept for five minutes.

We kept one thing that looks expensive on purpose: these requests still ask for traffic-aware times. In monsoon season, with one-way streets and roads that flood, a travel time without traffic is not worth much to a rider deciding which stop to do first. With the multiplication fixed, the cost of doing it properly is small.

The same bug, smaller, somewhere else#

Once you know the shape of a bug, you start seeing it. Our point-of-sale screens for restaurants calculate delivery distance in the browser using the same kind of matrix. They were sending two origins and two destinations, four elements, and reading one answer plus one address. We removed the origin that was never used. Two elements now, and the result on screen is byte for byte the same.

What was already right#

It is just as useful to know what not to touch. The customer app’s maps calls go through our own server, which caches them: an address lookup is remembered for a month, a route or a distance for two weeks. The public order tracking page asks for routes without traffic, requests only the fields it shows, and caches its answers. None of these were driving the bill, and “optimising” them would only have added risk.

Rules we keep now#

The fix took an afternoon. The rules are what we want to keep for years.

1. Never send a matrix to read a diagonal. If you need A to B and C to D, ask those two questions. The newer Routes API bills its matrix the same way, so it is not a way around this.
2. Cache by what changes. Fixed geometry, like store to store, can be remembered. A rider’s live position cannot.
3. Round the question. A rider moving one metre does not need a new route. Parts of the rider app already wait for real movement before asking again. Making every map screen do the same, with a cap on how many answers it keeps in memory, is next on our list.
4. Put a ceiling on the bill. Daily quotas per API mean that even a future bug can only cost a bounded amount before it stops itself. Setting those limits is part of the same clean up.

The best part of this fix is that nobody using Laayo noticed anything. Riders see the same run, restaurants see the same distance, and the money goes into making deliveries faster instead of into travel times nobody reads.