Laayo's tech stack, part three: realtime

A delivery app is a realtime app pretending to be a shopping app. The moment you place an order, everything that matters is live: has the restaurant accepted it, has a rider picked it up, where are they now, did they send a message?

For a long time, most of that reached your phone by polling: the app asked the server every few seconds whether anything had changed. Polling is simple and robust, and we still keep it as a fallback. But it is slow to notice changes, wasteful when nothing has changed, and expensive at dinner time when every open app is asking at once.

Our goal is that order status, a rider’s pin and a new order on a partner’s tablet show up in under a second, and that a store opening or closing is visible in the app within five seconds. This post is about how we got there.

Why we replaced our WebSocket server#

Our apps already spoke the Pusher protocol, a simple and widely supported WebSocket protocol for publishing events to named channels. On the server side, that protocol was handled by a PHP package running as several processes behind a load balancer.

That setup had two problems. The package had been abandoned upstream, which blocked us from upgrading Laravel. And it had a quieter bug: each PHP process kept its own list of who was subscribed to which channel, but the server published each event to only one of the processes. With three processes, an event reached roughly a third of the people listening. Chat and the support desk mostly worked because they also polled, which hid the problem.

We replaced it with one Go process that speaks the same protocol, frame for frame. Because the frames are identical, no app update was needed. We built a checker that runs the full protocol end to end, from connecting and subscribing through pings, publishing and rider location frames, and ran it against both servers until they matched. Then we pointed the load balancer at Go, kept the old processes ready for a day as a rollback, and retired them.

One process holding every subscription fixes the one-in-three bug by design. Go handles thousands of idle connections cheaply, so a single process is plenty for our scale, and it is easy to reason about.

Events after the commit#

The Laravel side publishes events whenever something meaningful changes. The most important is OrderStatusChanged. It fires when an order’s status or assigned rider changes, and only after the database transaction commits. Publishing before the commit is a classic bug: the app gets the event, fetches the order, and sees the old status because the write has not landed yet.

Each event goes to every channel that cares about it:

• the order channel, for the customer watching that order,
• the store channel, so a new order rings on the partner’s device,
• the rider channel, including the previous rider if an order is reassigned.

Laravel used to publish events through the public internet and back in to our own server, which cost about 80 ms per event. It now publishes to the Go hub directly on the same machine in about 11 ms.

Channel names are signed. Instead of a guessable name built from an order number, each channel name carries a short signature generated on the server. The server hands the right name to the app that is allowed to listen, and nobody can subscribe to someone else’s order by counting upward.

Store open and close in 0.2 seconds#

A complaint we heard from partners: “I closed my store, but customers could still see it as open.” The cause was not one bug but a stack of caches, each reasonable on its own: store lists cached for 45 seconds, search cached for five minutes, saved copies from part two, and apps that only refreshed when you reopened them. Scheduled opening and closing was worse, because a store crossing its closing time changes no row in the database at all.

We built storelive, a small Go service whose only job is to know which stores are open right now, and to say so the moment that changes.

• When a store, its schedule or its settings change in Laravel, the zone’s store version goes up and every cached list for that zone is dropped. Laravel also sends storelive a notice.
• storelive works out the effective open state with the same rules the API uses, including schedules, temporary closes and city-wide closes. It checks on a short interval, wakes immediately on a notice, and also wakes exactly at the next opening or closing time in the zone.
• On any change it publishes stores.changed on the zone’s channel, listing which stores changed and whether they are now open.

On our tests environment the event reached a phone on the public internet about 150 ms after the partner saved. A scheduled close landed within a third of a second of the minute. City-wide closes are sent once more a few seconds later as a safety net, and apps still refresh when they come back to the foreground, in case a socket was asleep. The server side is live, and the app side reaches phones with the next customer app release.

The same path now carries offers. When menuwatch from part two sees a discount or coupon change, storelive pushes it out, so a new offer shows up as soon as it starts.

Rider positions#

Live tracking is the most demanding realtime feature we have, because data flows in constantly, not just when something changes.

Riders send their position every few seconds, over a WebSocket where possible and over plain HTTP otherwise. Both paths now land in Go, which checks the rider’s token, acknowledges the update and does three things with the newest point:

1. Redis GEO index. Each zone has a geospatial index of online riders, plus a freshness score per rider. Questions like “which riders are near this restaurant” become one fast Redis query instead of a scan of a database table. Riders who stop reporting fall out of the index on their own.
2. Last known point. A small record per rider keeps the latest position, so every screen that asks “where is this rider” sees the newest point without touching the database.
3. Fan out. The point is published to the customers tracking that rider’s order, who see the pin move.

The database still gets the position, but at most once every ten seconds per rider instead of on every ping. Readers that load a rider from the database get the fresher Redis point laid over the stored one automatically, so about 25 places in the code that read rider positions got fresher data without a single change.

Apps: realtime with a seatbelt#

On the phone, the native customer app has one shared realtime client, written once in Swift and once in Kotlin with the same design. Screens subscribe to channels and get events. The client handles everything else:

• Shared connections. If the tracking screen and the chat both need the same order channel, they share one subscription, counted by reference and closed when the last screen leaves.
• Heartbeats. A ping every 30 seconds, and a connection that has heard nothing for 75 seconds is treated as dead and replaced.
• Backoff. Reconnects wait one second, then two, up to thirty, so a server restart does not cause a stampede.
• Resubscribe and catch up. After a reconnect, screens are told so they can fetch anything they might have missed.

Polling has not gone away. It runs more slowly as a safety net. Realtime makes the app fast, and polling makes sure it is never wrong for long. When an event arrives, the app usually fetches the fresh order rather than trusting the event alone, so the event says “something changed” and the API stays the source of truth.

What’s next#

The next step is moving order events onto Redis Streams, so dispatch and notifications can be driven by a durable log of what happened instead of each feature reacting on its own. That work is planned and we will write about it when it is live.

In part four we move to the phone itself: why we rebuilt the customer app in native Swift and Kotlin, and how we did it without logging anyone out.

This is part three of our series on Laayo’s tech stack.