A support assistant that knows when to hand over

Laayo’s in app support chat is answered first by an AI assistant. For most questions that is exactly what people want: where is my order, how do I change my address, why was I charged a delivery fee, can I pay with E-Payment. The assistant reads the customer’s live order data and our help articles, and it answers in seconds at two in the morning.

But support is not a trivia game. Some chats need a person, and the worst thing an assistant can do is pretend it has called one. This post is about the mistake that taught us that, and how we rebuilt the assistant around one rule: never promise a human you have not actually called.

The chat that waited#

A customer could not receive the verification code needed to change their phone number. The assistant suggested checking the country code. When that did not help, it said something like “I’ll escalate this, hang tight.”

Nobody was alerted. At the time, the assistant had no way to bring in a person at all. The phrase came from our own help articles, which had been written for an older setup where escalation was real. The model read those articles, saw that escalating was the right move, and said so. A team member found the chat by chance many hours later.

Two lessons came out of that chat.

The model will repeat whatever promises your documents make. If your help content says “I will connect you to an agent”, the assistant will say it, whether or not that is possible. We rewrote every scenario in the knowledge base so that escalation means a concrete, real action, and we added an explicit rule against phrases such as “the team will review this” or “hang tight” unless a handover has actually started.

Check what the model really sees. Our knowledge base had grown longer than the amount of text we actually sent to the model. Everything past a certain point, including several important policies, was being cut off before the model ever read it. We now treat the size of the prompt as something to test, not something to assume, and the most important rules sit at the top.

A handover that is real#

The new design makes handing over an action, not a sentence.

When the assistant decides a chat needs a person, for example a refund dispute, a missing item it cannot verify, or a customer who simply asks for a human, it writes one short line and a special marker. The server sees the marker and does three things at once: it posts “Please wait while I connect you to an agent” into the chat, it marks the conversation as waiting, and it sends a loud alert to the support team’s Admin App. When a team member opens the chat, the customer sees that they have joined, by first name.

The server also watches the conversation for signs that a person is needed, independently of the model:

• Asking for a human, in English, in romanized Nepali and in Nepali script.
• Sounding upset, with a second look before acting, so a joke does not trigger an alarm.
• The assistant promising a follow up it cannot deliver, which is now treated as a bug to catch, not a sentence to send.

Tickets without a second inbox#

Once more chats reached people, our team had a new problem: too many conversations to read. So we added tickets, with one firm design rule. A ticket is a view onto the chat, not a second thread. The customer always replies in the same chat, and the team always replies there too.

Some reasons open a ticket at once. Others, like a customer using a strong word, go to the assistant first, which reads the whole chat and decides. In testing, a customer joking that a dish was “the worst, I loved it” was left alone, while “the rider took my cash” became a ticket immediately.

When a ticket opens, the assistant writes a brief for the team: a subject, a category, a priority and the order it is about. It can raise the priority but never lower it, and any order it links is checked to make sure it really belongs to that customer. The team member opening the ticket sees a summary at the top, instead of scrolling through twenty messages to find out what happened.

The ticket view and the handover flow are rolling out to our support team now.

Speaking the customer’s language#

Many of our customers write in Nepali, in Nepali script or in Nepali typed with English letters, and many switch between them in the same message. The assistant now follows the language the app is set to, or the language the customer writes in when they have not chosen one. Fixed sentences that our server adds, such as order status summaries, are translated with the respectful form of address and cached, so the same sentence is not translated again and again.

One small detail took longer than expected: a price written as “रु 219” inside an English sentence must not make the system decide the customer is writing in Nepali.

Turning complaints into reports#

When a customer describes a real problem with an order, the assistant can file it as a proper order report, the same kind a customer could file from the order screen. It always asks first and files only after the customer confirms. If the problem needs a photo, such as a damaged package, the assistant opens the report screen for the customer with the reason already filled in, because a photo is evidence our team needs to see.

Learning from what it could not answer#

Every night, a job looks at the questions the assistant could not match to anything on our menus, such as a local spelling of a dish or a nickname for a restaurant. It maps them onto the real dishes and stores them as search synonyms. That makes the assistant better, and because the same synonyms feed normal search, it makes the search box better too.

What’s next#

We are measuring how often a chat ends with the customer’s problem solved, with or without a person, and how long people wait for a human once a handover starts. Those two numbers, not the number of chats the assistant closes, are how we will judge it.