An 80 ms classifier in front of my voice assistant

Introduction#

I’m building Meia-Sete, a personal voice assistant that runs on Hermes Agent. Every voice assistant has the same problem: every utterance goes to an LLM, even when it’s just “turn off the living room light”. That costs seconds of latency and tokens, and in a voice conversation every second of silence counts.

The classic fix is an intent router: a small, fast model looks at the utterance first and decides whether it can take a shortcut (run a command directly) or needs the LLM. When Laya showed up on my radar, promising typed decisions in ~33 ms and in over 100 languages, I decided to measure instead of taking its word for it.

This post is the log of that test: installation, real numbers in Portuguese, what didn’t work and how I wired it all into Hermes.

What Laya is#

Laya is not an LLM. It’s an encoder (ModernBERT/mmBERT, 322M to 421M parameters) trained to answer fixed-format questions about a text, all in a single pass through the model:

There are three checkpoints: english, multilingual and typed-decisions. A Router detects the language and picks the checkpoint on its own. The package ships a Python SDK, a CLI, an HTTP server and an MCP server, which is the part that matters for using it with agents.

Prerequisites#

My machine: a 16-thread CPU, 31 GB of RAM, no NVIDIA GPU. Every number in this post is from the CPU.

Hands-on#

Installation#

Installing CPU PyTorch first avoids downloading ~3 GB of CUDA libraries I wasn’t going to use:

python3.11 -m venv ~/.venvs/laya
~/.venvs/laya/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
~/.venvs/laya/bin/pip install "laya[mcp,serve]"

Versions I used: laya 0.4.0, torch 2.14.1+cpu, transformers 5.19.0.

The first test was through the CLI, with the support triage preset. The input is Portuguese for “I was charged twice on my card, I want a refund”:

LAYA_DEVICE=cpu laya "Fui cobrado duas vezes no cartão, quero o estorno" --preset triage --json

The response came back with "language": "pt", routed to the multilingual checkpoint, and got the intent right (refund). So far, everything as promised.

The router’s questions#

For Meia-Sete I defined six intents. The state field the question reads goes between backticks (`fala`, “utterance”), a Laya convention. The instructions and criteria are in Portuguese on purpose; more on that in the numbers:

INTENT_QUESTIONS = {
    "intent": {
        "type": "choice",
        "instructions": "O que o usuário quer com `fala`?",
        "criteria": {
            "casa": "controlar a casa: luz, tomada, ar-condicionado, TV, roteador ou bloquear site",
            "musica": "tocar, pausar, pular ou mudar o volume de música ou mídia",
            "lembrete": "criar alarme, timer, lembrete ou compromisso na agenda",
            "consulta": "uma pergunta factual rápida: hora, clima, cotação, conta",
            "tarefa": "um trabalho de várias etapas: pesquisar, escrever, programar, resumir",
            "conversa": "papo, cumprimento, opinião ou desabafo, sem pedido concreto",
        },
    },
}

The labels, in English: casa is home control (lights, plugs, air conditioning, TV, router, blocking a site), musica is music and media, lembrete is reminders, alarms, timers and calendar, consulta is a quick factual question (time, weather, exchange rate, arithmetic), tarefa is a multi-step task (research, writing, coding, summarizing) and conversa is small talk, greetings, opinions or venting with no concrete request.

Then I wrote 32 utterances the way I’d talk to the assistant: “liga o ar no 22” (set the AC to 22), “põe um timer de 10 minutos pro miojo” (set a 10-minute timer for the instant noodles), “tô meio triste hoje” (I’m kind of sad today). I split them into 16 for training, used to tune the criteria, and 16 for validation, which I only looked at to check, so I wouldn’t fool myself with overfitting.

from laya import Router

router = Router(preload=True, default="multilingual")
r = router.predict({"fala": "apaga a luz da sala"}, INTENT_QUESTIONS)
r["answers"]["intent"]   # {'choice': 'casa', 'probabilities': {...}, 'answer_confidence': 0.97, ...}

Testing: the numbers#

First round#

Two lessons right away:

1. Write the criteria in the language of the text. I expected English criteria to help, since the model was trained mostly in English. The opposite happened: accuracy dropped almost by half.
2. The multilingual checkpoint is the best and also the fastest on CPU. The automatic router already picks it for Portuguese, so there’s nothing to configure.

Attempts that didn’t work#

Sitting at ~65% bothered me, so I tried to improve it. Nothing worked:

• Averaging over permutations of the option order. The documentation warns that the model has a position bias. I averaged over 3 rotations of the options: same accuracy, 2.3x slower.
• Rewriting the criteria with verbs and with examples (“e.g.: remind me to, wake me up, set a timer”): it got worse, 6/16 and 7/16. The consulta label started pulling in almost everything.
• Replacing the choice with six yes/no questions, each asking “is this about X?”, and keeping the one with the highest probability: 9/16 and 10/16, and 2x slower.

The turning point: using confidence as a gate#

If Laya doesn’t work as the final classifier, maybe it works as a filter: only trust it when it’s very sure. I measured precision at each confidence threshold over the 32 utterances:

It gets better, but there was still an error at confidence 1.00. The documentation warns about this: the distributed checkpoints are overconfident. Looking at the high-confidence errors, they all landed in consulta, tarefa or conversa, exactly the intents that need the LLM anyway.

So the final rule became: only take the shortcut if the intent is a command AND the confidence is ≥ 0.95. Everything else goes to Hermes. In the code, ATALHOS are the shortcut intents, LIMIAR is the threshold and destino is either the shortcut ("atalho") or Hermes:

from dataclasses import dataclass
from laya import Router
from intents import INTENT_QUESTIONS

ATALHOS = {"casa", "musica", "lembrete"}
LIMIAR = 0.95

router = Router(preload=True, default="multilingual", max_loaded=1)


@dataclass
class Rota:
    destino: str  # "atalho" ou "hermes"
    intent: str
    confianca: float


def rotear(fala: str) -> Rota:
    q = {"intent": INTENT_QUESTIONS["intent"]}
    a = router.predict({"fala": fala}, q)["answers"]["intent"]
    intent, conf = a["choice"], a["answer_confidence"]
    destino = "atalho" if intent in ATALHOS and conf >= LIMIAR else "hermes"
    return Rota(destino, intent, conf)

Results over the 32 utterances:

At 0.90, “bom dia, Meia-Sete!” (good morning, Meia-Sete!) became lembrete with 0.92. At 0.95 there were no errors on the shortcut, and the median latency was 76 ms. The price is that half the commands still go to Hermes. But that’s safe: the worst case is being slower, never doing the wrong thing.

Wiring it into Hermes over MCP#

The second use is letting the agent itself use Laya as a tool, for tasks like “classify the sentiment of these 500 reviews”. Doing that with the LLM costs tokens; with Laya, it’s local and free.

mcp_servers:
  laya:
    command: /home/voce/.venvs/laya/bin/laya-mcp-server   # absolute path
    env:
      LAYA_DEVICE: cpu
      LAYA_THREADS: '8'              # leaves CPU free for the rest of the machine
      LAYA_MODELS: multilingual      # loads only the checkpoint that matters for PT
      LAYA_DEFAULT_MODEL: multilingual
    connect_timeout: 60.0

The same can be done with hermes mcp add laya --command ... --env .... The server exposes 8 tools: laya_predict, laya_predict_batch, laya_preset, laya_decide and others.

The agent kept getting the format wrong#

In the first test I asked it to classify 4 reviews using laya_predict_batch. The final result was right, but looking at the session log I saw this:

invalid_questions: questions must be a non-empty JSON object keyed by question name
invalid_questions: questions[sentimento].instructions must be a non-empty string
invalid_questions: questions[sentimento].criteria must be a non-empty object of label -> description
MCP server 'laya' rejected the last 3 calls ... Paused for ~50s.

That was four bad calls and a forced pause before it got it right. The tool description isn’t enough for the model to get the nested schema right on the first try.

The fix: a one-page skill#

Hermes loads skills from ~/.hermes/skills/. I wrote one with the exact format, a complete example and the lessons from this post. It’s in Portuguese because that’s the language I use with the agent:

---
name: laya
description: Classificar muitos textos localmente (sentimento, categoria, sim/não, nota)
  com o servidor MCP Laya, sem gastar tokens do LLM.
---

## Formato exato (siga à risca, o servidor rejeita qualquer desvio)

- `choice`: `criteria` é um **objeto** rótulo -> descrição.
- `score`: `criteria` é uma **lista** ordenada do menor para o maior.
- `noul` (sim/não): sem `criteria`.
- `instructions` cita o campo do estado entre crases, ex.: `texto`.

(... exemplo JSON completo ...)

## Regras aprendidas em testes com português

- Escreva `instructions` e `criteria` em português quando o texto for em português.
- A confiança vem otimista demais: trate abaixo de 0.9 como "incerto".

In short: the skill describes when to use Laya (classifying many texts locally without spending LLM tokens), spells out the exact shape of criteria for each question type, and carries two rules from this post: write instructions and criteria in Portuguese when the text is in Portuguese, and treat confidence below 0.9 as “uncertain”.

I ran it again, with 5 reviews:

hermes chat -s laya -q 'Classifique o sentimento destas avaliações com o Laya...'

This time it was one call and zero errors. And the agent followed the confidence rule on its own: it flagged “É ok, nada demais” (“It’s ok, nothing special”; positive, 0.64) as uncertain and said it would classify it as neutral.

Pitfalls and trade-offs#

Summary#

Conclusion#

Laya isn’t magic, but it’s an honest tool: fast, local, and with a confidence score that, filtered carefully, tells you when you can trust it. For Meia-Sete, its job is to be the doorman that lets through directly only what it’s sure about.

The next steps are recording real utterances to get a proper test set and trying laya-train, which ships with the package, to fine-tune on my own intents. If accuracy goes up, the threshold can come down and more commands take the shortcut. That’s for a future post.