docker compose exec server /app/collie intake-eval \
--workspace <workspace id> --channel <chat channel id> --scope group \
--dataset /data/cases.jsonl [--model <another model of intake's provider>] \
[--thresholds 0.85,0.90,0.95] [--record]
- It runs the same prompt and answer reader AI intake uses, with the workspace's intake provider and model and the channel scope's options (words to hide, patterns to mask, instructions). Each call is metered like any intake call, as asked by a person, and counts in the workspace's AI usage.
- Before calling the model it prints how many cases and calls it will make. The result is JSON on
standard output: per action (
create,attach,ignore) the labelled cases, how many it got right and its severe and light errors; how labels and answers were confused; how many it would have applied on its own at the thresholds (the scope's unless--thresholdssays) and how many of those were wrong; the calls' latency. --recordkeeps the counts for the scope, the model and this server's prompt version. A newer run of the same dataset replaces the older one. A run keeps counting when intake moves to another model, and counts for nothing once the server moves to another prompt version. The same isPOST /api/v1/channels/{id}/ai/intake/evaluations(an agent token's record waits for a person).
The dataset
One case a line, already anonymised:
{"place": "group",
"before": [{"role": "staff", "sender": "Hoa", "text": "Which store?"}],
"items": [{"role": "customer", "sender": "Lan", "text": "Store 12, the export still fails",
"quote": {"role": "customer", "sender": "Lan", "text": "the stock export fails"}}],
"candidates": [{"ref": "170", "reporter": "Lan", "subject": "Stock export fails", "status": "open",
"updated_at": "2026-10-20T03:00:00Z", "last_message": "We are looking at it"}],
"commands": ["open"],
"label": {"action": "attach", "target_ref": "170"}}
items are the messages to decide, before the ones before them in the place, candidates the
conversations they may join (by a reference of your own), and label what a person decided: create,
attach with target_ref, or ignore. A case whose label names a conversation it does not offer is left
out: AI could never get it right.
Severe errors are the ones that keep a gate closed: an attach of a new request (it hides it in an old
conversation) and an ignore of a request. Other wrong answers are light errors.
Was this page helpful?
Updated
This page is rendered from the repository. Spot a mistake? Open an issue or a pull request.