Auto-refine drafts

Auto-refine drafts

The platform auto-generates draft replies for many outbound guest messages — AI Suggest, send-message task actions, etc. Some of those drafts are good as-is; some need a refinement pass before they go out.

Auto-refine is the per-template / per-tenant policy that decides whether a flagged draft goes out automatically, or sits in a pending-approval queue.

The three states

For every outbound draft, the platform evaluates:

  1. Confident — draft is high-quality; send as-is.
  2. Needs refinement — draft has a flagged concern (e.g. didn't reference guest history, language mismatch, missing piece of info from the FAQ).
  3. Blocked — draft has a hard issue (e.g. language detection failed, required placeholder didn't resolve).

For state 2, the operator has three options per template:

  • Manual approval (default) — drafts queue in the operator's inbox; operator reviews + sends.
  • Auto-refine + auto-send (gh#506) — drafts run through a second LLM pass that addresses the flagged concern, then go out automatically. No operator step.
  • Auto-refine + queue (gh#506) — drafts run through the refinement pass, then queue for operator review.

Where the toggle lives (gh#506 Phase 3)

In the V28 task template editor → Auto-refine behavior:

  • manual (default)
  • auto-refine-and-send
  • auto-refine-and-queue

The toggle is per-template — Send Check-in might use manual (operator-confirmed always) while Pre-checkout reminder might use auto-refine-and-send (low-stakes, high-volume).

How "needs refinement" is detected (gh#504 Phase 2)

The outgoingMessage.needsRefinement gate runs on every draft and looks at:

  • Whether the draft references the guest's prior messages (history coverage check).
  • Whether the language matches the resolved outbound language.
  • Whether known FAQ entries that should apply have been incorporated.
  • Whether the draft has any of N flagged anti-patterns (placeholder boilerplate left in, generic apology with no specifics, etc.).

Score-gated: drafts above the refinement threshold pass; drafts below go through the refinement pass.

Courtesy-message guard (gh#502 Phase 1)

A guest sometimes sends a courtesy message after the operator's reply ("thanks", "ok", "got it"). These should NOT prevent an outbound send-message task from auto-completing (earlier they did — the task waited for an operator reply that wasn't necessary).

gh#502 added the courtesy guard: courtesy messages are detected and excluded from the "has unanswered guest message" gate. The auto-complete proceeds.

Audit

Every auto-refine pass is logged with:

Visible in Admin → Audit Log → Auto-refine.

Cost considerations

A refinement pass = one extra LLM call. For tenants with high outbound volume, this can be material. The cost dashboard breaks out refinement-pass spend separately so you can decide whether the quality gain justifies the cost on each template.

If a template's refinement-pass volume is dominating cost and the refinement diffs are tiny (the draft was usually OK), flip the template to manual and let the operator decide on the rare flagged ones.


Implements: gh#502 (Phase 1: courtesy message guard), gh#504 (Phase 2: needsRefinement detection gate), gh#506 (Phase 3: auto-refine + V28 editor toggle). Related: gh#431 (special-request prompt-driven detection), gh#412 (LLM cost reduction work).

Source: the FlatsBratislava operator manual.