Everyone wants to talk about RAG. Nobody wants to talk about what happens after the retrieval. That's the part that actually matters. The retrieve-augment-generate pattern has b...

Everyone wants to talk about RAG. Nobody wants to talk about what happens after the retrieval.

That's the part that actually matters. The retrieve-augment-generate pattern has been a commodity since the back half of 2023. You can spin one up in an afternoon. Embed your docs, write a tool, hand it to Claude, watch it answer questions. Cool. Now what?

The "now what" is where most assistants quietly fall over. They retrieve the wrong thing. They retrieve the right thing and the model ignores it. They cite something that was true six months ago. They are wrong with confidence, and there is no system around them to notice or to learn.

Hub Helper Harry, the assistant I've been building inside ProvenLabs, is a HubSpot knowledge assistant. He answers questions about HubSpot. Specifically, he answers them with citations, he tells you when he doesn't know, and he gets better at it every week without me retraining anything.

The reason he gets better isn't the model. The model is Sonnet 4.6, with Opus 4.7 for code-heavy work. Anybody can use those. The reason is a five-stage loop wrapped around the retrieval that turns every conversation into either a confirmed answer, a flagged gap, or a curated correction.

I'm calling it the Harry Loop, because at some point you have to name the thing.

This article is the system, end to end. Retrieve, rank, verify, correct, curate. Five stages. Each one is a known technique on its own. The interesting part is what happens when you wire them together and let them feed each other.

The thesis: the moat isn't the model

Models are getting commoditized faster than anybody predicted. Sonnet 4.6 is excellent. So is Gemini. So is whatever Grok shipped last Tuesday. If your only edge is "we use Claude," you don't have an edge.

The edge for a vertical assistant lives in two places. The first is the data you have that the public internet doesn't. The second is the feedback loop that converts every user interaction into a slightly more accurate system tomorrow.

The Harry Loop is the second thing.

It is unsexy. There are no GIFs of it. It is mostly cron jobs, a few database tables, and the discipline to actually use the feedback users give you. But it compounds. Every week the KB gets a little cleaner, the corrections list gets a little smarter, and the verified-responses cache gets a little richer. None of that happens to a stock LLM behind a chat interface.

Here's what each stage does, and why I chose it.

Stage 1: Retrieve (more than vector search)

If you ask Harry "how do I create a workflow that sends a Slack message when a deal closes," there are three things that need to go right before any model output is even possible.

First, the query gets cleaned up. A query optimizer strips fluff like "hey can you" and expands HubSpot terminology so "ops hub" finds "Operations Hub." Cap at 400 characters. Keyword-focused. This is dumb-but-important.

Second, Harry generates a hypothetical answer with Haiku and embeds both the original query and the hypothetical. This is HyDE, hypothetical document embedding. The reason it works is that a question and its answer live in different parts of the embedding space, and the answer-shaped embedding tends to match the documents you're looking for better than the question-shaped one does. Two vector searches run, the results get fused.

Third, hybrid search runs on top of that. BM25 keyword over a full-text index, pgvector cosine similarity against the embedding column, threshold 0.45. Reciprocal Rank Fusion (RRF) combines the rankings into a single ordered list. No magic. The trick is just running keyword and vector together instead of arguing about which one is "real" search.

The detail that matters most here is Contextual Retrieval, the technique Anthropic published for solving the "semantic orphan" problem. When you chunk a document for embedding, you prepend a few lines of document-level context to each chunk before you embed it. So a chunk that says "Click Save and exit" becomes "From the workflow editor in HubSpot Operations Hub: Click Save and exit." The isolated chunk is meaningless. The contextual one is searchable. This is the single highest-leverage retrieval upgrade we've made.

Harry stores the raw content and the contextual content side by side. Search runs against the contextual version. Display returns the raw version. The model sees what a human wrote, not the synthetic preamble.

Stage 2: Rank (the cross-encoder you can actually afford)

The top N candidates from hybrid search are not in the right order. They are in an order that two different similarity scores agreed on. That isn't the same thing.

The fix is a re-ranker. A model that looks at the query and a candidate together and scores how good a match they are. Cross-encoders are slow because they don't pre-compute embeddings, they re-read both pieces of text every time. That's why nobody used to put them in production.

Harry uses Claude Haiku as the re-ranker. It is not a purpose-built cross-encoder. But it is fast, it is good at relevance judgments, and it has a 2-second budget before we fall back to the pre-rerank order. That last part is non-negotiable. A re-ranker that occasionally fails open is fine. A re-ranker that occasionally hangs the whole chat is not.

The output of the re-ranker is the load-bearing part of the entire retrieval pipeline. The scores it returns drive whether the agent stays with the internal knowledge base or escalates to web search. Strong (>= 0.60) means stop, we have it. Moderate (0.45 to 0.59) means use the KB but consider the web if the user asked something the chunk doesn't cover. Weak (< 0.45) means call the web tool and log a gap.

If you remove those scores from the tool output to "clean it up," you break the escalation logic silently. The tool output is the API. I learned that the hard way.

Stage 3: Verify (escalation, fallback, cache)

The agent now has either a confident internal hit or a miss. Two things happen depending on which.

If it's a confident hit, Harry also checks a verified-responses cache. This is a separate table of question and answer pairs that have been manually approved as correct, each with its own embedding. If the current question is semantically close to one of them, Harry surfaces that response. It's a fast path for the questions we already know the answer to. Think of it as the "we already shipped this conversation, here's the official version" layer.

If it's a miss, the agent escalates to Tavily web search. Tavily is the public-internet retrieval tool we use because it returns clean, ranked, chunked content. It's bounded hard: domain allowlist of knowledge.hubspot.com, developers.hubspot.com, hubspot.com, blog.hubspot.com, community.hubspot.com. Score floor of 0.65. Three chunks per source. Raw content included.

Harry is a HubSpot assistant. The product promise is that he gives you HubSpot answers. The day he confidently quotes a third-party blog about how to do something in Salesforce is the day users stop trusting him. The allowlist isn't a hack. It's the product.

Every web search that fires because the internal KB missed gets logged to a kbgaplogs table with the topic breadcrumb. That's the input signal for the next two stages.

Stage 4: Correct (the cheapest moat in software)

Here is the stage almost nobody builds. It is also the cheapest one to build.

Every Harry response has a thumbs-up and a thumbs-down button. The thumbs-down opens a text field. The user types what was wrong: "this is outdated, the menu moved," "the workflow type is wrong, this only works for company-based workflows," "the API endpoint you cited is from v1, current is v3."

Those corrections go into a corrections table. On every subsequent chat request, the route handler injects relevant corrections into the system prompt before the model ever sees the user's question. So the next time someone asks about that workflow, the model already knows the menu moved.

That's it. That's the whole mechanism. A database table and a string concatenation.

What it does in practice is convert your user base into a continuous accuracy patch. Every wrong answer is a one-time wrong answer. The cost of an error decays toward zero because the next person doesn't get hit by the same error.

This is not magic. It is the most boring possible implementation of a learning system. But you have to actually do it. Most products collect thumbs-down and then never look at the data. Harry looks at the data on every request.

The deeper version of this is the autonomous learning loop, which takes corrections that get repeated or admin-approved and promotes them into proper knowledge-base documents. That part is genuinely interesting and worth its own writeup. But you don't need it to start. You need a table.

Stage 5: Curate (the cron job that runs your knowledge base)

The corrections loop fixes the agent. It does not fix the knowledge base. For that you need curation.

Curation is four rot-detection signals, all running as cron jobs or inline hooks, all writing to a single kbcurationlog table.

Stale-and-relevant. Live docs older than 30 days whose keywords overlap with recent user queries from the gap log. The signal says "people are asking about this, our doc is old, an admin should look."

Contradiction. When auto-ingestion fetches a doc whose Jaccard distance from an existing live doc is over 30%, flag both. Either the source updated and we need to update too, or our existing doc is wrong.

Dead source. HEAD-request every live doc's source URL. 404, 410, or cross-host redirect, flag it. The doc may still be locally accurate, but its citation is dead, and that breaks user trust.

Negative feedback. When a user submits a thumbs-down with a correction that cites a specific source URL, flag the doc behind that URL.

All four signals call the same flagRot() function, which is idempotent. A doc already flagged doesn't get re-flagged, but the log stacks so admins can see "this doc has been flagged by three different signals over the last two weeks, maybe deal with it."

The admin curation page is a tabbed interface (Pending Review, Flagged, Archived, Recent Activity) with one-click Approve, Archive, Restore, Keep Flagged, and Rewrite actions. Bulk select is supported. Every action writes to both the curation log and the audit log. A 7-day rollback primitive lets you undo accidental archives.

Once a day, a digest email goes out to admins summarizing the last 24 hours of curation events, including the 7-day web-search escalation rate and the delta versus the prior 7 days. If escalation is climbing, the KB is rotting.

Why this works (and why it works for vertical assistants specifically)

Each stage feeds the next.

Retrieval finds candidates. Ranking orders them and emits the signals the agent uses to escalate. Verification either confirms the hit, hits the cache, or fires the web fallback. The web fallback writes to the gap log. The gap log feeds the rot signals. The rot signals feed the admin curation page. The admin actions promote, archive, or rewrite docs. The next retrieval is operating on a slightly better corpus.

Corrections plug into a parallel rail that operates on the agent itself rather than the KB. A user correction lands in the corrections table. The next request includes it in the system prompt. The agent stops making that mistake immediately, before the KB curation pipeline has even noticed.

The two rails together (KB curation and corrections) cover both classes of error: the corpus is wrong, and the agent is making the wrong inference from a right corpus. Most systems handle one or neither.

The reason this works for vertical assistants specifically is that the entire loop assumes a bounded domain. Tavily has a domain allowlist. The rot signals fire on docs within a single knowledge base. The verified-responses cache only makes sense if you can expect the same questions to come back. The autonomous learning loop only works if user corrections compound across users, which only happens if users are asking similar questions, which only happens if they're all in the same domain.

A general-purpose assistant cannot do this. The surface area is too wide. The questions never repeat closely enough. The "right answer" is too contested. Vertical scope is what makes the loop economical, and the loop is what gives the vertical assistant a moat that doesn't depend on which model is hot this month.

The honest tradeoffs

Every stage has a cost. HyDE doubles your embedding calls per query. The re-ranker adds one to two seconds of latency. Corrective RAG (an additional grade-and-rewrite step we have but leave off by default) at least doubles latency on a miss. Contextual Retrieval requires re-embedding your entire corpus during ingestion.

The corrections injection grows your system prompt as the corrections table grows. Harry doesn't inject all of them, only the ones semantically relevant to the query. But you still pay tokens.

The rot signals produce false positives. A doc can be 31 days old and still completely accurate. A 404 can be a transient host issue. The cron job runs anyway, and admins are the ones who decide what's noise. We keep the thresholds conservative for exactly that reason.

None of this is free. The bet is that the cost is dominated by what you gain in accuracy and trust. For a vertical assistant where a wrong answer is a churn event, it usually is. For a horizontal product where the user's bar is "ChatGPT-ish," it usually isn't.

The takeaway

If you are building a vertical AI assistant in 2026, the thing that will separate yours from the next person's is not the model. It will be the model in a year. The loop you build around the model is what compounds.

Five stages. Retrieve, rank, verify, correct, curate. Each one is a known technique. Together they are a self-correcting system that gets more accurate the more it gets used.

That's the Harry Loop. Now I have to go write the corrections that come in from the people who read this article and tell me I got something wrong.

🤓