Most write-ups of a company knowledge base ai skip the boring middle part: how the data actually gets from scattered systems into something an assistant can query safely. This is a worked example, illustrative rather than a verified named deployment, walking through that middle part for a composite 90-person logistics company we'll call Northfield Freight. Nothing here is a real customer account. It is a realistic scenario built to make the before and after concrete.
The company, before
Northfield Freight runs dispatch, billing and customer support across three systems that don't talk to each other. Their CRM holds account terms and contact history. Slack carries the actual day-to-day decisions, driver reassignments, rate exceptions, the things nobody writes down formally. A shared spreadsheet tracks fuel surcharge overrides that finance updates weekly.
A dispatcher fielding a customer call needs facts from all three, fast, and usually gets them by asking a colleague who has been there longer. New hires take months to get fluent in "which spreadsheet is current" and "who actually knows the answer." That is the real cost of scattered systems: not that information doesn't exist, but that finding the current, correct version of it depends on tenure and luck.
Why the obvious fix doesn't work
The instinctive answer is to point an AI assistant at everything: export the CRM, index Slack, ingest the spreadsheet, and let a chatbot answer questions across all of it. Northfield tried a version of this first, with a general search tool pointed at their Google Drive and CRM export.
It worked, briefly, and then produced two kinds of failure. First, answers went stale. The fuel surcharge spreadsheet gets a full re-index only once a week, so an assistant answering a Tuesday question was often quoting Thursday's rate. Second, and worse, a support contractor asked the assistant about a specific account's payment history and got a complete answer, including a note about a dispute that only the finance team was meant to see. The search tool had indexed everything with a single access level, because building genuine permission checks per document was more work than the initial build budget allowed.
Why does permission-aware retrieval matter more than search quality?
Because a wrong answer is a nuisance, but a leaked one is a liability. A search tool that finds the right document 95% of the time but occasionally shows the wrong person the right document has actually made things worse than not having AI search at all, since it has created a new way for the mistake to happen at machine speed. Even the vendors built for enterprise scale, whose revenue growth reflects real demand for this category, describe permissioning as core infrastructure rather than a bolt-on. Permission aware rag is not a nice-to-have layered on top of good retrieval. Without it, retrieval quality is the wrong thing to be optimising for first.
"LLM plugins processing untrusted inputs and having insufficient access control risk severe exploits like remote code execution."
OWASP Top 10 for Large Language Model Applications, on insecure plugin design
That warning is written about plugins calling external systems, but the underlying point transfers directly to retrieval: a tool that hands a model broad access and trusts it to behave is making the same mistake in a different shape.
The after: what actually changed
Northfield rebuilt the setup around three decisions, not a bigger model.
They connected three sources with explicit scopes, not one blob. The CRM's account terms table got a scope of accounts:read, open to everyone customer-facing. The dispute notes, which lived in a separate table in the same database, got finance:disputes, held only by the finance team and two account managers. The spreadsheet became a proper source with a six-hour freshness window instead of a weekly export.
Retrieval checked entitlement before anything was drafted. When the same contractor from the earlier failure asked about that account again, the dispute note scored zero for them and never reached the model that wrote the answer. Not redacted afterward. Never retrieved.
Freshness became a property of the data, not the pipeline schedule. The surcharge spreadsheet source re-checks itself every six hours rather than waiting for a weekly batch job, so a Tuesday question gets a Tuesday answer, or an honest failure state if the re-check itself fails, rather than a confident but outdated one.
| Dimension | Before (general search over an export) | After (condensed, entitlement-aware knowledge base) |
|---|---|---|
| Where permissions are checked | Not checked, single access level for all indexed content | At retrieval, before synthesis, per asker |
| Freshness | Weekly re-index regardless of how often data changes | Per-source TTL, re-fetched from source when stale |
| Failure mode on a stale fact | Confidently repeats it as current | Returns a labelled failure rather than a guess |
| Setup effort | One-off export and index | Per-source scope and TTL decision, revisited as sources change |
| Who can answer "why did it say that" | Nobody, the index is opaque | The retrieval log shows which memory object matched and why |
Table 1: what changed for Northfield Freight between the general search attempt and the entitlement-aware rebuild.
A day in the new system
A dispatcher asks the assistant what a specific account's current fuel surcharge is. The retrieval step matches the spreadsheet source, checks that the dispatcher holds the required scope (open to all dispatch staff), confirms the object was fetched two hours ago against a six-hour TTL, and answers with the current rate and when it was last confirmed. The same afternoon, the same dispatcher asks about a dispute on a different account. The dispute note scores zero for them, because they don't hold finance:disputes, and the assistant reports that it found nothing it could show them for that query, rather than a vague non-answer that leaves them unsure whether the information exists at all.
How do you decide what needs a scope and what doesn't?
Start from harm, not from system boundaries. Northfield didn't scope every table in their CRM separately, because most of it (contact names, account tiers, delivery addresses) carries no real sensitivity and restricting it would only slow people down for no benefit. They scoped the two tables where a wrong disclosure actually costs something: payment disputes and driver personal details. A scope you add "just in case" without a specific harm in mind usually means someone locked out of information they needed, with no corresponding safety gained.
Common pitfalls in a project like this
- Ingesting everything before deciding who should see what. It is much harder to retrofit scopes onto a fully ingested system than to decide them per source as you connect it.
- Treating a weekly export as good enough because nobody complained yet. Staleness complaints are usually silent. People stop trusting the tool rather than reporting why.
- Assuming a smaller company doesn't need entitlement at all. Northfield is 90 people. The dispute note issue would have caused real damage at a tenth that size.
- Skipping a test from another role's point of view. The leak in the first attempt was found by a customer complaint, not by the team testing their own build. Testing as a lower-privileged member before launch would have caught it for nothing.
Northfield's rebuild used self hosted ai memory running against their existing Postgres instance, connecting the CRM directly rather than through an export, which is also what closed the staleness gap: a live connection can be re-checked, an export cannot. If you're evaluating ai knowledge management software for a similarly-sized team, the two questions worth asking any vendor are the ones this case study turned on: where is entitlement checked, and what happens when a fact goes stale. Contextely's own answer to both is in the security model, and the pricing page shows what running this at Northfield's scale actually costs.