Most guides to self hosted ai memory skip straight to a docker run command and leave out the two decisions that actually determine whether the result is useful: what you connect first, and who is allowed to see it. This is a walk-through of both, written at the level of what you configure and why, rather than a copy-paste command list that goes stale the moment a flag name changes.
Self hosted ai memory means running a context layer, on your own infrastructure, that keeps a condensed, current copy of a handful of your company's systems of record and answers questions against it with per-asker permissions enforced. That is a different product to a vector database with a chat interface bolted on, and the setup steps reflect the difference.
What you need before you start
Three things, and nothing more exotic:
- A container runtime. Docker and Docker Compose on a single machine, virtual or physical, that can run 24 hours a day. This does not need to be large. A company of 200 people generates far less retrieval traffic than a single busy web app.
- A Postgres database. Version 14 or newer. Docker Compose can bring one up for you, or you can point the deployment at a database you already run, such as a managed Supabase or Neon instance.
- An API key for one LLM provider. This is the one genuine external dependency. Condensing a raw record into a memory object is a real language model call, and a self-hostable product that pretended otherwise would be lying about where the intelligence comes from.
Nothing else is required to get a working deployment answering questions. A login dashboard is a convenience, not a dependency: the retrieval API and any MCP-speaking agent work on API keys alone.
Step 1: bring the container up
Docker Compose starts the application container alongside its database, so a first run needs exactly two things set as environment variables: a database connection string, and the LLM provider key. Once both are present, docker compose up --build should give you a running service on a local port within a couple of minutes, most of that being the image build the first time. If you have never used Docker Compose before, its own documentation covers the handful of commands you actually need day to day, and none of the rest is specific to a context layer.
Two details are worth knowing before you flip this switch. First, the database credential is read server-side only and never reaches a browser, so there is no way for it to leak through a client bundle. Second, if the LLM key is missing or wrong, ingestion should fail loudly with a clear error rather than silently writing a placeholder summary. A self-hosted tool that condenses your company's contracts and support tickets should never invent a fallback that pretends to have understood something it didn't.
Step 2: connect your first system of record
Pick one source, and pick a boring one. A read-only view over a CRM table, or a small Postgres schema you already trust, is a better first connection than your entire Slack history. You are learning the shape of the tool at the same time as configuring it, and a narrow blast radius makes mistakes cheap.
A source connection needs two things defined: how to list records once, and how to re-fetch a single record later by its reference. For a database, that is genuinely just two SQL statements, both read-only, both scoped to the columns you actually want summarised:
| What you define | Why it exists |
|---|---|
| A list query | Populates the initial set of memory objects for that source |
| A fetch-by-reference query | Re-reads one record when its memory object goes stale |
| A time-to-live | How long a condensed object is trusted before it is re-checked |
| Required scopes | Which entitlement grants an asker needs to retrieve anything from this source |
Table 1: the four decisions a new source connection actually requires, independent of which system it points at.
For a tool that already speaks MCP itself, the same idea applies without you writing any SQL: you point the deployment at the other server's endpoint and name a list tool and a fetch tool, and the freshness loop calls them the same way it would call a database. This is also why a context layer built to be an MCP client, not only an MCP server, matters in practice: it can reach anything that already speaks the protocol without a bespoke connector.
How long does the first sync take?
Longer than a search index rebuild, because condensation is doing real work per record rather than just tokenising text for a vector store. Expect the first pass over a few hundred records to take minutes, not seconds, and expect it to call your LLM provider once per record it has not seen before. A second pass over unchanged records should be close to instant, because a source whose raw content hasn't changed simply has its freshness clock reset with no model call at all.
Step 3: set entitlement scopes
This is the step a lot of self-hosting guides quietly skip, and it is the one that decides whether what you have built is a company knowledge base ai tool or an accidental way for a support intern to read the payroll spreadsheet.
Entitlement in a well-built context layer is not inherited automatically from the source system's own access controls. You define it: a source can require a scope such as finance:payroll, and every memory object it produces inherits that requirement. A member holds grants, and a grant ending in a wildcard covers everything under it, so finance:* satisfies a requirement of finance:payroll without you having to enumerate every sub-scope by hand.
"Each user is assigned one or more roles, and each role is assigned one or more privileges that are permitted to users in that role."
NIST, Role Based Access Control project
That is the textbook description of role-based access control, and it is worth reading closely, because a lot of self-hosted tools stop at "roles exist" and never connect a role to a retrieval-time check. Setting scopes without verifying they are actually enforced at the moment a question is asked, rather than only at ingest, is the gap that turns a permissions screen into theatre.
Is self-hosting actually more secure than a SaaS tool?
Not automatically, and this is worth saying plainly. Self-hosting answers one question: does your company's data leave your network. It does not answer a second, separate question: does the retrieval step check who is allowed to see a given fact before it is used to answer anything. You can self-host a tool that has no meaningful entitlement model at all, and plenty of open source projects in this space do exactly that, leaving permission enforcement as a "coming soon" item. Ask specifically where entitlement is checked in the pipeline before assuming self-hosted means locked down.
A worked example
A 45-person recruitment firm sets this up on a Friday afternoon. They bring the container up against a managed Postgres instance they already pay for, connect a read-only view over their applicant tracking system as the first source, and set a TTL of six hours on it, since candidate status changes throughout the day but not by the minute. They require the scope ats:read on that source and grant it to everyone except two contractors who only handle a specific client account, who get ats:read:clienta instead. By Monday, their internal support bot can answer "what stage is this candidate at" correctly and instantly refuses to answer it for the contractors' excluded clients, because the retrieval step scores those objects at zero for them rather than returning a vague apology after the fact.
Common pitfalls when self-hosting AI memory
- Connecting a broad source before understanding TTLs. Bringing in your entire Drive on day one, with a badly-chosen freshness window, means either constant re-condensing or answers that go stale silently. Start narrow.
- Treating entitlement as a one-time setup task. Scopes need revisiting whenever a source or a role changes. Nothing propagates automatically from your CRM's own permission model, by design, because that keeps the logic auditable in one place.
- Assuming a missing LLM key degrades gracefully. It shouldn't, and if the tool you're running lets ingestion complete anyway with a low-quality guess, that is a warning sign about the rest of its design, not a convenience.
- Skipping the "ask as" check. Configuring scopes and never testing them from another member's point of view is the single most common way permission bugs ship unnoticed.
What's the cheapest way to self host ai memory?
Run it on a small existing server, point it at a Postgres instance you already have spare capacity on, and route condensation to a lower-cost model while you're testing. There is no licence fee and no seat count to pay for. Your only real running cost scales with how many records you condense and how often they need refreshing, not with headcount, which is the opposite of most enterprise search pricing.
If you would rather skip the infrastructure and start on the hosted free tier first, which covers 500 retrievals a month with the same entitlement engine, that is a reasonable way to learn the concepts before committing a server to it. Full setup instructions and the documentation cover the exact environment variables and connector details. If you're coming from a fully SaaS product and specifically want a self hosted glean alternative, the shape of this setup is the same regardless of which vendor you're replacing. When you're ready to compare hosted pricing against running it yourself, the pricing page has both side by side, and signing up costs nothing to try either path.