An AI Assistant Needs Reliable Sources

The pitch for an internal AI assistant is easy to like.

You upload your handbook, your service manual, your old support emails, your contract templates, your standard operating procedures. The assistant reads them. From now on, anyone in the team can ask it a question and get the answer drawn from your own documents instead of guessing or pinging a colleague.

That pitch is not wrong. The version that works really does feel like that.

The thing nobody warns you about is that the assistant is the easy half. The work that decides whether it ends up useful or quietly abandoned is the work nobody finds glamorous: looking at every document you fed it, asking whether it is true, asking whether it is current, and deleting most of it.

Reliable answers start with reliable sources

A team that buys an internal AI assistant tends to buy it as a tool. A search box. A chat window. Maybe an integration into the team's project tools.

But the tool is not really the thing being delivered. The thing being delivered is the answer the assistant gives to a member of staff at half past three on a Tuesday when they are trying to do a job.

That answer comes from documents. If the documents are out of date, conflicting, vague, or missing the bit the staff member actually needs, the answer is going to be out of date, conflicting, vague, or wrong. No amount of clever wording in the prompt will change that. The assistant cannot know more than its sources. It can only summarise them faster.

This is the part that surprises buyers. They expect the difficulty to be technical. The difficulty is editorial.

What ready source material looks like

Most internal documents were never written to be read by an assistant. They were written to be read by one specific person, on one specific occasion, often in a hurry. They contain in-jokes, unstated assumptions, references to people who left two years ago, and steps that have been replaced by a quieter informal process nobody bothered to document.

Before an assistant can reliably answer from them, four properties need to be in place.

Clarity. Each document should answer one question, or describe one process, or define one thing. Long documents that try to be three documents at once tend to confuse the assistant the same way they confuse a new starter. A page titled "Onboarding" that drifts into holiday booking, expense rules, and the office WiFi password will get cited oddly when somebody asks one of those questions in isolation.

Freshness. Old answers do not get more correct because they are written down. If the price changed, the policy changed, or the process changed, the old version needs to come out. An assistant reading two contradictory documents will pick one or quietly mix them. Either is worse than not answering.

Scope. The assistant should know what is in its remit. A staff handbook is not a legal opinion. A service manual is not pricing advice. The boundary needs to be in the documents, or in the prompt, or both. Otherwise the assistant will try to be helpful about things it has no business answering.

Ownership. Each document needs someone who is responsible for keeping it correct. This is the part that quietly fails first. People are happy to write a document once. They are less happy to be the person who reviews it every quarter. Without ownership, freshness rots within a few months of launch.

If those four properties are missing, the project is really a documentation project with an AI project sitting on top. That is fine. It is just better to know which one you have signed up for.

A staff policy assistant shows the problem

A while back we looked at a knowledge base that had been built up over a decade. Roughly four hundred documents. The team wanted an assistant that could answer staff questions from it.

The first version, with everything ingested, gave plausible answers that were quietly wrong about a third of the time. Old supplier names. Out-of-date approval thresholds. A policy that had been replaced two years earlier but never deleted.

The fix was not a better model. It was three afternoons of two people sitting with a list, marking documents as keep, archive, or rewrite. About 60% went to archive. The remaining 40%, properly tagged and lightly rewritten for clarity, made the assistant trustworthy. The model did not change. The sources did.

That is the pattern most knowledge-base AI projects hit. The first launch is too eager. The useful version comes after the prune.

Outdated sources produce outdated answers

When an internal assistant disappoints, the failure mode is usually one of four:

The sources contained things that should never have been ingested in the first place. Personal notes. Draft policies. Single-customer exceptions. Internal banter. The assistant is now confidently quoting a draft that was never agreed.

The team treats it as a search box and asks fact-finding questions, but the documents do not actually contain the facts. The assistant either answers anyway, which is dangerous, or admits it does not know, which is correct but feels like a failure.

The team asks it for judgement. "Should we accept this contract?" "Is this customer worth keeping?" Those are not questions for a knowledge assistant. The assistant can summarise the relevant policy. It cannot decide whether the policy applies.

Nobody owns the source review. Six months in, the documents have drifted, the answers have drifted, trust quietly evaporates, and people stop using it. The tool still works. The discipline around it stopped.

Check the source library first

Before commissioning an internal AI assistant, the most useful question to ask is not about the technology. It is about the documents.

Pick six questions you actually want it to answer. Find the documents that contain those answers. Read them. Ask: is this clear, current, in scope, and owned?

If the answer is yes for most of the six, the assistant has a chance.

If the answer is no, the project is really a documentation project. You can still do the AI work, but the order matters: clean the sources first, ingest second. Doing it the other way round teaches the team that the assistant is unreliable, which is a hard impression to undo.

AI still misreads missing context

A well-built internal AI assistant will still be wrong sometimes. It will misread a clause. It will quote one document when a more recent one exists. It will produce a confident sentence about a topic it should have said it could not answer.

That is why the useful versions all have the same boring habit: every answer carries the source it came from, so the reader can check. Not as a footnote. Not as a tooltip. Visible in the answer itself. If a team member cannot tell which document an answer came from, they cannot fact-check it, and over time they will stop trusting it.

For anything where the cost of being wrong is high, the assistant should still be a draft for a person to confirm. Pricing answers, contract questions, anything customer-facing, anything legal. The assistant prepares the answer. A person signs it off.

We prepare the sources before the assistant

Our most useful internal AI assistants are the ones built on top of well-pruned, well-tagged sources, not the ones with the cleverest wording.

We use a small in-house cross-agent memory tool to share notes between the AI workflows we run for ourselves, and a tidy handbook viewer for things the team needs to find quickly. Both work because someone is responsible for what goes in and what comes out, and because the entries are short, named, and dated. When that discipline slips, the answers slip with it.

So the warning is short. If you are buying an internal AI assistant, spend less time on the model and more time on the documents you are about to feed it. The model is mostly the same wherever you go. The documents are yours, and they are the part that decides whether the answer at half past three on a Tuesday is the one you wanted to give.