Knowledge with an expiry date: Keeping your AI assistant reliable

New rules, old answers? A practical workflow for current sources, effective access rights and reliable knowledge assistants, informed by Deepico’s experience.

Wissen mit Verfallsdatum: So bleibt Ihr KI-Assistent verlässlich

English edition — originally published in German as Wissen mit Verfallsdatum: So bleibt Ihr KI-Assistent verlässlich.

On Monday, the AI assistant correctly answers a question about purchasing approval. On Tuesday, a new rule takes effect. On Wednesday, it still explains the old one. The answer sounds plausible, even cites a source, and can nevertheless trigger the wrong action.

Making company knowledge accessible through AI therefore requires more than a collection of good documents. It must be clear which version applies, in which context, and who may see it. The practical test starts with change: new rules, new software versions, deleted sources and revoked permissions.

:::takeaways{title="For implementation"}

:::

What Deepico's experience shows

The published experience report on AI-assisted ERP system documentation describes a specific problem: an initial chatbot could not adequately account for different customer-specific modules and versions. According to the report, Deepico subsequently uses NotebookLM internally, reviews the generated content and adds customer-specific information.

This suggests an important design question: even a factually correct instruction can be unsuitable for the wrong system version. The workflow below is a proposed operating model for knowledge assistants. It does not describe an additional Deepico feature that has already been implemented.

First define the scope of applicability

Start with a manageable knowledge area, such as internal instructions for a particular software module. For each source, record an owner, its business approval status and its scope. These may include product version, organisational unit, language or customer group.

A modification date alone is insufficient. Correcting a spelling mistake yesterday does not make an old process rule valid again. Conversely, instructions unchanged for two years may still apply. What matters is business approval for the intended use.

If two sources contradict each other, a designated owner must resolve the conflict. Until then, the assistant should make the conflict visible or hand the question to a responsible person. More search results do not replace a decision about which rule is authoritative.

Check four stages of a change

In a typical solution, a source is ingested, prepared for search and then used as context for an answer. Copies and caches may sit between these stages. A change in the original document is therefore not proof that the next answer will already use the updated information.

Check four stages: Has the change been approved in the authoritative source? Has it been successfully ingested? Does search return only the version valid for the question? Does the answer actually refer to that version? Make the last successful synchronisation visible; starting a synchronisation run does not count as success.

Microsoft provides a specific technical example: Azure Storage indexers detect new and changed content using timestamps. Detecting deleted content, however, requires an appropriate deletion strategy. Without one, orphaned search documents may remain. This is a product-specific mechanism with a general test question: does a withdrawn source also disappear from the answer's knowledge base? Microsoft: Changes and deletions

Time-sensitive rules require a deliberate approach to delay. An organisation might, for example, decide that affected answers are paused and referred to the business team while a synchronisation has failed. The acceptable delay depends on the consequences of an outdated answer.

Permissions must take effect before the answer

An employee changes department and loses access to confidential documents in the document system. The knowledge assistant must honour that revocation too, including affected search extracts and stored answers.

The check belongs before content is passed to the language model. An instruction in the prompt not to mention confidential details is not access control. Nor does a source link that the person cannot subsequently open prevent the answer from already containing information from that source.

Microsoft documents several approaches to document-level control in Azure AI Search. For approaches using synchronised permission metadata, the effect of changes to source permissions depends on when that metadata is updated in the index. Some native approaches are explicitly marked as preview. A specific implementation therefore requires checking the mechanism, maturity and synchronisation behaviour. Microsoft: Document-Level Access Control

Also check whether answer caches and conversation histories continue to use content that the person may no longer access. Information already read cannot retrospectively be unread. For new answers and subsequent access, however, the application must enforce a clear rule.

Five acceptance tests for everyday operations

A useful test series does not need spectacular questions. It represents ordinary changes with clearly expected results:

| Test | Expected behaviour | |---|---| | A process rule is replaced | The new answer uses the approved version and cites it as the source. | | A document is withdrawn | Its content is no longer offered as valid guidance. | | A person loses read access | New answers and search extracts contain no restricted content. | | Two module versions differ | The assistant clarifies the version or uses the known context. | | Synchronisation fails | The outdated state is detected and the agreed restriction takes effect. |

Run these tests with authorised test data and different roles. Record the source, change, time and observed answer. Repeat them after relevant changes to search, the model, permissions or data connections. A set of questions answered successfully once does not establish that updates will keep working over time.

Knowledge maintenance belongs in the business case

Ongoing maintenance is part of the product. The budget therefore includes business approvals, resolving contradictory sources and investigating failed updates. Otherwise, the pilot looks inexpensive while the work later lands with support.

:::callout{type="info"} Hypothetical planning example: 60 relevant source changes per month each require an average of six minutes of expert review. That adds up to six hours. Add four hours for spot checks and incidents, and ten hours should be budgeted. At an assumed CHF 90 per hour, this represents CHF 900 of valued effort, in addition to platform costs. These are not measured Deepico figures. :::

Whether this maintenance makes economic sense depends on the whole process. Measure time to a usable answer, including follow-up questions, and the number of subsequent corrections. An assistant can answer quickly yet create more work if employees must check every statement for validity.

At the start, appoint a business owner for the sources and a technical owner for reliable ingestion. Together, they agree how changes, withdrawals and incidents are handled. This turns the knowledge base into a maintained working resource whose limitations users can understand.

Would you like to base a knowledge assistant on current sources with appropriate access controls? We can assess a bounded knowledge area and suitable acceptance tests with you.

Sources and interpretation

Research: 19 October 2026. The practical reference comes exclusively from the publicly available Deepico report. Microsoft's documentation establishes specific technical mechanisms, not universal product suitability. The operating model, test series and planning calculation are our editorial interpretation. Further reading: Integrating AI into existing systems.