Enterprise Knowledge Repository

Overview

A company-wide AI knowledge repository: 8,000+ SharePoint files turned into a searchable semantic index, so anyone in the organization could ask a question and get the right document passage instead of a filename guess. Built end to end on Azure, rolled out to 5,000+ users globally.

The unglamorous truth of enterprise search is that the ingestion pipeline is the product. Any tutorial can embed a folder of PDFs once. The real problem is that SharePoint is alive — people upload, edit, rename, and delete files all day — and a semantic index that drifts out of sync with reality is worse than no index, because people trust it.

So the thing I actually engineered here wasn’t retrieval. It was freshness.

Architecture — the delta-sync loop

flowchart LR
    SP["SharePoint<br/>8,000+ live files<br/>edited all day, every day"]
    subgraph AZ["Azure Logic Apps — the watchers"]
        POLL["Poll the directories<br/>every ~5 minutes"]
        DIFF{"What<br/>changed?"}
    end
    MAP[("Path → chunk map<br/>every file knows<br/>its own vectors")]
    EMB["Extract · chunk · embed<br/>just that file"]
    subgraph W["Weaviate"]
        UPD["Surgical update —<br/>only the changed slices"]
        IDX[("Semantic index")]
    end
    SP --> POLL --> DIFF
    DIFF -->|created / updated| EMB --> UPD
    DIFF -->|deleted| UPD
    MAP -.-> UPD
    UPD --> IDX

A fleet of Azure Logic Apps sits watching the SharePoint directories, checking roughly every five minutes. When a file is created, updated, or deleted, the pipeline doesn’t shrug and re-index eight thousand documents — that would be absurd at this scale. Instead, we stored the path and metadata for every chunk, so each file knows exactly which vectors in Weaviate belong to it. A changed file re-embeds only itself; a deleted file surgically removes only its own slices.

That one mapping decision is what made the index live instead of stale. Whatever happened in SharePoint was reflected in search within minutes, forever, without anyone babysitting it.

The serving path

flowchart LR
    UI["React front end<br/>5,000+ employees"]
    API["Search service<br/>Node"]
    W[("Weaviate<br/>semantic index")]
    UI -->|question| API -->|embed + retrieve| W
    W -->|passages + sources| API -->|under 500ms| UI

Retrieval is semantic, so the passage that answers the question surfaces even when the wording never matches the document. Answers come back with sources attached — a passage you can’t trace is a passage nobody in an enterprise will act on — and the round trip stays under 500ms, because search only gets adopted if it feels instant.

The front end is a React app the whole company hits. I treated ingestion, retrieval, and UI as one product, not three components — that’s the difference between “we have a vector database” and “five thousand people actually use this.”

Engineering decisions

• Logic Apps as the ingestion backbone — SharePoint connectors, scheduling, and failure handling came from the platform instead of custom pollers. More than one person can maintain this pipeline, which is a feature.
• Delta sync over re-index — the path→chunk mapping means the cost of an update scales with the change, not with the corpus. This is what makes 8K+ living documents sustainable.
• Sources on every answer — trust is the adoption currency in enterprise search; citations are non-negotiable.

Highlights

• 8,000+ SharePoint files in a continuously fresh Weaviate index
• ~5-minute sync latency from a SharePoint edit to searchable
• 5,000+ users company-wide, <500ms query responses
• Zero-babysitting ingestion — the watchers and the chunk map do the maintenance