SemanticFS — Local-First Semantic Search for Personal Documents

Overview

SemanticFS is a local-first semantic search system designed for personal documents and knowledge archives. Instead of relying on exact keyword matching, the system allows users to search files based on meaning and contextual similarity.

The project focuses on offline retrieval infrastructure rather than cloud-dependent AI tooling. Documents are indexed locally, embeddings are generated on-device, and semantic retrieval is performed entirely on the user’s machine.

The system was built as a CLI-first retrieval engine with a modular indexing pipeline and inspectable storage architecture.

---

Problem

Traditional file search breaks down when users remember concepts instead of exact filenames or keywords.

For example:

• remembering “that note about incremental indexing”
• recalling a document by meaning instead of wording
• searching across fragmented folders and inconsistent naming structures

Most semantic retrieval tools also introduce additional problems:

• dependency on cloud APIs
• privacy concerns for personal documents
• opaque storage systems
• limited control over indexing behavior

I wanted a retrieval system that could:

• work fully offline
• remain inspectable and debuggable
• support incremental indexing
• separate metadata storage from vector retrieval
• provide semantic search without external services

---

Solution

I built a local semantic indexing pipeline in Python.

The system scans folders recursively, extracts text from supported files, chunks content into searchable segments, generates embeddings locally, and stores both metadata and vector representations for retrieval.

The architecture intentionally separates concerns:

• SQLite handles structured metadata
• ChromaDB handles vector similarity search
• sentence-transformers generates embeddings locally
• the CLI orchestrates indexing and querying workflows

Search results return:

• similarity score
• chunk preview
• source file path
• rule-based “why matched” explanations

The project prioritizes transparency and controllability over abstraction-heavy AI workflows.

---

Architecture

Indexing Pipeline

Scanner
  → Parser
  → Chunker
  → Embedder
  → SQLite Metadata Store
  → ChromaDB Vector Store

Each stage is isolated to keep the system extensible and debuggable.

---

Metadata Layer

SQLite is used for:

• indexed file tracking
• chunk metadata
• indexing history
• file hashes
• incremental indexing logic

This keeps metadata queryable and easy to inspect manually.

---

Vector Layer

ChromaDB stores embeddings separately from metadata.

This decision keeps retrieval concerns isolated from relational storage and allows future replacement of the vector backend without rewriting indexing logic.

---

Incremental Indexing

The system computes SHA-256 hashes for indexed files.

If a file has not changed since the previous indexing run, the system skips reprocessing entirely.

This avoids unnecessary embedding generation and significantly reduces indexing cost on large document collections.

---

PDF Handling

The parser currently supports text-based PDFs only.

Scanned/image-only PDFs intentionally fail with explicit messaging instead of silently producing empty results.

This was a deliberate design decision to avoid unreliable OCR assumptions inside the indexing pipeline.

---

Key Engineering Decisions

Local Embeddings Instead of Cloud APIs

I intentionally avoided OpenAI-style embedding APIs.

Reasons:

• offline usage
• privacy
• deterministic local behavior
• no per-query cost
• reduced infrastructure dependency

---

SQLite + Chroma Separation

Instead of storing everything inside a single vector database, I separated:

• structured metadata
• semantic vectors

This improves inspectability and keeps retrieval architecture modular.

---

CLI-First Product Direction

I chose to validate the retrieval engine before building a desktop or web interface.

The CLI-first approach made it easier to:

• debug indexing
• benchmark retrieval quality
• inspect chunk outputs
• iterate on architecture quickly

---

Rule-Based Match Explanations

Search results include lightweight “why matched” explanations.

I intentionally kept this deterministic instead of using LLM-generated explanations to avoid:

• extra latency
• hallucinated reasoning
• additional API dependencies

---

Validation & Benchmarking

The project includes:

• retrieval benchmarks
• chunk inspection tooling
• indexing summaries
• parser failure reporting
• automated test structure

This was important because semantic retrieval systems can appear functional while producing poor chunk quality or unstable matches internally.

The project focuses on validating retrieval behavior rather than only producing demo outputs.

---

Current Limitations

Current limitations include:

• no OCR support for scanned PDFs
• no real-time filesystem watcher
• no hybrid keyword + semantic ranking
• no distributed indexing support
• CLI-only interface

These limitations are intentional tradeoffs during the current architecture stage.

---

Future Directions

Potential future expansions include:

• OCR pipeline support
• hybrid retrieval ranking
• local desktop interface
• filesystem watcher daemon
• embedding model swapping
• semantic tagging pipelines
• retrieval feedback loops

The long-term direction is evolving the project into a controllable local retrieval infrastructure rather than a thin AI wrapper.

---

Reflection

This project was less about “adding AI to search” and more about designing a retrieval system with clear boundaries and infrastructure decisions.

The most important part of the project was not embedding generation itself, but:

• storage separation
• indexing strategy
• retrieval transparency
• incremental processing
• local-first architecture decisions

It became an exercise in building inspectable AI infrastructure instead of opaque automation.