SpiderMind is a personal web intelligence system built around a simple premise: a signal is only useful when it can be connected to the person receiving it.
Instead of producing another generic feed, SpiderMind maintains a living model of my projects, goals, technical interests, work context, public positioning, and topics I deliberately avoid. It uses that context to decide what to search for, filters incoming signals, explains why selected items matter, and prepares action drafts without acting on my behalf.
The system is deployed as a single-owner production application. It is intentionally not a multi-user SaaS product yet.
The problem
Keyword alerts, feed readers, and generic AI digests can find content, but they usually stop at retrieval. They do not know whether a GitHub issue is relevant to an active project, whether a discussion supports a current learning goal, or whether a job post creates meaningful career leverage.
This creates two failures:
- High recall, low usefulness. The user receives many technically related links but still has to determine why each one matters.
- Static search profiles. Queries become stale as projects, goals, and work context change.
The difficult part was therefore not collecting more links. It was testing whether the system could reliably turn signal × personal context into a useful, grounded opportunity brief.
How the system works
Context inputs
→ append-only memory events
→ current personal-context read model
→ source-specific query generation
→ multi-source collectors
→ noise filtering and cross-run deduplication
→ Haiku relevance prescreen
→ Sonnet Opportunity Brief
→ tiered inbox, feedback, and draft-only actions
1. Living personal context
SpiderMind stores identity, projects, goals, interests, work context, positioning, avoided topics, and recent external activity as independent memory items.
The memory layer is event-sourced. Updates append events such as created, updated, status_changed, archived, and conflict_detected; existing records are not silently overwritten. A deterministic read model replays these events to produce the active context used by the rest of the system.
Incoming information can come from several paths:
- Manual context management in the dashboard
- A conversational context studio
- Read-only ingestion from my portfolio database
- My own GitHub and X activity
- Derived recurring activity topics
When incoming content conflicts with an existing identity-level fact, SpiderMind preserves both sides and requires an explicit resolution. Volatile changes such as engagement metrics or evidence growth are treated as updates rather than false conflicts.
2. Context-driven query generation
Collector queries are derived from the live read model instead of a fixed list of keywords. A context change can therefore alter the next scan without editing collector configuration.
Each source has its own query grammar and budget:
- GitHub and Hacker News prioritize technical topics rather than branded project names.
- X builds bounded OR queries and adds negative terms from avoided topics.
- RSS and job feeds use broader phrase and token matching.
- Hugging Face uses separate search lanes for models, datasets, spaces, and daily papers.
- Stack Exchange maps context into site and tag-oriented searches.
This separation matters because one generic query strategy produced different failure modes across sources. Search quality had to be handled per source rather than hidden behind a single abstraction.
3. Multi-source collection and orchestration
The production configuration supports seven active sources:
- GitHub Issues and Discussions
- Hacker News
- X
- RSS feeds
- Job feeds
- Stack Exchange
- Hugging Face Hub and Papers
A Reddit collector also exists, but remains disabled because external API access was not available. I kept the adapter passive instead of presenting an unverified integration as complete.
Collectors normalize their results into a shared raw-signal shape. The pipeline then applies source-specific recency windows, noise filtering, per-source limits, and URL-based deduplication across previous runs.
Scans can run manually or through per-source APScheduler jobs. Overlapping scans from the same source are blocked, and the LLM phase is serialized across sources to avoid rate-limit, cost, and database collisions.
4. Cost-aware relevance and brief generation
Sending every collected item to the strongest model would be expensive and would amplify noise. I split the LLM pipeline into two stages:
- Haiku prescreen: scores each signal from 1–10 and gives a short relevance reason.
- Sonnet brief generation: runs only for signals that pass the calibrated threshold, plus a controlled review band for selected sources.
I chose a lightweight reasoning model instead of embedding similarity for the first stage. Embeddings are cheaper, but primarily measure topical proximity; SpiderMind needed to judge whether a signal connects to a concrete project, goal, or work context and explain that decision.
Each accepted signal becomes a bilingual Opportunity Brief with 13 structured sections, including:
- What happened
- Why it matters generally
- Why it matters to me
- Linked personal context
- Opportunity type and opportunity map
- Suggested entry angle
- Draft response
- Risks to avoid
- Follow-up possibilities
- Recommended action
Briefs receive one of four tiers: Act Now, Save & Develop, Watch, or Ignore. Tier assignment is not left entirely to free-form generation. Weak personal-context links and stale or closed signals can deterministically lower the tier.
5. Human-in-the-loop authoring
SpiderMind can generate context-aware X replies, original posts, threads, project-link suggestions, and optional draft images. It can also ingest published posts and engagement metrics to connect a draft with its eventual outcome.
However, the system has no posting path.
This is a permanent product boundary rather than a missing feature. The frontend exposes copy and review actions, not send actions. The repository also includes a structural draft-only test that guards against introducing posting behavior into the UI or authoring code.
The same approval rule applies to context chat: the LLM can propose creates, updates, archives, or conflict resolutions, but memory changes occur only after the owner explicitly applies a proposal. Chat sessions, turns, and apply/reject outcomes are stored append-only.
Production architecture
Backend
- Python 3.13
- FastAPI and Pydantic
- SQLite with a persistent Railway volume
- APScheduler for source-specific jobs
httpx and dedicated clients for external sources- Anthropic Claude for prescreening, briefs, context chat, and text authoring
- Optional Gemini image generation
- Structured application and pipeline logs
- Single-owner HMAC-signed bearer authentication
Frontend
- Next.js 16
- React 19
- TypeScript
- Tailwind CSS
- Views for scans, briefs, X authoring, context management, conversational context, and operations
External connections
- GitHub API
- Hacker News Algolia API
- X API
- RSS and job feeds
- Stack Exchange API
- Hugging Face API
- Read-only Neon PostgreSQL connection to the personal website database
- Railway backend deployment
- Vercel frontend deployment
Key decisions and tradeoffs
Prove brief quality before source breadth
The first version used only GitHub. Building seven collectors before validating the core assumption would have multiplied bad outputs rather than reduced product risk.
The go/no-go criterion was explicit: at least five of ten briefs had to be genuinely useful, with no more than two complete misses. Only after the initial pipeline passed that gate did I expand source coverage.
Preserve history instead of overwriting context
A personal context system changes over time. Direct row updates would make it impossible to understand why the system believed something or how a conflict appeared. Event sourcing introduced more repository and replay logic, but preserved history and enabled explicit conflict resolution, soft deletion, restoration, and auditability.
Use SQLite until multi-user requirements are real
The product is currently a personal tool. SQLite provides simple local development and a low-operations production path through a persistent volume. PostgreSQL is used only as a read-only external source for portfolio content. A full application-database migration is deliberately postponed until multi-user requirements justify it.
Separate raw signals, model outputs, and feedback
Raw signals, relevance scores, briefs, feedback, authoring drafts, engagement data, and memory events are stored separately. This avoids destructive transformation and makes pipeline failures, model decisions, and later outcome analysis inspectable.
Delay adaptive scoring until there is enough fuel
The infrastructure can collect feedback and engagement outcomes, but learned ranking is paused. The available feedback volume is still too small to justify changing production ordering. The next step is shadow scoring, followed by a gated comparison against the existing quality threshold—not immediate automated promotion.
Validation and outcomes
The production quality gate reviewed ten generated briefs:
- 7 useful
- 2 borderline
- 1 junk
This passed the original five-of-ten continuation threshold.
Manual owner review also marked:
- 11 of 11 reply drafts as publishable
- 10 of 11 original-post drafts as publishable
These are internal quality evaluations, not claims of external user adoption.
At the latest completed documented sprint, the backend regression suite contained 548 tests, alongside TypeScript checks and the dedicated draft-only invariant test.
Current boundaries
SpiderMind is production-usable for a single owner, but several constraints are intentional:
- Adaptive scoring remains paused until enough feedback and engagement data accumulate.
- Reddit remains disabled until API access is available.
- Automated external posting is permanently excluded.
- Multi-user architecture and application-level PostgreSQL are postponed until the personal workflow proves durable value.
The project is useful because it does not merely summarize the web. It maintains an inspectable model of changing personal context and uses that model to decide what is worth attention, why it matters, and what a safe next action could be.