Source Registry in Obsidian: Controlled RSS Signals

The first ingestion of our controlled RSS fixture ended with the message created 1. The second run reported reused 1.

Those two lines look less impressive than an AI-generated summary or a finished Digest. Yet they verified one of the most important properties of Engineering Radar: running the same operation again does not create a second version of the same event. The system retains the Signal identity regardless of how many times we read the source.

In the first article in this series, I explained why we built the Radar as a local-first system in Obsidian and did not give AI autonomous publication authority. This time, we will examine the first technical boundary: turning a set of RSS URLs into a controlled Source Registry and feed items into structured, deduplicated Signals.

The central idea is simple: RSS is only transport. Reliability begins with a source contract, provenance, stable identity, bounded recovery, and a clear lifecycle for full source text.

A URL is not yet a Source

A conventional RSS reader needs little more than a feed URL and a name. Engineering Radar has to answer more questions.

• Is this an official source or a community publication?
• What role does it play: primary, discovery, engineering blog, or research?
• Are we allowed to use it?
• How often should it be checked?
• Is it Critical?
• When did the last successful check happen?
• Is the source blocked after an error or policy review?
• Which HTTP mechanisms are available for conditional requests?

A Source in our Vault is therefore not a line in a configuration file. It is a Markdown file with validated frontmatter and a human-readable description of the rules.

The contract can be divided into five groups:

1. Identity: entity type, stable ID, schema version, name, and URL.
2. Trust: role, source type, authority score, and legal assessment.
3. Operations: priority, collection method, enabled state, cadence, and rate-limit policy.
4. Health: last attempt, last success, error state, and blocking reason.
5. HTTP recovery: status, ETag, Last-Modified, and the last response timestamp.

The Source Registry reads these files, validates their schema, and builds a runtime snapshot. A file with an invalid URL, an unknown collection method, or incomplete frontmatter never becomes an active Source. A duplicate ID is not resolved using “last file wins” either: the registry records a validation issue pointing to the first declaration.

This is an important shift. If a source influences a public conclusion, its rules should not exist only in the author’s head or inside adapter code.

Five production Sources and one fixture

For v0.1, we fixed the completed scope at five official sources:

• OpenAI;
• React;
• Next.js;
• GitHub Changelog;
• Cloudflare.

The Sources directory contains a sixth file: a mock RSS Source. It is not a sixth production Source.

The mock has a manual cadence, does not use the network, and does not call OpenAI. Its purpose is to verify validation, ingestion, stable identity, and atomic file creation deterministically. It supplies the same input regardless of external-site availability or changes to a real RSS feed.

The distinction matters in both documentation and metrics. “There are six Source files” describes an inventory. “The Radar processes five production Sources” describes the actual scope. Combining the two would present a test fixture as production coverage.

We moved expansion to twenty sources into the post-MVP backlog. Five sources are enough to validate the working flow and several external feeds. They are not enough to claim proven scalability for any number of adapters.

Source as a trust boundary

Even a valid Source is not necessarily eligible to run. The runtime respects business state:

• the Source must be enabled;
• blockedReason must be empty;
• legalAssessment must be approved;
• its cadence must indicate that a check is due.

This protects against a common automation error: a technically accessible endpoint should not automatically be considered permitted.

For OpenAI, for example, we approved the official RSS feed. Page extraction does not become an allowed fallback merely because the feed contains only a short excerpt. Another publisher may have different rules, so the policy stays next to the Source record.

Priority is also part of the contract. After downtime, the Radar checks Critical Primary and Discovery Sources first, then the rest. One Source failure must not stop the entire collection run: each source is processed in isolation, and the recovery report shows passed and failed counts separately.

The Source Registry is therefore not a bookmark catalogue. It is a controlled trust boundary between the external internet and a private Vault.

From an RSS item to a stable Signal

When a Source is approved and due, ingestion passes through several checks.

First, the Radar performs an HTTP request with a timeout. If the feed previously returned an ETag or Last-Modified, it includes conditional headers. A 304 Not Modified response ends the check without repeating XML parsing.

For a new response, the system validates the HTTP status and content type. XML parsing then validates every item: guid, title, canonical link, observation date, and description.

Next, the Radar creates a stable identity. Conceptually, it looks like this:

identityInput = sourceId + canonicalUrl + sourceItemId
signalId = observedTimestamp + stableShortHash(identityInput)

if signalPath exists:
  return reused

write temporary file
rename temporary file to signalPath
return created

The timestamp keeps the ID readable and helps sorting, while the hash binds it to the Source and item. The article title is not part of the identity: a publisher can correct a headline without turning the same event into a new Signal.

File creation is atomic. The Radar first writes a temporary file, then checks whether the target appeared concurrently, and only then performs the rename. If another process already created the Signal, the temporary file is removed and the outcome becomes reused.

A per-Source lock also prevents two checks of the same Source from running simultaneously inside one process. The main idempotency guarantee, however, lives in the file identity rather than only in plugin memory.

Technical note. The current v0.1 runtime rejects RSS responses larger than 2 MB and processes no more than 50 eligible items in one Source check. These are safety limits for this version, not universal recommendations for every Radar implementation.

Provenance matters more than a convenient date

An RSS pubDate often looks convincing enough to record immediately as a publication date. We deliberately do not do that.

The date from the feed is stored in a Signal as an rss-observation. The publishedAt field remains empty until we have a verified first-party structured date.

The reason is not limited to poor-quality feeds. A date can represent publication, an update, syndication, or a feed rebuild. If the Radar turns the observation into a fact, the error propagates into selection, the Digest, sorting, and public MDX.

The Signal therefore stores provenance in separate fields:

• Source ID and Source item ID;
• requested URL and canonical URL;
• collection method;
• fetched-at timestamp;
• HTTP status;
• path to the Source record;
• validation state.

This makes uncertainty visible. An empty publishedAt with an explanation is better than a precise but unsupported date.

Recovery after downtime without replaying scheduler history

Local Obsidian is not a high-availability service. A laptop may sleep, Obsidian may be closed, and the network may be unavailable. Recovery is therefore a normal operating condition, not an exception.

The most naive implementation would replay every missed scheduler run. If a Source had a six-hour cadence, two days of downtime could trigger eight nearly identical requests. That adds failure states and load without increasing coverage.

We chose a different policy:

coverageFrom = max(lastSuccessfulCheck, now - 7 days)

The Radar performs one current check for each overdue Source and considers eligible items since the last success, but never looks back more than seven days. It then reports the downtime period, overdue count, passed, failed, and created results.

During the live v0.1 recovery, the system found five overdue Sources. The outcome was passed 5, failed 0, and created 5. The following scheduled check reported overdue 0.

This confirms the recovery flow. It does not guarantee that the seven-day window can never miss an older event. The cap is a deliberate trade-off among relevance, request load, and the local nature of the system. A business that needs complete historical backfill requires a separate workflow with its own rules.

Full text is temporary context, not an archive

A Signal does not retain an unrestricted copy of someone else’s publication. Its durable part is a structured artifact: identity, provenance, observation, analysis state, and a link to the primary source.

Full retrieved text may exist only in Radar/Temporary Content/. It is present while the AI analysis or Digest generation needs context. All full texts are deleted when the weekly cycle closes.

This rule addresses several concerns at once:

• the Vault does not become an uncontrolled content archive;
• backups do not contain temporary third-party texts;
• retention can be verified independently;
• cleanup cannot cross into Site Content and touch published MDX.

After the final R2 cleanup, the retained full-text count was zero. That is the evidence we need: not a statement that “we delete the texts,” but a verified empty temporary boundary after closure.

How full text is sent to the model, which fields structured analysis returns, and where the prompt-injection boundary sits are topics for the next article.

From the first spike to production scope

We did not begin with five external feeds. The sequence was deliberately small:

1. Create and validate a Source file.
2. Install a deterministic mock Source.
3. Run ingestion and receive created 1.
4. Repeat it and receive reused 1.
5. Add one official OpenAI RSS feed.
6. Expand the registry to five approved official Sources.
7. Add conditional HTTP, timeout, Source locking, a recovery window, and failure isolation.
8. Verify live recovery and a healthy scheduled state.

The mock fixture proved idempotency and the file boundary. Real feeds introduced what the fixture could not show: HTTP errors, content-type mismatches, remote format changes, conditional requests, and policy differences.

This is a good example of the MAE approach: a spike does not pretend to prove production readiness. It closes one specific uncertainty. The next increment then adds a contract and new evidence.

What the decision provides—and what it does not

The advantages of a file-based Source Registry are concrete:

• Source records remain readable without the plugin;
• policy, cadence, and health history stay together;
• idempotency does not depend on process memory;
• recovery has a bounded and explainable scope;
• every Signal retains provenance;
• a mock fixture verifies ingestion without the network.

The disadvantages are concrete as well:

• a Markdown registry requires strict schema validation;
• atomic file operations are more involved than a simple append;
• RSS feeds contain format edge cases;
• a local Obsidian runtime does not provide continuous collection;
• the seven-day cap deliberately excludes older events;
• five Sources do not prove scalability to the post-MVP scope of twenty.

For our scenario, this is an acceptable trade-off. We optimized for a controlled stream of current engineering Signals, not for an exhaustive archive.

A minimal Source Registry for your Vault

If you want to reproduce the approach without our plugin source, start with deterministic ingestion rather than the OpenAI API.

1. Create Sources/, Signals/, and Temporary Content/ directories.
2. Define a Source schema and reject invalid records and duplicate IDs.
3. Add one local fixture without network access.
4. Define stable identity before the first ingestion.
5. Verify the sequence created 1 → reused 1.
6. Add one official Source with an explicit usage policy.
7. Bound timeout, response size, and items per run.
8. Add a per-Source lock and failure isolation.
9. Define the recovery window and temporary-text retention.
10. Only then add AI analysis.

v0.1 evidence snapshot

• Official production Sources: 5.
• Mock fixture: 1.
• Source Registry tests: 27.
• Source Registry commands: 10.
• Signal-file inventory at snapshot: 49.
• Recovery cap: 7 days.
• Live recovery: 5 passed / 0 failed / 5 created.
• Post-recovery overdue: 0.
• Retained Temporary Content: 0.

The 49 Signal files are an inventory at the evidence snapshot, not the number of Signals in the production Digest. The number of commands is not evidence of system value either; it only describes the size of the operational surface.

The next layer is analysis

The Source Registry answers these questions: where did a Signal come from, are we allowed to use the source, have we seen the item before, and which provenance can we prove?

It does not yet answer the harder question: what does this change mean for an engineering team?

In the next part, we will examine Structured AI analysis: how to send temporary context to a model, receive a schema-bound result, defend against prompt injection, and keep severity and Recommended Action under human control.