Nine Production Failures That Changed the Architecture

English output changed tags or recommended actions.

This message stopped generation of the English Digest. The model returned text in the correct shape, but changed fields that had to remain shared between Ukrainian and English. The system detected the divergence and rejected the result.

The easiest fix would have been to ask the model again and hope for a more accurate answer. We did something else. Tags and Recommended Actions stopped being free-form model output. The English artifact began to inherit them forcibly from the approved Ukrainian Digest.

That change captures the entire first production cycle of Engineering Radar. The problem was not simply that AI sometimes makes mistakes. Production exposed places where the system still relied on an implicit expectation instead of a contract. We tried to turn every such case into an invariant, validation rule, test, or separate workflow.

In this article, “we” means the author and an AI coding agent. I defined the business rules, editorial authority, and acceptance gates. The agent helped analyze failures, change the implementation, and verify fixes. We worked with the MAE approach: instead of masking problems with another prompt or a manual workaround, we returned each observation to discovery, contract, implementation, and verification.

MAE did not make the first production cycle error-free. Its value was elsewhere: a failure did not remain an anecdote in a working chat. It became the next verifiable version of system behavior.

Nine incidents in three acts

We used the same structure for every case:

symptom
  → cause
  → fix
  → new contract
  → residual risk

This is more important than a catalog of error messages. A fix without a new contract can disappear during the next refactoring. A contract without a residual risk creates the false impression that one test has permanently closed an entire class of problems.

At the same time, it would be inaccurate to call all nine cases “AI mistakes.” Only some began with model output. The catalog contains at least four distinct classes:

• model behavior — the response is structurally valid but changes identity or has weak editorial quality;
• implementation gap — code reads stale state or validates against the wrong downstream limit;
• platform/runtime behavior — the Obsidian process did not load the new plugin instance;
• delivery coordination — a Git branch was based on state that another merge had already changed.

The classification changes the fix. A new prompt cannot repair a stale runtime process. Another Zod schema does not resolve a Git conflict. Human review should not compensate for a partial filesystem write. When a failure is classified incorrectly, the system receives another workaround instead of the required responsibility boundary.

For each mini-case, we therefore asked not “who is to blame?” but “which component should own this decision, and what repeatable evidence can prove its behavior?” That is how an operational observation becomes architectural input.

Act 1. Obsidian and runtime state

1. The command existed in code but not in Obsidian

After another plugin increment was installed, the expected command was absent from the Command Palette. The source files and built main.js already contained it, but the active Obsidian process was still running an older plugin instance.

The symptom looked like an implementation failure: the command had not been created. The cause was operational. Hot reload did not guarantee that the running process had actually loaded the new manifest and runtime bundle.

We reloaded the plugin and repeated a live smoke through the Command Palette. Since then, command registration in source or tests has not counted as proof that a user can already execute the command.

The new contract separates three states:

source updated
  ≠ plugin installed
  ≠ active Obsidian process reloaded

After an increment is installed, a runtime smoke is required: the command is visible, operates on the correct active file, and returns the expected notice. This is a platform-specific operational gate, not a universal Obsidian defect. The residual risk is also platform-specific: a desktop process can remain stale after its bundle is replaced externally, so the user guide must contain an explicit reload step.

2. Approval was written, but the next command read old state

During the Revision flow, the Ukrainian Digest moved atomically from draft-uk to an approved state. Immediately afterward, English generation could still see old frontmatter and respond that an approved or exported Ukrainian Digest had to be opened.

The file on disk was already correct. The stale state came from the active Obsidian object retained from the earlier operation. A sequence of commands exposed an invisible difference between “the plugin has written the file” and “the current in-memory representation has refreshed.”

The fix in Weekly Digest 0.3.1 stopped relying on the cached active file for this transition. English generation reads the current Digest directly through the Vault adapter and validates frontmatter again.

const current = await vault.read(activeDigestPath);
const digest = parseAndValidate(current);
 
assertState(digest, ["approved-uk", "exported-uk"]);

The new contract is that a state-changing command does not guarantee synchronous refresh of every UI representation. The next critical operation must read durable state from the Vault. This does not mean that UI cache is always wrong. It means approval, export, and provenance cannot depend on an assumption about its freshness.

The residual risk is a concurrent manual edit between read and write. Approval hashes and guarded atomic replacement address that risk, but a direct read alone does not.

Act 2. AI output and editorial contracts

3. The model changed tags and Recommended Actions

The opening incident occurred during English generation. The strict schema proved that tags and actions had the right types, but it could not prove that they matched the approved Ukrainian state.

The initial implementation asked the model to return those fields and compared them afterward. That turned deterministic identity into a probabilistic task. Even a strong language adaptation could reorder a tag, normalize its spelling, or propose an action that seemed more appropriate to the model.

In Weekly Digest 0.2.1, responsibility changed. The model owns localized prose. Identity fields are forcibly inherited from the Ukrainian source:

const english = {
  ...validatedModelText,
  tags: approvedUk.tags,
  recommendedActions: approvedUk.recommendedActions
};

Signal membership, order, URLs, and the action attached to each Signal still undergo strict checks. But the model no longer has authority to “improve” shared identity.

The new contract is simple: if a value can be inherited deterministically, it should not be delegated to an LLM. The residual risk moves upstream. If the Ukrainian reviewer approved an incorrect action, English will inherit that mistake precisely. Human-in-the-loop remains the upstream authority; inheritance protects parity, not truth.

4. The excerpt exceeded 300 characters

Another model response was meaningful but failed the Digest schema because its excerpt exceeded 300 characters. A repeated prompt might return a shorter version, but would make the pipeline less predictable and spend another request on a mechanical constraint.

Weekly Digest 0.3.2 introduced deterministic normalization before schema validation. An oversized excerpt is shortened at a safe boundary rather than being sent back for regeneration.

function normalizeExcerpt(value: string, limit = 300) {
  if ([...value].length <= limit) return value;
  return shortenAtWordBoundary(value, limit);
}

The boundary matters. Normalization is allowed for a presentation field. It does not apply to Signal membership, URLs, Recommended Actions, or factual content. Shortening a description is an acceptable deterministic operation; silently dropping a fact or Signal is not.

The new contract separates mechanically repairable output from semantic output. The residual risk is that an automatically shortened excerpt can become weaker or lose an important nuance. It still requires editorial review.

5. A random character was a schema-valid excerpt

The next incident demonstrated the opposite problem. The excerpt was not too long. It consisted of a random character and formally satisfied the minimum string schema.

This was not a typing defect. The schema answered its question correctly: the field existed, had the type string, and remained within the limit. It could not answer whether the text was useful to a reader, SEO, or a social preview.

We did not try to turn the Zod schema into a universal editor. We could add a minimum length, reject some patterns, or require sentence-like structure, but every such heuristic has false positives and still cannot prove meaning.

The new contract is that schema validation controls shape and bounded constraints, while editorial approval controls meaning and quality. Before export, a person checks the title, excerpt, introduction, Signal cards, actions, and links.

The residual risk is reviewer fatigue. A human gate is not magic: if the interface hides a field or review becomes a mechanical button press, a random character can continue downstream. That is why a Digest remains a readable Markdown artifact rather than an opaque approval dialog.

6. The portable title limit diverged from the production SEO gate

The MDX exporter had portable validation, but the site loader applied a stricter production constraint. A title that passed the local contract failed native site validation because of the 60-character SEO limit.

This was a classic integration gap: both systems had validation, but enforced different rules for the same field. More validation does not help when it proves a different contract.

In MDX Export 0.3.1, the portable limit was aligned with production. A frontmatter title may contain no more than 60 characters. The full editorial heading does not need to be truncated artificially; it can remain in the body or be adapted to the site rendering contract.

editorial heading: full meaningful title
frontmatter title: SEO-safe, ≤60 characters

The new contract is that downstream native validation is part of the export gate, and the portable schema must match its public constraints. The residual risk is that the production contract can change. The loader therefore runs during every export rather than being copied once into documentation and trusted forever.

Act 3. Bilingual transaction, Revision, and Git delivery

7. Two valid MDX files did not yet make a valid bilingual pair

The first English export had to update uk.mdx to locales: [uk, en] and create the neighboring en.mdx. If the first write completed and the second failed validation, the repository could remain in a state no reviewer had ever approved.

An atomic write protects one file, not a multi-file invariant. The exporter therefore received a transaction boundary: temporary files, backups, native validation, commit of the whole operation, or rollback in reverse order.

try {
  const writes = await prepareAll([ukTarget, enTarget]);
  await validateNativePair();
  await commitAll(writes);
} catch (error) {
  await rollbackCompletedInReverseOrder();
  throw error;
}

The new contract is that bilingual identity is validated as one operation. The site cannot be left with new English and old Ukrainian locales, or the reverse.

The residual risk is that a filesystem transaction is not a database transaction and does not publish the site. Process crashes, permissions, and external file watchers remain failure modes. Backups stay until validation, while Git, deployment, and production smoke remain separate gates.

8. An already published Digest could not simply be extended

After the first W33 publication, we found five more Signals that belonged to the same period. The original Digest had already been approved, exported, and published. Editing the file directly would have destroyed its link to the earlier approval hash and hidden which Signal set the reviewer had seen.

The happy path closed the Weekly Cycle and deleted Temporary Content. It had no legitimate transition backward. This was not one missing button; the domain model lacked a lifecycle for controlled extension of a historical artifact.

That produced the Revision flow:

W33 approved Digest
  └─ R2
      ├─ base: 1 Signal + base approval hash
      ├─ added: 5 Signals
      ├─ revised UK + new approval
      ├─ revised EN + new approval
      └─ atomic replacement + controlled closure

The closed Cycle was not reopened, and deleted context was not restored. R2 retained ancestry, separate current hashes, and cleanup evidence. The final Digest contained six Signals.

The new contract is that history is not rewritten; a change to a published period receives a Revision identity. A new thematically related Signal normally enters the next weekly cycle and may create a confirmed Before → After comparison instead of automatically reopening an old publication.

The residual risk is that a Revision lifecycle adds complexity and can become a habitual substitute for weak selection review. The first R2 proves that the flow works, but does not establish a normal monthly revision rate.

9. The R2 branch conflicted with the already merged English Digest

The Revision was ready locally, but main had changed after the English Digest was merged through PR #26. PR #27 for R2 touched the same uk.mdx and en.mdx, so the remote branch could no longer advance as a simple continuation of its old base.

This was not a defect in the Radar domain model. Atomic export had produced the correct pair on disk. The failure was in delivery coordination: a locally valid artifact had been built from stale Git state.

The branch was rebased onto current main, conflicts were resolved as a new coherent bilingual pair, and the gates ran again. The remote branch was updated with --force-with-lease, not unrestricted force push.

fetch current main
  → rebase publication branch
  → resolve UK/EN as one pair
  → rerun gates
  → push --force-with-lease

The new contract is that export evidence does not transfer automatically across a rebase. A changed base requires repeated validation. --force-with-lease protects against overwriting an unknown remote update, but does not prove that conflict resolution was correct.

The residual risk is that two independent delivery branches can still edit the same publication pair. Serial publication ownership or automation could reduce that risk later, but controlled Git discipline was consciously sufficient for v0.1.

An evidence snapshot, not a vanity metric

The v0.1 closure snapshot contained:

• Integration incidents in the first cycle: 9.
• Signals in W33 R2: 6 — 1 base + 5 added.
• Plugin tests: 98 across 6 separate suites.
• Site tests: 445 across 80 files.
• Production build: 169 pages.

The 98 plugin tests and 445 site tests are not added into one “system test count.” They verify different codebases and belong to a particular acceptance snapshot. Earlier spike documents contain lower counts for earlier plugin versions. That is development history, not a contradiction to conceal.

We likewise do not claim that a Digest can be prepared reliably within 30 minutes. W33 had no active-work instrumentation, pause events, or explicit preparation boundaries. The KPI remains not measured, even where individual elapsed intervals look shorter.

What the incidents actually changed

The nine fixes compress into five architectural rules.

1. Durable state outranks UI state. A critical command rereads the Vault instead of trusting the freshness of an active object.
2. Deterministic identity is not delegated to AI. The model localizes prose; Ukrainian approval owns membership, tags, and actions.
3. Schema controls shape; a reviewer controls meaning. Both gates are required and answer different questions.
4. A related set of files needs a transaction boundary. One-file atomic writes are insufficient for a Ukrainian/English pair.
5. Publication is a chain of evidence. Export, Git, merge, deployment, and production smoke are not synonyms.

These rules were not entirely absent at the beginning. Some existed as intentions in the BRD or a spike. Production forced us to turn intention into executable behavior and negative tests.

That is the practical value I see in MAE. The methodology does not promise to predict every integration failure before the first run. It provides a way not to waste a failure twice: the observation returns to an artifact, contract, implementation, test, and user instruction.

What remains conscious post-MVP debt

Not every residual risk needs another plugin increment in v0.1.

• Five official Sources remain the completed scope; expansion to 20 requires separate adapter and legal contracts.
• Active-time telemetry is required before the Digest preparation KPI can be evaluated.
• Monthly bilingual ratio, publication cadence, missed Critical Signals, and revision rate require a complete observation period.
• CLI control remains the third phase; v0.1 is operated through the Obsidian UI.
• Full plugin source will not be published until the monthly operational review is complete.

This is not a list of hidden unfinished defects. It separates functionally complete v0.1 scope from monitored outcomes and later product phases.

One month of operation before opening the code

The next evidence should come not from another prompt, but from repeated use of the system. Over a month, we need to verify:

• whether five Sources continue to pass scheduled collection and recovery;
• whether Critical candidates and breaking changes remain visible through selection;
• how much active editorial time a Ukrainian/English cycle takes;
• whether Recommended Actions remain current after related Signals appear;
• how many Digests and Revisions are actually published;
• whether backup and isolated restore continue to pass as the Vault changes.

We can then compare architectural contracts with operational behavior, publish the feedback, and decide which plugin sources are safe to open.

The next part takes a different step: a reproducible tutorial for your own Obsidian, from a controlled RSS fixture to bilingual MDX. It will not expose the complete private implementation, but it will include enough structure, commands, state transitions, and troubleshooting to reproduce the approach.