E2E Learning Loop Blind Spots: Skip-as-Success and More

This morning’s closure checklist assumes the ledger is honest. Most open supervised learning loops fail before that checklist, as operations bugs.

We trusted a green nightly. The feature scan had run. The RCA smoke had opened and closed a draft. Concurrently, a required suite skipped on a missing credential, a newer pending run replaced an older one, and the product fix from last week was still not on the environment the tests hit. The model was fine. The bookkeeping was not.

Key takeaways

Green workflow status is not proof every scenario executed.
Unread pages and top-N seeds are incomplete, not empty.
Merge, deploy, and re-assert are separate joins.

Eight blind spots from cadence mismatch to missing post-deploy verification

Fix the ledger before blaming the agent.

Blind spot	What goes wrong	Closure
Cadence mismatch	Feature discovery mistaken for a full anomaly sweep	Keep Loop A and Loop B explicit
Top-N / pagination	Latest N failures or truncated APIs omit the rest	Report limits; mark unread incomplete
Skip as success	Job success masks skipped suites	Inspect scenario counts, not only conclusion
Fixture versus product	Missing creds or apps labelled as product bugs	Fail preflight explicitly; no runtime verdict
Merge ≠ deploy	Merged SHA not live, or telemetry from another env	Observe exact image or build on the target
Lost proposals	Plans unimplemented; fix drafts cleaned up unread	Retain coverage drafts; adopt fixes deliberately
Queue starvation	Newer pending run replaces older; windows vanish	Record what actually ran; chase missing windows
No post-deploy verify	Fix merged, assert never run on new build	Link a verification run to the finding

Related CI honesty themes live in evals in CI/CD: a required-task skip is not a pass. Block’s agent testing pyramid keeps live LLM work off every-PR CI partly because flaky or skipped probabilistic jobs teach the wrong lesson about green. AWS adds shadow comparison for the same reason staging lies.

Applause calls out another failure mode that looks like Loop B success: automation that rewrites the test to pass. That is skip-as-success with better marketing. Closing a finding by deleting the assert leaves the product broken and the ledger smug.

Operator checklist

For each scheduled job, write whether it is coverage scan, anomaly review, or plain regression.
Store absolute window bounds and watermarks on findings.
On every green run, open the scenario summary before trusting the conclusion.
Label fixture and credential failures as preflight, not product RCA.
Join namespace, cluster, and deploy revision before accepting a root cause story.
Give every durable candidate an identity, owner, and disposition.
After product merge, require exact-revision observation plus the linked assert.
Weekly, sample one “no gap” and one “no fix justified” outcome for evidence quality.

What not to optimize

Do not chase PR volume. Do not treat prompt text as an authorization barrier. Do not assume a concurrency group is a FIFO review of every commit. Do not weaken asserts to make a shared environment look healthy.

The pyramid still places checks at the cheapest useful layer. The learning loop only makes the system better when these blind spots are named and closed in the ledger, not when the agent sounds confident in a draft.

FAQ

What blind spots keep an E2E learning loop open?

Cadence mismatch, top-N or pagination treated as complete, skip-as-success, fixture problems labelled as product bugs, merged SHA not deployed, lost or cleaned-up proposals, non-FIFO missed windows, and merges without post-deploy verification.

What is skip-as-success in CI?

A job concludes successfully while required scenarios never ran because prerequisites were missing. Inspect scenario counts and JUnit, not only the workflow conclusion.

Why is top-N dangerous for anomaly review?

Seeding from the latest N failed runs or truncated API pages omits other failures. Unread data must be marked incomplete, never inferred as healthy.

How do you keep proposals from disappearing?

Retain coverage drafts for human adoption by default. Treat fix-draft cleanup as proposal smoke unless someone deliberately adopts the candidate into a review queue.