agentctl Research Audit — Learn AI Curriculum

agentctl Research Audit

Fact-check status

Reviewed claim by claim on 2026-09-02 against the primary or official sources identified inline as S1 and S2. Recommendations are design guidance unless a cited standard makes them normative.

Key jargon

Term	Plain-language meaning
Provenance	Evidence describing where a claim or artifact came from. S1
Research lead	A candidate source or idea that still requires verification. S1
Accepted claim	A conclusion retained after checking scope and supporting evidence. S1
Rejected claim	A conclusion excluded because it was false, overbroad, unsupported, or unsuitable. S1

Key concepts

Multi-model research can widen discovery, but agreement among models is not independent evidence. S1
Record failures and rejected claims so readers can distinguish research activity from validated knowledge. S1

Concept map

flowchart LR
    A["Run bounded research tasks"] --> B["Collect claims and source leads"]
    B --> C["Verify against primary evidence"]
    C --> D["Accept narrow claim or reject"]

Diagram semantics verified against S1 and S2 on 2026-09-02.

BLUF. agentctl was used as a research reviewer, not as an unquestioned authority. Its output supplied candidate concepts, labs, misconceptions, and primary-source leads. Claims entered the curriculum only when they agreed with an original paper, a standard, or official documentation in the source ledger. S1

Research runs

Workstream	Provider	Result	Used for S1
RAG, evaluation, security, governance, automation	`agy`	Completed	Hybrid retrieval, judge-bias exercises, NIST/OWASP/MITRE framing, approval gates S1
Foundations and model behavior	`codex`	Timed out	No evidence accepted from this failed run S1
Harnesses, agents, MCP, state, permissions, evaluation	`claude`	Timed out	No evidence accepted from this failed run S1
Foundations plus production harness retry	`agy`	Completed	Tokenization/decoding exercises, tool-call boundary, checkpointing, idempotency, telemetry S1
Multi-provider orchestration attempt	planner	Failed schema validation before research	No research output; replaced with bounded `ask` runs S1

The runs were research-only and instructed providers not to edit the vault. Temporary raw outputs were stored under /tmp/atoz-ai-agentctl-20260902/; this note is the durable synthesis. S1

Publication fact-check challenge pass

After the curriculum rewrite, five additional bounded agentctl ask --to agy reviews received numbered lesson text directly in their prompts. An initial attempt to let the provider read vault files was denied by the headless permission policy; the retry preserved that boundary rather than bypassing it. These are local run records, not external evidence.

Bundle	Lines	Bytes	SHA-256
Foundations	219	25,631	`a69f7b310090c621c6aac329acb4beb27d946e85e0d40545033eda4823eb0b1c`
Harnesses	299	22,440	`06b707271e272de844d87ccb9bfad87f5c4ecb903474fd26f227d4bb2ae04787`
Knowledge and deployment	167	22,212	`df29a546ed4f3a7093c92929ffe5a278ba5a9e2d738948f09412105f8e1ba77b`
Evaluation and security	203	26,028	`bf85806302d5689b4c0e70a21475b56055e03ea35fc9f289570f40275b1e0992`
Automation and labs	148	21,000	`5c4fa70a981ed14e66dd9c04a419499ad3f6c37876697aed6422f7078e7ed24e`

The five outputs total 1,036 lines and 117,311 bytes. They found no direct contradiction in the 65 core pages and treated the 68 diagrams as aligned with their surrounding lessons. Their suggested citations were not accepted automatically: blog or encyclopedia suggestions were replaced by original papers, standards, or official documentation during the primary-source pass. The temporary outputs are under /tmp/atoz-ai-factcheck-20260902/ for session-level inspection.

Accepted conclusions

A useful curriculum separates model behavior from the application harness that assembles context, invokes models, validates tool requests, enforces authorization, stores state, and records evidence. S1
Tool calls are proposals. The surrounding application must validate, authorize, execute, and report the result. S1
Model quality must be measured at several layers: deterministic contract checks, task-specific graders, human calibration, online telemetry, and regression gates. S1
Retrieval quality is not answer truth. Retrieval, grounding, generation, and citation correctness need separate tests. S1
Side-effecting tools require least privilege, approval boundaries, durable state, and idempotency controls outside the prompt. S1
Current APIs, model names, prices, framework syntax, security taxonomies, and telemetry conventions are deliberately marked as staleness-sensitive. S1

Rejected or narrowed claims

Candidate claim	Decision
An LLM judge must use temperature zero and hidden chain-of-thought	Rejected as universal advice. Judge settings are empirical choices; use explicit rubrics, repeated/calibrated tests, and human labels. Do not require hidden reasoning disclosure. S1
A two-model parser/executor pattern neutralizes indirect prompt injection	Narrowed. Privilege separation can reduce impact, but parsed data can still carry adversarial meaning. Authorization and constrained tools remain mandatory. S1
Fine-tuning primarily changes style and cannot add facts	Narrowed. Fine-tuning can adapt behavior and domain knowledge, but it is a poor substitute for frequently changing, attributable knowledge. S1
Longer context removes the need for retrieval	Rejected. Context size does not provide provenance, relevance ranking, freshness, or access control. S1
Framework abstractions make model providers interchangeable	Rejected. Adapters reduce integration work; semantics, tool behavior, safety controls, latency, and output shape still vary. S1

Reproduction prompt pattern

Research-only curriculum review. For each assigned area provide:
core concepts, common misconceptions, a hands-on exercise,
primary or official source URLs, and a staleness warning.
Do not edit files.

Evidence rule

Re-run this review when a lesson depends on current product behavior. Keep enduring concepts tied to primary papers or standards, and put changing implementation details behind dated official documentation. S1

← AI Reference · Learn AI