agentctl Research Audit — Learn AI Curriculum

agentctl Research Audit

Key jargon

Term	Plain-language meaning
Provenance	Evidence describing where a claim or artifact came from.
Research lead	A candidate source or idea that still requires verification.
Accepted claim	A conclusion retained after checking scope and supporting evidence.
Rejected claim	A conclusion excluded because it was false, overbroad, unsupported, or unsuitable.

Key concepts

Multi-model research can widen discovery, but agreement among models is not independent evidence.
Record failures and rejected claims so readers can distinguish research activity from validated knowledge.

Concept map

flowchart LR
    A["Run bounded research tasks"] --> B["Collect claims and source leads"]
    B --> C["Verify against primary evidence"]
    C --> D["Accept narrow claim or reject"]

BLUF. agentctl was used as a research reviewer, not as an unquestioned authority. Its output supplied candidate concepts, labs, misconceptions, and primary-source leads. Claims entered the curriculum only when they agreed with an original paper, a standard, or official documentation in the source ledger.

Research runs

Workstream	Provider	Result	Used for
RAG, evaluation, security, governance, automation	`agy`	Completed	Hybrid retrieval, judge-bias exercises, NIST/OWASP/MITRE framing, approval gates
Foundations and model behavior	`codex`	Timed out	No evidence accepted from this failed run
Harnesses, agents, MCP, state, permissions, evaluation	`claude`	Timed out	No evidence accepted from this failed run
Foundations plus production harness retry	`agy`	Completed	Tokenization/decoding exercises, tool-call boundary, checkpointing, idempotency, telemetry
Multi-provider orchestration attempt	planner	Failed schema validation before research	No research output; replaced with bounded `ask` runs

The runs were research-only and instructed providers not to edit the vault. Temporary raw outputs were stored under /tmp/atoz-ai-agentctl-20260902/; this note is the durable synthesis.

Accepted conclusions

A useful curriculum separates model behavior from the application harness that assembles context, invokes models, validates tool requests, enforces authorization, stores state, and records evidence.
Tool calls are proposals. The surrounding application must validate, authorize, execute, and report the result.
Model quality must be measured at several layers: deterministic contract checks, task-specific graders, human calibration, online telemetry, and regression gates.
Retrieval quality is not answer truth. Retrieval, grounding, generation, and citation correctness need separate tests.
Side-effecting tools require least privilege, approval boundaries, durable state, and idempotency controls outside the prompt.
Current APIs, model names, prices, framework syntax, security taxonomies, and telemetry conventions are deliberately marked as staleness-sensitive.

Rejected or narrowed claims

Candidate claim	Decision
An LLM judge must use temperature zero and hidden chain-of-thought	Rejected as universal advice. Judge settings are empirical choices; use explicit rubrics, repeated/calibrated tests, and human labels. Do not require hidden reasoning disclosure.
A two-model parser/executor pattern neutralizes indirect prompt injection	Narrowed. Privilege separation can reduce impact, but parsed data can still carry adversarial meaning. Authorization and constrained tools remain mandatory.
Fine-tuning primarily changes style and cannot add facts	Narrowed. Fine-tuning can adapt behavior and domain knowledge, but it is a poor substitute for frequently changing, attributable knowledge.
Longer context removes the need for retrieval	Rejected. Context size does not provide provenance, relevance ranking, freshness, or access control.
Framework abstractions make model providers interchangeable	Rejected. Adapters reduce integration work; semantics, tool behavior, safety controls, latency, and output shape still vary.

Reproduction prompt pattern

Research-only curriculum review. For each assigned area provide:
core concepts, common misconceptions, a hands-on exercise,
primary or official source URLs, and a staleness warning.
Do not edit files.

Evidence rule

Re-run this review when a lesson depends on current product behavior. Keep enduring concepts tied to primary papers or standards, and put changing implementation details behind dated official documentation.

← AI Reference · Learn AI