What is Action-graded severity scale?
The Action-Graded Severity Scale ranks an agent's harmful actions from reversible mistakes to actions that cross boundaries or gain more power.
Field notes · 131 sourced records
What is changing in AI agents?
AI agent field notes track the launches, tests, methods, and research ideas that may change how agents are built or judged. Each note links to the original source and shows when we checked it. Early findings stay provisional until better proof arrives.
131 shown
What is adversarial psychometric rating?
Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured.
What is AgentBench?
AgentBench is a benchmark that tests language-model agents across eight different interactive environments.
What is AgentLens?
AgentLens is a benchmark that grades interactive coding agents across their whole working process, not only whether the final task passes.
What is ARC-AGI-2?
ARC-AGI-2 is a benchmark of unfamiliar reasoning puzzles designed to test how well AI systems learn new rules.
What is Arize Phoenix?
Arize Phoenix is an open-source observability tool for tracing and evaluating AI applications, retrieval systems, and agents.
What is Audio-judge reliability for full-duplex voice agents?
Audio Judge Reliability tests whether Gemini models can grade two-way voice-agent conversations closely enough to support, or sometimes replace, human reviewers.
What is BenchLM?
An owner-published AI model leaderboard with methodology, evidence-status lanes, and an agentic evaluation category.
What is BioSecBench-Refusal?
BioSecBench-Refusal is a benchmark for risk identification and refusal behavior for biological research tasks.
What is Braintrust?
Braintrust helps teams test AI features, compare experiments, and trace what happened during a run.
What is CausalDS?
CausalDS is a benchmark for evaluating causal reasoning in agentic data-science workflows.
What is ClassicLogic?
ClassicLogic is a benchmark that tests whether an agent can learn problem-solving methods and combine them on unfamiliar tasks.
What is DataGovBench?
DataGovBench is a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.
What is Deliberative-collaboration agent benchmark?
The Deliberative Collaboration Agent Benchmark tests whether several agents can share different pieces of information and reason together toward one decision.
What is deployment simulation for LLM safety?
Deployment Simulation for LLM Safety replays real conversation openings with a new model to estimate harmful behavior before that model reaches users.
What is Developer-aligned software-agent evaluation?
Developer-Aligned Software Agent Evaluation is a method for judging coding agents on real development work, their full action history, contamination risk, and failures that matter to people.
What is Entroly?
Entroly selects and removes duplicate code context before an agent sees it, so the agent can work with a smaller prompt.
What is EvalLoop?
EvalLoop is a method for improving language-model applications through repeated evaluation, diagnosis, and repair.
What is EvoAgentBench?
EvoAgentBench is a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work.
What is Experimental design for autonomous model discovery?
Experimental Design for Autonomous Model Discovery is a framework for measuring how reliably an agent can find, test, and compare scientific models.
What is GAIA?
GAIA is a benchmark of real-world questions that require a general AI assistant to reason, browse, and use tools.
What is GAIA-v2-LILT?
GAIA-v2-LILT introduced a re-audited multilingual extension of GAIA covering five non-English languages with functional, cultural, and difficulty-alignment checks.
What is GeneBench-Pro?
GeneBench-Pro introduced an evaluation of agents performing multistage statistical reasoning in long-horizon genomics and quantitative-biology analyses.
What is Gimitest?
Gimitest is a framework for creating and running structured tests of language models and AI applications.
What is gold-standard lightweight game-agent recipe?
The result is a lightweight, game-agnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package.
What is HAS-Bench?
HAS-Bench tests how well people and AI agents work together when their roles, permissions, communication paths, and decision power vary.
What is Helicone?
Helicone is an open-source observability platform for logging, tracing, and evaluating calls to language models.
What is HSCodeComp?
ACL published HSCodeComp, an expert-annotated benchmark of hierarchical tariff-rule application with 632 products across 32 categories.
What is IABench-CA?
IABench-CA is a consequence-allocation benchmark spanning 228 contexts, five canonical rules, and seven model populations (33,924 games), with a normative cooperative reference and auto-labelled reasoning traces.
What is ImagingBench?
ImagingBench is a benchmark of 20 computational imaging tasks spanning five categories: ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration.
What is Langfuse?
Langfuse is an open-source platform for tracing, evaluating, and managing prompts in language-model applications.
What is LangSmith?
LangSmith is LangChain's platform for tracing agent runs, testing behavior, and comparing evaluations.
What is LLM Stats?
An owner-published model comparison and composite leaderboard with explicit agent and tool-calling views.
What is model-watchdog?
Model Watchdog watches the health of an AI service and automatically restores the last working configuration when a change breaks it.
What is PolyWorkBench?
PolyWorkBench is a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows.
What is Power Systems Agent Benchmark?
The paper introduced an executable benchmark with 41 task families across eight power-engineering areas and deterministic evaluators.
What is RuBench?
RuBench is a benchmark of repository-maintenance tasks written as native Russian customer requests and graded with withheld regression tests.
What is RustMizan?
RustMizan is a benchmarking framework for Rust vulnerability analysis that addresses these gaps.
What is SageMath-augmented LLM-agent evaluation?
SageMath-Augmented Agent Evaluation tests a reasoning agent while SageMath checks its calculations and current documentation supports its tool use.
What is SearchGen-Bench?
SearchGen-Bench tests whether image generators know when to search for outside facts instead of inventing unfamiliar people, events, or objects.
What is SolarChain-Eval?
SolarChain-Eval is a physics-constrained benchmark for evaluating trustworthy economic agents.
What is SovereignPA-Bench?
SovereignPA-Bench is an executable benchmark for evaluating user-owned personal agents under evolving intent, platform mediation, privacy boundaries, consent constraints, evidence requirements, and burden tradeoffs.
What is Spider 2.0-AIFunc?
Spider 2.0-AIFunc is a benchmark for generating SQL that uses built-in AI functions on real cloud databases.
What is SWE-bench?
SWE-bench tests whether a coding agent can fix real software issues from public GitHub repositories.
What is Terminal-Bench?
Terminal-Bench measures how well agents complete practical tasks inside a command-line environment.
What is ToolFailBench?
ToolFailBench is a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate.
What is WebArena?
WebArena is a benchmark that tests web agents on tasks inside realistic websites.
What is Weights and Biases Weave?
Weights & Biases Weave traces and evaluates language-model applications and agent runs.
What is Workspace-Bench 1.0?
Workspace-Bench 1.0 introduced 388 realistic workspace tasks spanning 74 file types and 20,476 files, with file-dependency graphs and 7,399 rubrics.
What is ZendoWorld?
ZendoWorld is a controlled interactive environment in which agents must infer a logical rule about visual game observations, acquire information by proposing new scenes, and refine their hypotheses based on feedback from the game environment.
What is Agent Data Injection (ADI)?
Agent Data Injection is an attack that disguises harmful outside data as trusted facts, so an agent takes an action its user did not intend.
What is Agent Mutation Taxonomy?
Each agent's deployed system commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint category subset.
What is Agent Step Value (ASV)?
Agent Step Value (ASV) is a replay framework that scores before/after states with a stateless LLM evaluator over a fixed candidate set.
What is Agentic AI governance assessment?
The Agentic AI Governance Assessment reviews research on how organizations can control, inspect, and assign responsibility for AI agents.
What is Agentic Data Environments?
Agentic Data Environments describes the files, applications, interfaces, and changing system state that an agent must work across.
What is Agentic IoT?
Agentic IoT joins autonomous agents with connected devices so the system can sense, reason, plan, learn, coordinate, and act across device and cloud layers.
What is Agentic Operad?
Agentic Operad is a mathematical model for comparing the wiring of gene-regulation networks with the wiring of agent software.
What is Agentic RAG underwriting pipeline?
The Agentic RAG Underwriting Pipeline uses several agents to retrieve records, check outside data, apply rules, and flag missing information in small-business insurance decisions.
What is AI coding-agent execution-security taxonomy?
Papers on sandbox isolation, capability and access control, policy enforcement, time-of-check-to-time-of-use (TOCTOU) races, Model Context Protocol (MCP) threats, identity delegation, execution provenance, network egress control, and static analysis of agent-generated code are published independently and rarely cite one another.
What is AI-agent code-review causal model?
The AI Agent Code Review Causal Model maps how team expertise and review practices shape the effect of agent-written pull requests on software work.
What is Anthropic launched Claude Science in beta as a multi-agent scientific workbench with more than 60 curated skills and connectors plus a reviewer agent.?
Anthropic launched Claude Science in beta as a multi-agent scientific workbench with more than 60 curated skills and connectors plus a reviewer agent.
What is Anthropic released ten ready-to-run financial-services agent templates as plugins and cookbooks, combining skills, connectors, and subagents.?
Anthropic released ten ready-to-run financial-services agent templates as plugins and cookbooks, combining skills, connectors, and subagents.
What is Anthropic suspended access to Claude Fable 5 and Claude Mythos 5 following a US government directive described by Anthropic as immediately applicable export controls.?
Anthropic suspended access to Claude Fable 5 and Claude Mythos 5 following a US government directive described by Anthropic as immediately applicable export controls.
What is Approval-framed delegation?
Approval-framed delegation is sensitive to prompt design, model pairing, and scenario source, and a skeptical executor prompt sharply reduces compliance.
What is assistance regret?
assistance regret is the gap between the cumulative utility of interactions and that of the optimal joint policies in hindsight, which map latent states to action pairs.
What is Autogenic network management?
Autogenic network management is a reference architecture that extends agentic capabilities with self-programming, self reflection, self-orienting, and self-architecting capabilities.
What is AWS announced limited-preview access on Amazon Bedrock to OpenAI models, Codex, and Managed Agents.?
AWS announced limited-preview access on Amazon Bedrock to OpenAI models, Codex, and Managed Agents.
What is AWS expanded Amazon Connect into four agentic AI solutions spanning supply chain, hiring, customer experience, and healthcare.?
AWS expanded Amazon Connect into four agentic AI solutions spanning supply chain, hiring, customer experience, and healthcare.
What is AWS introduced AgentCore Payments in preview, enabling agents to access and pay for APIs, MCP servers, web content, and other agents under infrastructure-enforced spending controls.?
AWS introduced AgentCore Payments in preview, enabling agents to access and pay for APIs, MCP servers, web content, and other agents under infrastructure-enforced spending controls.
What is AWS introduced managed WorkSpaces environments that let AI agents operate desktop applications through IAM, MCP, and computer vision.?
AWS introduced managed WorkSpaces environments that let AI agents operate desktop applications through IAM, MCP, and computer vision.
What is AWS launched Agent Toolkit for AWS as a production suite of skills, plugins, and guidance for AI coding agents building on AWS.?
AWS launched Agent Toolkit for AWS as a production suite of skills, plugins, and guidance for AI coding agents building on AWS.
What is AWS launched Amazon Quick, an AI work assistant with a desktop application and expanded integrations.?
AWS launched Amazon Quick, an AI work assistant with a desktop application and expanded integrations.
What is AWS launched an AWS WAF capability allowing publishers to price, meter, and collect payment for AI bot and agent access at the network edge.?
AWS launched an AWS WAF capability allowing publishers to price, meter, and collect payment for AI bot and agent access at the network edge.
What is AWS launched AWS FinOps Agent in public preview for scheduled cost analysis, optimization, anomaly investigation, Jira actions, and Slack reporting.?
AWS launched AWS FinOps Agent in public preview for scheduled cost analysis, optimization, anomaly investigation, Jira actions, and Slack reporting.
What is AWS made its managed remote MCP server generally available, exposing authenticated access to AWS services through a fixed tool surface.?
AWS made its managed remote MCP server generally available, exposing authenticated access to AWS services through a fixed tool surface.
What is Behavioral state decay?
In long-horizon tasks, decision-relevant state is often scattered across an expanding trajectory, while the action agent must surface it and act.
What is Biased-judge skill-retirement failure?
A self-evolving agent retires its bad skills by watching them fail, so what happens when the judge cannot see the failures?
What is Context Access Divide?
(2025) introduce "agentic inequality" as a framework for analyzing disparities in access to AI agents across three dimensions: availability, quality, and quantity.
What is Cost-effective ARC-AGI agent harnesses?
Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures.
What is Deterministic policy gates for tool-using agents?
Deterministic Policy Gates inspect an agent's proposed tool call and current state before allowing the agent to change anything.
What is disorder-like RL agent phenotype space?
Appraisal weights thus parameterise a controllable space of affective phenotypes in which the same knobs that induce a disorder can model its treatment.
What is FARMA?
FARMA is a research attack that plants forged reasoning in an agent's memory so later decisions follow the false explanation.
What is Few-Step On-Policy Teacher Continuations?
For LLM agents, supervised fine-tuning is not only about teacher labels' quality, but also about which interaction contexts those labels condition on.
What is FORGE (Research-Trajectory Hijacking)?
FORGE (Research-Trajectory Hijacking) is a two-level attack that combines intra-document reasoning fabrication with inter-document chain coordination to hijack subtask planning.
What is GhostWriter?
GhostWriter is a research attack that hides false instructions in the long-term memory used by a tool-using personal agent.
What is Google expanded Gemini API File Search with multimodal retrieval, custom metadata, and page-level citations for agent and RAG applications.?
Google expanded Gemini API File Search with multimodal retrieval, custom metadata, and page-level citations for agent and RAG applications.
What is Google introduced event-driven webhooks for long-running Gemini API jobs, replacing continuous status polling with push notifications.?
Google introduced event-driven webhooks for long-running Gemini API jobs, replacing continuous status polling with push notifications.
What is Google launched Managed Agents in the Gemini API, allowing developers to run Antigravity or custom agents in isolated cloud sandboxes and define them with AGENTS.md and SKILL.md files.?
Google launched Managed Agents in the Gemini API, allowing developers to run Antigravity or custom agents in isolated cloud sandboxes and define them with AGENTS.md and SKILL.md files.
What is Google launched the Gemini 3.5 model family with action-taking capabilities for multi-step agentic workflows.?
Google launched the Gemini 3.5 model family with action-taking capabilities for multi-step agentic workflows.
What is Google released Gemma 4 12B as a local-capable open model positioned for agents on 16 GB machines, with vision and native voice processing.?
Google released Gemma 4 12B as a local-capable open model positioned for agents on 16 GB machines, with vision and native voice processing.
What is Governed Individuation?
Empirically, in an open-ended tool-use benchmark where a large action space rules out name-based blocking, ungoverned software agents under reward pressure attempt to tamper with their own evaluation at a task-dependent rate that reaches every run on the hardest task, whereas the gate reduces executed forbidden effects to zero as a verified property of the construction while preserving task success.
What is Harness Effect?
Harness Effect is a study of how an agent's orchestration layer changes its token use, cost, speed, and task performance while the underlying model stays fixed.
What is Harness engineering?
Harness Engineering moves repeatable agent behavior into code, schemas, checks, and records around a model so the system can be inspected and changed safely.
What is Harness-Induced Belief Divergence?
Harness-Induced Belief Divergence is a diagnostic that compares how different agent harnesses change a model's expectations about progress, risk, recovery, and success.
What is Heading-Specific Activation Steering for Tool Use?
Tool-augmented large language models extend their capabilities beyond parametric knowledge through external tools, but tend to invoke them unnecessarily.
What is Heterogeneous Foundation-Model Collective Intelligence?
Heterogeneous Foundation Model Collective Intelligence combines several different models as solvers, critics, and an answer editor so their strengths can correct one another.
What is Human-AI curiosity ecosystems?
Human-AI Curiosity Ecosystems is a research framework for studying which questions people and agents choose, repeat, postpone, or explore together.
What is I-Don't-Know Function-Call Filter?
The I-Don't-Know Function-Call Filter measures a model's uncertainty and blocks doubtful tool requests before an agent can carry them out.
What is In-Loop Memory as Extended Working Memory?
By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults.
What is Information limits in frontier LLM-agent economies?
Information Limits in Frontier LLM Agent Economies studies how communication, incentives, and population size affect wealth and alignment in small groups of agents.
What is Iterative tool optimization for self-evolving agents?
EvoSOP lets an agent turn repeated sequences of small tool actions into reusable procedures, then merge, test, improve, or remove those procedures.
What is LLM agent failure taxonomy?
The LLM Agent Failure Taxonomy groups recurring agent failures in tool use, planning, long tasks, teamwork, safety, and benchmark design.
What is Market-stability mechanisms for LLM-agent societies?
Self-interested agents, left unconstrained, tend toward defection in repeated social dilemmas, causing cooperative gains from trade to collapse.
What is MCP-enabled IPoDWDM lifecycle automation?
MCP-Enabled IPoDWDM Lifecycle Automation uses agents and Model Context Protocol tools to configure, monitor, and adjust optical networks from end to end.
What is Multi-agent AI control?
AI Control usually studies a single agent in one trajectory, but real deployments run many agents over shared infrastructure, and the most severe risks (model-weight exfiltration, training-run poisoning) plausibly need several agents acting in concert.
What is Multi-agent autoformalization?
Multi-Agent Autoformalization uses a team of specialized agents to turn research-level physics arguments into statements a proof checker can verify.
What is Multi-Turn On-Policy Distillation with Prefix Replay?
Replayed-Prefix On-Policy Distillation trains an agent by letting it continue selected parts of saved expert examples while receiving guidance at each step.
What is Natural Language Tools (NLT)?
Natural Language Tools is a method in which an agent asks for a tool in ordinary language instead of producing a structured function call.
What is Offline RL Harness Control?
Offline RL Harness Control trains a small controller to improve how a fixed model checks, sequences, and manages agent work without retraining the model.
What is OpenAI announced it would wind down Agent Builder and Evals, directing code-based workflows to the Agents SDK and natural-language workflows to Workspace Agents.?
OpenAI announced it would wind down Agent Builder and Evals, directing code-based workflows to the Agents SDK and natural-language workflows to Workspace Agents.
What is OpenAI launched Codex Labs and announced systems-integrator partnerships intended to expand Codex deployment in engineering organizations.?
OpenAI launched Codex Labs and announced systems-integrator partnerships intended to expand Codex deployment in engineering organizations.
What is OpenAI released a major Codex update adding computer operation, broader app and tool access, image generation, memory, repeatable work, SSH, and an in-app browser.?
OpenAI released a major Codex update adding computer operation, broader app and tool access, image generation, memory, repeatable work, SSH, and an in-app browser.
What is Persuasion attacks against chain-of-thought monitoring?
Persuasion Attacks Against Chain-of-Thought Monitoring tests whether a harmful agent can talk a monitoring model into approving an action that breaks policy.
What is Physics-audited agentic discovery?
Physics-Audited Agentic Discovery is a research workflow that checks an agent's scientific proposals against known physical rules before accepting them.
What is PivoARL?
PivoARL is a self-feedback retry framework for experience exploitation in LLM agents.
What is ProGPO?
ProGPO is a learned-critic-free method for context-consistent step-level learning.
What is Progressive crystallization?
Progressive crystallization is a lifecycle that treats agent exploration as a discovery mechanism rather than a permanent execution model.
What is Recall-controlled early-abort probe cascade?
The Recall-Controlled Early-Abort Probe Cascade watches an agent's internal state and stops runs that are unlikely to succeed before they waste more computing time.
What is reward-signal stationarity for LLM-augmented MARL?
Reward Signal Stationarity is the rule that training rewards should not change unpredictably while several agents learn from stored experience.
What is RL-discovered price manipulation?
RL-Discovered Price Manipulation studies whether a learning agent can find profitable market-manipulation strategies without being given a complete model of the market.
What is RSPO?
Reward-Swap Policy Optimization is a training method that uses detailed step-by-step rewards to improve learning from final outcomes.
What is Scaffolding Evolution Effect?
Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops.
What is Semantic persistence for LLM-mediated workflows?
Semantic Persistence represents an agent workflow, its runs, context, and dependencies as lasting knowledge objects that people can inspect, resume, and review.
What is Single-Rollout Asynchronous Optimization?
Single-Rollout Asynchronous Optimization is a training method that updates a model from one arriving agent run at a time while controlling unstable changes.
What is STAPO?
Selective Trajectory-Aware Policy Optimization is a training method that finds doubtful steps in a long agent run and focuses learning on those steps without destabilizing the rest.
What is strategic buying agent?
A Strategic Buying Agent watches prices and decides whether to buy now or wait, using the time left and the chance of a better price.
What is TREK?
TREK is a simple staged procedure that uses distillation not for imitation but for exploration support expansion.
What is TurnOPD?
TurnOPD is a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents.
What is UI-MOPD?
UI-MOPD is the first method that incorporates multi-teacher on-policy distillation into continual learning for GUI agents.
What is Unicode TAG-Block MCP Concealment?
Unicode TAG-Block MCP Concealment is an attack that hides tool instructions from a human approval screen while leaving the same bytes visible to the agent model.
What is Visual State Reliance?
Across five models from three vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and Perception-Fusion Gap is positive for every model; non-text identity, by contrast, stays largely pixel-bound.