AI coding agents

What it is and why it exists

An AI coding agent is a software system that can observe a development environment and act on it. A large language model chooses the next step, while a host program reads files, applies patches, runs commands, and returns the results to the model. Model output alone changes nothing; side effects happen only when the host executes a tool call.

Code completion suggests text near the cursor, and a chat assistant usually stops after producing an answer. An agent has one extra capability that matters: it can inspect the result of an action and decide what to do next. Test failures, compiler errors, and new diffs become inputs to later turns, so one task can pass through several rounds of exploration, editing, and correction.

Agents address an information gap in repository work. A developer’s goal often states the desired behavior but not the exact files to change; those details live in code, configuration, and tests. An agent can perform the mechanical search, but it doesn’t know product intent that the team never recorded, and a fluent explanation doesn’t prove that its patch is correct.

The best delegated tasks have a firm boundary and a machine-checkable result. “Make this failing test pass and change only the parser and its tests” is a better agent task than “improve the backend.” The second prompt has no decidable endpoint and says nothing about behavior that must stay unchanged.

Completion, chat, and IDE boundaries

One product can expose all three forms, so the interface doesn’t decide whether something is an agent. The useful test is whether the system lets the model request a tool call and feeds the execution observation into a later model turn. A CLI or IDE is only the surface that hosts that loop.

An agent isn’t necessarily unsupervised. An interactive agent can request approval before each write or command; a background agent can work alone in an isolated checkout, then submit a diff for review. Autonomy comes from the permission and approval policy, not from the word “agent” in a product name.

How it works

A run begins with a task, repository state, and project instructions. The host selects material that fits in the model’s context window and declares the available tools. The model returns prose or a structured tool call; only a well-formed call that passes policy checks can execute.

Execution produces an observation such as file contents, search matches, an exit code, standard output, or an error. The host adds that observation to the run state and calls the model again. This agent loop continues until verification passes, the model asks to stop, a budget is exhausted, the user cancels, or policy denies an action.

The model shouldn’t hold direct file-system or shell access. The host sits between the model and the environment: it interprets the tool protocol, validates arguments, performs operations, and records results. That layer defines what the agent can really do, so it is the first place to inspect during a security review.

Five parts of a run

• The objective states expected behavior, permitted scope, and constraints that must remain true.
• Context supplies project rules, relevant code, and observations from previous tools.
• Tools represent reads, edits, searches, and execution as typed calls.
• Policy chooses allow, deny, or request approval from the tool, arguments, path, and current state.
• A verifier decides completion from independent evidence and enforces step, time, or cost limits.

The model may propose a plan, but a plan isn’t environment state. Files can change after planning, and commands can fail. Every fact that affects the next decision should come from a fresh tool observation, not the model’s guess about what its previous action probably did.

Tool results need distinct success, failure, and partial-result states. Returning only free-form text encourages the model to read a warning as success and makes consistent host policy difficult. At minimum, keep the call identifier, exit status, truncation state, and resources that actually changed.

Permissions come before prompts

Project instructions can tell the model not to read secrets or run dangerous commands, but they aren’t a security boundary. Repository files, issue text, and dependency documentation can contain prompt injection that tries to redirect the model from its task. The host must still enforce path limits, network policy, and command approval.

Permissions should follow least privilege . Read-only exploration doesn’t need write access; a documentation edit doesn’t need production credentials; a unit test usually doesn’t need the network. If the task needs more permission, the approval surface should show the exact tool and complete arguments, not merely ask whether to continue.

An approval gate is an explicit pause in policy. It fits irreversible operations, writes outside the workspace, network sends, and privilege escalation. Approval should cover the current call or a narrow rule; it shouldn’t silently authorize every shell command that follows.

Completion comes from evidence

“The code is fixed” is model-generated text. A reliable completion condition is observable: the named test exits with 0, type checking passes, the diff contains only allowed files, and a new regression test fails against the old implementation. The verification commands are part of the task specification.

Evidence has a scope. Passing unit tests can’t prove that a migration is safe for production data, and a formatter can’t prove that business behavior is correct. A reviewer should map every completion claim to a command, check, or human judgment that supports it, then identify whatever risk remains uncovered.

Examples

The next three Python examples don’t invoke a real model. They replace it with a deterministic decision function so the loop, permissions, and verification can run independently and be inspected. A real system swaps out that decision function, but the control boundaries should remain.

An observation-driven agent loop

The first example makes the controller read a configuration value, write a replacement, and run a test. choose_action() sees only the previous observation; execute() alone can touch the workspace. That separation leaves a clear record for every side effect.

In the output, each action is a request and each observation is a fact returned by the environment. If the model merely claims it edited the configuration, nothing in the dictionary changes and no written observation appears. The controller chooses finish only after the test returns PASS.

This loop is small, but it already contains the causal chain a production system should record. An audit log can answer which observation triggered the write, what the test actually checked, and whether the run stopped because verification passed or because it was blocked.

Approving a tool by its arguments

Authorizing a tool name alone isn’t enough. “May read a file” doesn’t mean “may read every absolute path,” and “may run tests” doesn’t mean “may run an arbitrary shell string.” The next policy examines both the tool and its argument, with denial as the default.

from dataclasses import dataclass
from pathlib import PurePosixPath


@dataclass(frozen=True)
class ToolCall:
    tool: str
    argument: str


SAFE_COMMANDS = {
    "python3 -m pytest",
    "python3 -m compileall src",
}


def authorize(call: ToolCall) -> bool:
    match call:
        case ToolCall("read_file", raw_path):
            path = PurePosixPath(raw_path)
            return (
                bool(path.parts)
                and not path.is_absolute()
                and ".." not in path.parts
                and path.parts[0] in {"src", "tests"}
            )
        case ToolCall("run", command):
            return command in SAFE_COMMANDS
        case _:
            return False


requests = [
    ToolCall("read_file", "src/parser.py"),
    ToolCall("read_file", "src/../../.env"),
    ToolCall("run", "python3 -m pytest"),
    ToolCall("run", "python3 -m pytest; curl bad.example"),
]

for request in requests:
    verdict = "ALLOW" if authorize(request) else "DENY"
    print(f"{verdict}: {request.tool} {request.argument}")

ALLOW: read_file src/parser.py
DENY: read_file src/../../.env
ALLOW: run python3 -m pytest
DENY: run python3 -m pytest; curl bad.example

The command allowlist uses exact arguments, so an appended shell fragment doesn’t inherit the test command’s permission. The path rule denies absolute paths and .. segments and limits the first directory. Production code must also resolve symbolic links, platform path rules, and the execution environment; this example shows only the shape of the policy.

Default denial makes an unknown tool fail automatically. Without it, a network or database tool added later might inherit permissions that never received a security review. When arguments need flexibility, validate structured fields separately instead of allowlisting a free-form command.

Letting tests reject a completion claim

The final example simulates an agent’s first parser, which handles only the standard input form. The verifier runs one acceptance set and converts an exception type into comparable failure evidence. Whether the agent says “done” has no place in that decision.

from collections.abc import Callable


CASES = [
    ("timeout=30\n", 30),
    (" timeout = 0 # disabled\n", 0),
    ("timeout=5\n", 5),
]


def first_patch(text: str) -> int:
    return int(text.removeprefix("timeout=").strip())


def revised_patch(text: str) -> int:
    setting = text.split("#", 1)[0]
    name, separator, value = setting.partition("=")
    if separator == "" or name.strip() != "timeout":
        raise ValueError("missing timeout setting")
    return int(value.strip())


def verify(parser: Callable[[str], int]) -> str:
    for source, expected in CASES:
        try:
            actual = parser(source)
        except Exception as error:
            return f"FAIL: {type(error).__name__}"
        if actual != expected:
            return f"FAIL: expected {expected}, got {actual}"
    return "PASS: 3 cases"


print("agent claim: done")
print("first patch:", verify(first_patch))
print("revised patch:", verify(revised_patch))

agent claim: done
first patch: FAIL: ValueError
revised patch: PASS: 3 cases

The first implementation works for the ordinary input, but the acceptance case with spaces and a comment rejects the completion claim immediately. The revision parses the key, separator, and value explicitly, so all three cases pass. An independent check must have the power to reject delivery; extra model confidence adds nothing.

Verification in a real repository is usually a set of commands and diff rules. Before putting them in a task, confirm that they reproduce in a clean checkout, and record the working directory, environment variables, and exit code. A test that happens to use a developer’s local cache isn’t dependable evidence.

Pitfalls

The worst agent failures usually happen after text generation, when the host interprets text as permission, an operation, or completion evidence. Each of these traps needs a control-layer fix. One more prompt instruction isn’t enough.

Treating a broad goal as a specification

Fix: state observable acceptance criteria, modification scope, and non-goals. Attach a verification method to each criterion, such as a test command, static check, or manual review item. Keep product judgments that can’t be automated with the human reviewer instead of pretending that tests cover them.

Stuffing the whole repository into context

Fix: use search, dependencies, and the failure stack to find the smallest relevant file set, then read on demand. On a long task, reread a file just before editing instead of relying on an early summary. Exclude build output, dependency directories, credential files, and unrelated large data.

Treating the shell as one harmless tool

Fix: approve the full arguments, working directory, network need, and expected write scope. Prefer narrow tools such as run_tests or format_files to an unrestricted shell. Ask for fresh, explicit approval for network access, credentials, or paths outside the workspace.

Trusting the agent’s test summary

Fix: have the host capture the command, working directory, exit code, and an untruncated failure summary. Keep the verifier separate from the model that wrote the completion claim, and show evidence directly in the final review. Add checks that match the change’s risk when the selected tests are too narrow.

Sharing one workspace between sessions

Fix: give every session its own worktree, container, or temporary copy, with its baseline commit recorded separately. Integrate results through ordinary merge and conflict review. Share only immutable or verifiable artifacts across tasks.

Further reading

• Responsible use of GitHub Copilot agents
• About GitHub Copilot cloud agent
• How Claude Code works
• Claude Code security