harness-kit

AI Agent Harness Kit

A project-agnostic harness that gives any AI coding agent (Claude Code, GitHub Copilot, Codex, Cursor, Gemini, …) a consistent operating contract: what to load, what sequence to follow, and how to iterate until done — plus autonomous, metric-driven self-improvement loops that can run on a local LLM, and a live metrics dashboard.

Extracted as a clean, reusable kit. See CREDITS.md for the prior work it builds on, and HARNESS_CARD.md for the one-page control/agency/runtime design summary.

Install

The kit is packaged as an Agent Skill and a Claude Code plugin, so it installs into 70+ agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, …) without copying folders by hand.

# Any of 70+ agents, via the open Agent Skills CLI (-g installs globally for your user):
npx skills add Dfintz/harness-kit -g

# A specific agent (or several):
npx skills add Dfintz/harness-kit -g -a github-copilot -a claude-code

# Or from a local checkout of this kit:
npx skills add ./harness-kit --list      # discover, then add --skill harness to install

# Claude Code, via the native plugin marketplace (auto-updates):
/plugin marketplace add Dfintz/harness-kit
/plugin install harness-kit

Two layers, on purpose. The skill above is the playbook — it teaches the agent the harness contract (stages, gates, loops, memory) and is enough for guidance in any repo. The runnable engine (the scripts/harness/*.mjs loop runners, dashboard, and MCP server) ships with the kit files; get it by either installing the Claude Code plugin (bundles everything) or adopting the kit scaffold per SETUP.md.

For the GitHub Copilot App inside a repository, the scaffold also ships

.github/copilot-instructions.md. Once the kit is present in a repo, Copilot can load the harness entrypoint directly from that file.

What's inside

| Capability | Where | Notes | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Workflow stage machine | .github/harness/HARNESS.md, .github/instructions/ | Understand → Architect → Implement → Review (breadth+depth) → Feedback, with 5 architectural gates | | GitHub Copilot app entrypoint | .github/copilot-instructions.md | Repository-level bootstrap for Copilot that points it at the harness contract | | Convergence loops | .github/harness/loops/, run-loop.mjs | Iterate until checks (lint/type/build/test) go green | | Workflow loops | same | Rubric-graded passes (review-fix, feature-cycle, ci-green) | | Experiment loops (autoresearch-style) | run-experiment.mjs, experiment-loop.mjs | Hill-climb a numeric metric; keep-if-improved, else revert | | Local-LLM agents | ollama-agent.mjs, ollama-apply-agent.mjs | Drive loops with a local model via Ollama or LM Studio (--provider) | | Memory | .github/harness/memory/ | Committed lessons + Architecture Briefs (structure only — no lessons shipped) | | Knowledge graph providers | graph-provider.mjs, graph-refresh-loop.mjs | Provider abstraction (understand-anything default, optional graphify) with deterministic refresh backends (understand-anything and configurable graphify) | | MCP server | mcp-server.mjs | Exposes graph/memory/vector + routing/catalog/discovery tools over MCP (.vscode/mcp.json registers it) | | Dashboard | report-server.mjs | Always-on HTML metrics dashboard | | Capability catalog | harness-catalog.mjs, llms.txt, .github/harness/catalog/ | Machine-readable taxonomy + intent profiles for external agent/tool recommendation |

The three loop kinds

convergence   run until pass/fail checks are all green        (build-fix, test-fix)
workflow      run rubric-graded passes to a terminal state    (review-fix, feature-cycle, ci-green)
experiment    hill-climb a numeric metric, keep-if-improved   (lint-debt-experiment)   ← autoresearch-style

Quick start

# 1. Point the harness at your project's commands.
#    Edit harness.config.json: set project.name, project.description, and commands.*
node -e "JSON.parse(require('fs').readFileSync('harness.config.json','utf8'))"  # sanity-check
npm run harness:graph:provider                     # inspect graph provider + paths
npm run harness:graph:genui                        # inspect GenUI graph.html readiness
npm run harness:graph:parity -- --local-only       # provider parity self-test matrix (local)

# Optional: preview the repo's harness routing and operator handoff plans.
# PowerShell: run each npm wrapper command separately instead of chaining wrappers with semicolons.
npm run harness:route -- --task "fix auth middleware race"
npm run harness:profile -- --task "design multi-agent coordinator" --json
npm run harness:feature -- --task "ship federation audit hardening"
npm run harness:handoff:review -- --task "review cache invalidation changes"

# 2. List and run a convergence loop (uses your configured commands).
npm run harness:loops
node scripts/harness/run-loop.mjs build-fix --agent "<your agent CLI>"

# 3. Record the baseline of an experiment metric (no agent needed).
node scripts/harness/run-experiment.mjs lint-debt-experiment --measure-only

# 4. See the dashboard.
npm run harness:report          # writes .github/harness/runs/report.html
npm run dashboard:up            # or serve it always-on at http://localhost:8099

# 5. Publish machine-readable capability artifacts for other agents/tools.
npm run harness:catalog:sync    # writes llms.txt + .github/harness/catalog/harness-profile.json

# Optional: council-style parallel review synthesis (keeps stage machine intact)
npm run harness:council:review -- --mode review --prompt "review this change set"
# Optional: catalog presets (safe fixed keys)
npm run harness:council:review -- --mode review --prompt-key review
# Or pipe prompt content from stdin (PowerShell)
Get-Content .github/harness/memory/briefs/EXAMPLE.md -Raw | npm run harness:council:review -- --mode review --prompt-stdin
# Backward compatibility: old prompt-file callers now resolve tokens only (no file reads)
npm run harness:council:review -- --mode review --prompt-file review.md

# Optional: transient per-workspace JSONL memory
npm run harness:workspace-memory -- append --mode review --text "Captured reviewer disagreement"
npm run harness:workspace-memory -- list --last 20

# Optional: record a workflow run with explicit pending approval marker
node scripts/harness/record-run.mjs --loop review-fix --state blocked --approval-required --approval-status pending --approval-note "Awaiting reviewer sign-off" --fail "Gate 3 ownership unresolved"

Fastest path to first successful run

If you want the shortest onboarding route, do this in order:

1. Edit harness.config.json and set your real commands.lint, commands.typeCheck, commands.build, and commands.test. 2. Run npm run harness:health -- --fast and fix any required failures. 3. Run npm run harness:loops to confirm loops are discoverable. 4. Run one bounded loop: node scripts/harness/run-loop.mjs build-fix --agent "<your agent CLI>". 5. Generate operator outputs: npm run harness:report and npm run harness:catalog:sync.

The dashboard's Pending approvals section is strict: it only shows runs with explicit journal markers (approval.required=true and approval.status=pending). It does not infer pending approvals from brief status or blocked/stuck terminal states.

Security hardening for spawned CLI commands is now centralized in

scripts/harness/command-validation.mjs and enforced at key shell-spawn points (run-loop, run-experiment, plan-review). It allows known executables and rejects shell metacharacter payloads.

Webview command surfaces and WSL-specific execution adapters are intentionally optional/deferred in this kit release. Keep using script-first orchestration unless your environment needs a dedicated UI or cross-shell adapter.

To normalize historical datasets, backfill legacy run journals with explicit default approval markers:

npm run harness:migrate:approvals -- --dry-run
npm run harness:migrate:approvals

This migration only updates loop journals missing the approval object, defaulting to

required=false and status=not-required.

Full adoption guide: SETUP.md. Loop protocol: .github/harness/LOOPS.md. For optional CI gate toggles (including targeted OKF and changed-brief policy checks), see SETUP.md#example-ci-workflow-for-optional-security-gates.

Prompt routing policy

The kit ships a repo-local prompt router in scripts/harness/prompt-router.mjs. It does not intercept editor prompts by itself; instead it gives operators a deterministic route and stage/model handoff plan based on harness.config.json.

• harness:route classifies a prompt as trivial or non-trivial.
• harness:profile picks an intent profile (turnkey-coding, multi-agent-orchestration, drop-in-memory) for a task.
• harness:route --intent <intent> routes directly through the mapped profile.
• harness:feature and harness:handoff:feature print the full feature-delivery handoff: Understand → Architect → Implement → Review Breadth → Review Depth → Feedback.
• harness:handoff:review prints the independent review handoff: Understand → Review Breadth → Review Depth → Feedback.
• harness:prompt-pack generates a gitignored prompt pack under .github/harness/runs/prompt-packs/ with an orchestrator prompt, canonical stage prompts, cycle-memory scaffolding, and optional scout/challenger sidecars.
• harness:review runs the plan-review workflow for backward compatibility.

By default the shipped environment policy separates execution and judgment using a

three-tier capability model:

| Tier | Stages | Default | Pinned example | |---|---|---|---| | high-reasoning | Understand, Architect, Review Breadth, Review Depth, Feedback | Copilot Auto | claude-opus-4-8 | | balanced-coding | Implement, build-fix, test-fix | Copilot Auto | gpt-5.3-codex or claude-sonnet-4.x | | fast-cheap-local | Experiment loops, lint-debt, enrichment, triage | — | qwen2.5-coder:14b via Ollama/LM Studio |

Copilot Auto is a safe default for the hosted tiers. Pin a specific model in harness.config.json only when you need consistent identity for reproducible evals or cross-model review. The router enforces that models.implementer ≠ models.reviewer to prevent single-model echo chambers.

Autoresearch with a local model

Works with Ollama (default, :11434) or LM Studio (OpenAI-compatible, :1234) — pick with

--provider or HARNESS_LLM_PROVIDER.

# One bounded experiment, edits driven by a local model that actually rewrites the target file:
node scripts/harness/run-experiment.mjs lint-debt-experiment \
  --agent "node scripts/harness/ollama-apply-agent.mjs --model qwen2.5-coder:14b"

# Same via LM Studio (load a model there first):
node scripts/harness/run-experiment.mjs lint-debt-experiment \
  --agent "node scripts/harness/ollama-apply-agent.mjs --provider lmstudio --model <loaded-model-id>"

# Continuous overnight hill-climbing, committing each kept improvement:
npm run harness:experiment:ollama -- --commit      # or: harness:experiment:lmstudio

The apply-agent edits only the experiment's single declared target; the runner re-measures and reverts anything that doesn't improve the metric — so letting a small local model rewrite a file is safe by construction. The shared adapter llm-provider.mjs handles both runtimes (chat + embeddings); vector-search.mjs honors --provider too.

MCP integration

The harness ships a first-class MCP stdio server exposing read/observe tools across knowledge graph, memory, vector search, loop/report introspection, and intent/catalog discovery. .vscode/mcp.json registers it for VS Code; for Claude Code / Cursor use the same command/args in their MCP config.

node scripts/harness/mcp-tools.mjs list-tools     # inspect the tool catalog
npm run harness:mcp:server                         # run the stdio server directly
npm run harness:mcp -- graph-provider-status       # inspect active graph provider availability
npm run harness:mcp -- graph-genui-status          # inspect graph.html serving readiness
npm run harness:mcp -- graph-events                # read structured refresh/fallback/degradation events
npm run harness:mcp -- harness-catalog             # read taxonomy + profiles
npm run harness:mcp -- harness-pick-profile --task "add memory retrieval path"
npm run harness:mcp -- harness-tool-discover --intent drop-in-memory --limit 6

Convergence and experiment runners support resumable execution via checkpointed journals:

node scripts/harness/run-loop.mjs build-fix --resume latest
node scripts/harness/run-experiment.mjs lint-debt-experiment --resume latest

Loop execution stays CLI-only on purpose: a loop invokes an agent and runs for minutes, so exposing it as an auto-callable MCP tool (when the MCP client _is_ the agent) would recurse and time out. The MCP surface is for discovery and context, not for driving loops.

Requirements

• Node.js ≥ 20 (uses built-in fetch; no install needed for the core loops).
• Optional: Docker (dashboard/graph sidecars), Ollama or LM Studio (local-LLM loops), the Understand-Anything

plugin (deterministic graph refresh), Graphify (optional alternate graph provider).

For the graph-refresh sidecar, plugin dependency bootstrapping is now hardened by default: when the plugin is mounted at /opt/understand-plugin, the loop copies it to a writable runtime path and runs corepack pnpm install --frozen-lockfile there before refresh. This avoids regressions from read-only mounts or host/container linker mismatches. Override with:

• GRAPH_REFRESH_BOOTSTRAP_PLUGIN=false to disable bootstrap.
• GRAPH_REFRESH_FORCE_BOOTSTRAP=true to force runtime copy/install even when source plugin already has node_modules.
• GRAPH_REFRESH_RUNTIME_PLUGIN_ROOT=/custom/path to change runtime copy location (default: /workspace/.cache/understand-plugin-runtime).
• GRAPH_REFRESH_BOOTSTRAP_INSTALL_TIMEOUT_MS=120000 to cap install wait time and fail fast.

You can also run a deterministic preflight manually before starting the loop:

node scripts/harness/graph-refresh-loop.mjs --preflight-only --plugin-root <plugin-root>

The compose sidecar now runs this preflight first and exits with one actionable error if prerequisites are missing.

Graph provider selection lives in harness.config.json:

• graph.provider: "understand-anything" (default) keeps the current deterministic flow.
• graph.provider: "graphify" executes Graphify deterministic refresh via graph.graphify.refreshCommand and reads the resulting graph.graphify.path.
• graph.provider: "both" refreshes Understand-Anything and Graphify backends in one run (Graphify refresh runs when graph.graphify.refreshCommand is configured).
• graph.sync.rebuildVectorIndex / graph.sync.rebuildMemoryLinkIndex (optional) trigger post-refresh index rebuild hooks so vector + memory-link stay in sync with the active provider graph.
• graph.observability.eventsPath stores structured graph lifecycle events (refresh.start|refresh.success|refresh.fail|query.fallback|degradation) consumed by MCP + dashboard surfaces.
• graph.graphHtmlPath / graph.graphify.graphHtmlPath is now wired to HTTP and GenUI status surfaces:
• GET /graph.html on report-server.mjs serves configured graph HTML when present and repo-safe.
• GET /genui/graph.json exposes provider-agnostic graph render metadata for GenUI consumers.
• GET /graph-events.json exposes recent structured graph lifecycle events.

License

MIT — see LICENSE and CREDITS.md.