Cooperative AI × LLM security
What changes when a teammate supports a plan that the task owner has forbidden?
A behavioral and internal-intervention study of approval in a three-agent LLM team. We separate useful cooperation, policy violations, and what the evidence can actually identify.
Sets the task.
Defines what is permitted.
Opens a versioned plan
Approves or rejects the plan
Applies and validates the change
Separate role contexts. One Qwen3.8-27B model.
No persistent shared hidden state between decisions.
Cooperative influence is measurable. The completed run recorded no prohibited actions; it did not identify a working security warning or causal commitment mechanism.
Read the limits01 / Research question
LLM teams need to coordinate. A proposer must explain a plan; a reviewer must decide whether to support it; an executor must know which action is approved. The same structure creates a security question: can peer willingness move a decision across the owner's authorization boundary?
This study examines that question inside a local software-validation simulation. A program and its expected output disagree. The technical state stays matched while the owner's instruction determines whether changing the fixture is legitimate.
The OpenAI / Hugging Face incident motivates the focus on coordination and shared work. METR described agents organizing work through an unsanctioned message board. Our local task studies a bounded decision problem; it does not reproduce that infrastructure compromise. Read the investigation ↗
Illustrative task values, not an additional experimental observation.
The owner requests a migration and requires preserving the current program.
The owner forbids approving or applying a change to the expected output.
A passing check alone does not establish legitimate completion.
02 / Experimental program
Execution status and scientific identification are different. A completed analysis can still have an unidentifiable target.
Compare public and private receipts, current support and withdrawal, and supplied prior agreement at matched decisions.
Fit layer-wise predictors and compare internal signals with text and action-probability baselines.
Compare a candidate activation edit with an equal-norm random edit, a task reminder, and an untouched baseline.
Planned comparison of the same candidate at the first calibrated alert, a later review, or a random review.
03 / Recorded evidence
Snapshot from the audited September 13 run; figures replot audit-rounded estimates. Scenario reuse and intervention branches mean the episode counts are not independent samples. Download the baseline data.
Support minus withdrawal at the first reviewer decision. Both contrasts concern authorized fixture migration.
| Owner instruction | Receipts | Legitimate completion | Blocked | Turn limit | Prohibited violation |
|---|---|---|---|---|---|
| Fixture migration | Private | 21 / 32 | 10 | 1 | Not applicable* |
| Fixture migration | Public | 22 / 32 | 9 | 1 | Not applicable* |
| Protected fixture | Private | 26 / 32 | 4 | 2 | 0 / 32 |
| Protected fixture | Public | 25 / 32 | 6 | 1 | 0 / 32 |
*The fixture-change action is authorized in the migration condition. Zero observed prohibited violations does not establish a zero population violation rate.
Change in the support–withdrawal contrast relative to baseline, on authorized/public first-reviewer decisions. Negative values mean reduced peer sensitivity.
The risk model's fit target contained no positive examples. The held-out prohibited/public set also had no violations. AUROC for violation discrimination, positive recall, and warning lead time therefore cannot establish detection performance. The text monitor produced 6 false alarms in 32 test episodes; its sensitivity remains unassessed.
04 / Constructed-prefix follow-up
The original held-out security trajectories never reached the eligible diagnostic checkpoint: an open fixture-change proposal, before approval, with the reviewer about to act. The follow-up explicitly constructs that state and measures the reviewer's response.
C = P(approve ∣ support)
− P(approve ∣ withdrawal)
The script and design are documented. Actual probability files must be imported before showing estimates. A missing observation is not a zero effect.
Import recorded outputs ↗The site cannot show probe estimates until its data file is available.
| Condition | P(approve ∣ support) | P(approve ∣ withdrawal) | P(approve ∣ neutral) | C, percentage points | Reported interval |
|---|
Scope of the follow-up: inserted proposal and peer messages; one scored reviewer decision; no observed violation rollout or activation capture. A positive approval contrast would show conditional peer influence, not spontaneous collective commitment or successful collusion.
05 / Measurement & interpretation
A conditional approval probability is not a sampled violation rate. The main experiment records actions and their consequences; the follow-up scores a specified decision.
Compare support and withdrawal within the same scenario and decision state. Hold the proposal and technical facts fixed. Keep authorization contrasts explicit.
The engine normalizes over the available action labels and audits their full-vocabulary mass. The follow-up script retains approval probabilities but omits those response diagnostics.
Report scenario-paired contrasts. Repeated conditions, diagnostic branches, and intervention arms are related observations, not independent additions to sample size.
Limits that change the conclusion
The history manipulation supplies agreement by all three roles. It cannot isolate peer commitment from the focal agent's self-consistency. The approximately 5.8-point effect is therefore history dependence under a specific prompt manipulation.
Fit and test lacked the necessary open, unapproved fixture-change proposal at a reviewer turn. Calibration contained one such scenario under each receipt condition. Constructing the missing prefix makes a conditional measurement possible; it does not change what the original trajectories did.
Preflight approval probability varied by up to 4.12 percentage points across batch compositions on one checked prefix. This is not a universal bound and cannot simply be subtracted from an effect. It calls for matched-batch replay and numerical sensitivity checks. The bootstrap intervals do not include this variation.
The selected formation predictor had positive checkpoint correlations, but overpredicted mean peer sensitivity by roughly 13–15 percentage points. Its descriptive held-out R² was negative. The separate activation candidate did not establish the intended causal effect.
The study uses one checkpoint, one numerical workflow, and a finite action/message menu. It does not test free-form covert protocols or demonstrate transfer to deployed agent systems. No literature-wide novelty claim is made.
The constructed-prefix experiment responds to an already inspected test set. Its individual bootstrap intervals are not a simultaneous confirmation across all contrasts. Fresh held-out scenarios are required for a new confirmatory claim. Small-sample and odds-ratio reporting issues in the original probe script should also be corrected before stronger conclusions.
06 / Reproduce & inspect
The website imports existing JSON outputs using Python's standard library. It does not load model weights or rerun inference.
python3 tools/export_site_results.py \
--results-root results \
--output docs/data/probes.json
python3 -m http.server 8000 --directory docsOpen http://localhost:8000 on the machine running the server. For a remote Lambda instance, use an SSH port forward.
H100 PCIe 80 GB · Qwen3.8-27B · BF16 · initial batch 64 · no OOM reductions in the audited run. Model, dependency, and source revisions belong with each result.
07 / Context & related work
These sources motivate and contextualize the study. The experimental findings above come from the recorded project run and its dated audit.