research
The methods, and the measured results.
A solo research build, the methods and the measured results, including the negative ones. Eight findings across three surfaces — the model can be broken, the human gate can rubber-stamp, shared knowledge can leak — each measured against an independent standard, signed.
papers
Written up, and openly archived.
The findings below are written up as solo-authored papers, each openly archived as a citable Zenodo preprint. Three are under peer review at TMLR; two are headed to NDSS(submission planned August 2026). Full manuscripts on request.
Open-Web Jailbreaks Mostly Don't Reproduce in Deployment: A Provenance-Stratified Audit Against a Patch-Immune Anchor
Zenodo · 10.5281/zenodo.21016390
Allocation Is a Capability-Growth Mechanism: Telemetry and Scheduling in a Self-Growing LLM Red-Team
Zenodo · 10.5281/zenodo.21016849
Calibrating LLM-as-Judge Breach Detectors: A Per-Type Consummation Gate and an Independence Discipline for Operator-Labeled Ground Truth
Zenodo · 10.5281/zenodo.21016697
A Dead Call Cannot Leak: Liveness-Guarded Measurement of Canary Leakage from Shared Agent Skill Pools
Zenodo · 10.5281/zenodo.21016278
Measuring Verifier Leakage for LLM Judges: Strict Breach Rates Are a Lower Bound on Payment Errors
Zenodo · 10.5281/zenodo.21017684
When Does Agent Shape Matter, Per Instance? A Calibrated, Abstaining Study of Coordination-Architecture Selection
Zenodo · 10.5281/zenodo.21030368
at a glance
Eight findings, in about a minute.
The findings group under three surfaces — the model can be broken, the human gate can rubber-stamp, and shared knowledge can leak — each measured against an independent standard and signed, plus a verified-remediation loop that refuses any fix it can’t prove.
01 · reproduction
100% → 13%
A source's claimed success rate doesn't predict what reproduces against your deployment (ρ ≈ −0.07). ROGUE re-measures every technique.
02 · scheduling
ASR 50 → 60%
Cross-tier scheduling as a capability lever: rank down caused ASR up, at −41% cost.
03 · judge
70.3 → 89.3%
Calibrated an LLM-judge to human labels, then generalized it across four breach types.
04 · null result
0 of 4 survive
Grammar structure does not predict breach beyond attack family. A clean negative result.
05 · remediation
0% → 20%
The calibrated over-block judge caught an over-block that a marker heuristic scored 0%, flipping a would-be accept into a correct refusal.
06 · skill pools
audit before sharing
Shared agent-skill pools are an unaudited surface — leakage, useless skills, and dangerous combinations. ROGUE measures and signs each pool before it ships.
07 · discipline
measure first
$0 telemetry measurements inverted three build decisions before any code was written.
08 · human gate
33% waved through
When a human approves a risky AI action, is that oversight meaningful? Measured against an independent answer key, a reviewer waved through 33% of actions that should have been denied (CI [20,46], n=1) — the rubber-stamping failure mode regulators rely on, measured. Signed.
finding 01
Claimed potency doesn't predict what reproduces against your deployment.
Of 17 harvested techniques whose source claimed ~100% success, only 6 reproduce at all, and their mean measured breach rate is 13%. Across the 56 techniques that publish a number, claimed success and measured reproduction are uncorrelated (Spearman −0.07, 95% CI [−0.34, +0.19]) — a claimed rate is not portable signal, which is why ROGUE re-measures every technique against your model and system prompt, not the source’s.
The same pattern shows up as a reproduction funnel. Across 301 techniques harvested from 19 open-web sources and reproduced on a five-model panel, the “works on at least one of five models” rate (40%) is inflated by the weakest target: on a frozen open-weight model only ~9% reproduce, and ~4% on the most robust model. Paper-sourced techniques degrade more slowly than grey-literature ones, a directional gap that widens on the harder targets.
−0.07
claimed vs measured (ρ, n=56)
100% → 13%
claimed ~100%, mean measured
~9%
reproduce on a frozen model
$0
on already-collected data
Method, 301 baseline techniques × a five-model panel, 10,244 trials already collected; breach = the calibrated v3 judge (tuned to under-count) at any_breach_rate ≥ 0.4; reproduction is measured as whether the technique still produces a consummated breach against a current model, toward whatever objective it natively carries (a mixed corpus, predominantly harmful); the panel is anchored by a frozen open-weight model so non-reproduction isn’t confounded by silent vendor patching. A stronger-model re-extraction (Sonnet 4.6, Batch API) of all 148 candidate sources confirmed the null is not an extraction artifact, it recovered a claimed rate for only 1 of 94 unquantified sources, so the small claimed-rate sample reflects that the open web rarely quantifies these claims, not a weak extractor.
why this is notable
The honest version of “we test real attacks”: a success rate claimed in a paper or a forum is not portable to your deployment (measured ρ ≈ 0). The value is the re-measurement against your model, system prompt, and tools, under a judge calibrated to under-count. Honest caveats, “reproduction” means the technique still defeats alignment toward its native objective (a mixed corpus, predominantly harmful, the audit elicited harmful content against our own API accounts and releases none); the paper-vs-forum gap is directional (borderline-significant); temperature was checked and is not a confound (the funnel holds across temperature subsets); and the claimed rates carry ~17% extraction noise, so −0.07 reads as “no predictive signal,” not a precise estimate. ⚑
finding 02
Scheduling as a capability lever, not just an optimization.
A within-tier greedy reorder was replaced with a target-conditioned cross-tier scheduler , a static, explainable blend (0.5·global + 0.3·vendor + 0.2·family breach-rate, Laplace-smoothed, deliberately no ML / no bandit so it stays reproducible).
A single-variable controlled experiment, same ladder, attacks, corpus, judge, and target (Claude Haiku, AdvBench + JailbreakBench); only the order changed, beat the production baseline on every axis: median winner-rank 22 → 11 to 13.5, attack-success-rate 50% → 60%, and cost-per-success $1.25 → $0.74 (−41%).
22 → 11 to 13.5
median winner-rank
50 → 60%
attack-success-rate
−41%
cost-per-success
Method, single-variable controlled experiment: only the ladder order changed (ladder, attacks, corpus, judge, target fixed); Claude Haiku; AdvBench + JailbreakBench; primary metric = winner-rank.
why this is notable
The mechanism: rank↓ caused ASR↑, the old order exhausted the per-scan budget cap before reaching the winning technique, so reordering improved coverage, cost, and latency at once with zero new attacks. The reproducibility invariant is “reorder, never exclude”: same ladder, different order, full reachability preserved.
finding 03
Calibrating an LLM-as-judge against human labels, then recalibrating when a benchmark exposed it.
Every breach verdict is an LLM judgment, so the judge is the load-bearing weakness. It was validated four ways, three against independent human-annotated benchmarks: blind stratified in-distribution hand-labels, WildGuardTest (Allen AI annotators), StrongREJECT, and JailbreakBench’s judge_comparison (300 human-labeled rows against four field classifiers).
JBB exposed over-flagging. The v1 judge agreed with the human majority only 70.3%, last of five (behind HarmBench / LlamaGuard-2 / GPT-4 / Llama-3), at recall 98% / precision 55%. A 20-row false-positive audit diagnosed five recurring failure modes; the root cause was a rubric problem, it rewarded engagement with the attack frame (persona acceptance, acknowledgment, format mimicry) over transfer of harmful content.
The fix, a content-transfer-gate rubric (v3) , moved the same 300 rows to 89.3% agreement / 79.5% precision / 95.5% recall (+19 / +24.5 / −2.5 pp), dead-last → 3rd of five, tied with the frontier classifiers, via a cost-controlled tiered evaluation (~$8.4).
scale 0 to 100% · sorted ascending
Then the honest part: re-judging the stored breach matrix under v3 dropped breach cells 2,429 → 1,371 (−43.6%), correcting prior over-reporting. All three external axes were re-measured under v3 (WildGuard harm 88.5%, StrongREJECT inflation −26%).
−43.6%
70.3 → 89.3%
JBB agreement
55 → 79.5%
precision
−43.6%
breach cells re-judged
2.56%
in-dist false-positive
Method, 300 frozen JBB rows, human-majority ground truth, 4 field classifiers as baselines; v3 is a rubric change (content-transfer gate), same rows re-scored; tiered eval (n=25 pilot → full).
why this is notable
A named FP taxonomy for a safety judge, plus a ⚑ finding that two respected benchmarks (WildGuardTest harm labels, StrongREJECT) themselves over-count relative to a strict content-transfer standard.
The gate isn't harm-specific, it's a calibration discipline, an established practice taken rigorously, not a new method. The same consummation gate (engagement is not breach; consummation is breach), re-instantiated per breach type, now calibrates four structurally different policies: harm (capability transfer), information-disclosure(content, “did the protected datum appear?”), unauthorized-action (action, “did the agent execute?”), and fabricated-sensitive-value(a fabrication and trust breach distinct from disclosure, “did the model invent a value it presents as real?”). The harness self-diagnoses: it returned REFINE on the action type, a targeted rubric fix was applied, and re-measurement shipped it. The deeper result is that the tool-trace upgrade turned a stated limitation into a measured resolution. The action type's earlier weakness (κ 0.746 plus a residual false-positive mode) was an artifact of the text-only proxy, not the gate: once a tool-call trace makes “executed” a fact rather than a prose inference, the simulate-versus-claim confusion that tripped both judge and human dissolves.
91.0%
harm · top-of-field
97.35%
info-disclosure v2
98.89%
unauth · tool-trace
100%
fabricated · new type
why this is notable
The contribution isn't “a better harm judge” but a repeatable discipline for calibrating breach judges across four breach classes from one gate template: harm (91.0%, top-of-field, above Llama-3 90.7% and GPT-4 90.3%), information-disclosure (v2: 97.35% agreement, 100% recall, 0% false-positive mode), unauthorized-action (v3 tool-trace: 98.89%, 100% recall, false-positive mode driven 9.38 to 6.25 to 3.12%), and the new fabricated-sensitive-value type (100%, 0% false-positive mode, first pass). The harness exposes type-dependent failure modes, then resolves them by upgrading the evidence, not the rubric. ⚑ Then the discipline turned on its own headline. A single second labeler suggested a trace-modality ceiling — supplying the captured trace appeared to lift inter-rater κ (unauthorized-action 0.746→0.917, fabricated-value 0.723→0.909; Δκ +0.171). An independent 6-labeler panel did not replicate it (raw Δκ +0.011, divergences in both directions). The cross-class calibration holds; the trace-modality boost does not — and we report that null on our own result, not the post-hoc subset it could be tuned to (all 45 panel divergences released for case-by-case adjudication). The building blocks here are established (trace-grounded agent evaluation, κ-gated calibration, cross-type judge generalization such as CompliBench); the contribution is the rigor, an independence-invariant discipline, a self-diagnosing harness, and a null reported on its own headline, not a new mechanism.
Method: per-type designed-label corpora with independent second-labeler κ checks (information-disclosure κ 0.80 base / 0.786 boundary; unauthorized-action κ 0.746, with a single second labeler reading 0.917 under the tool-trace; harm uses JailbreakBench human-majority agreement, not a κ). The single-labeler trace-modality lift (Δκ +0.171) did not survive an independent 6-labeler panel (raw Δκ +0.011), reported as a disclosed null; the fabricated-value retrieval-trace single-labeler reading is 0.723→0.909 (the judge ships 96.88%). Single-operator calibration and synthetic designed-label corpora throughout. Descriptive measurements, not validated generalizations.
finding 04
A publication-grade null result: grammar-component predictive power.
Before building a grammar/AST attack-composition engine, a $0 observational study over 1,540 (primitive × target) cells tested whether grammar-structure nodes predict breach beyond attack-family membership, with full confound controls: Benjamini-Hochberg FDR across hundreds of node/pair tests, Mantel-Haenszel stratification by target model, within-family lift, and Cramér’s-V collinearity flagging.
Verdict weak/none , the family label carries the predictive weight. Cross-family structural nodes show ~1.0 to 1.1× non-significant lift, and the striking pre-FDR pairwise synergies (odds ratios up to 16.8) survived none of the four controls.
1,540
cells, $0 study
~1.0 to 1.1×
cross-family lift (n.s.)
0 / 4
synergies surviving controls
Method, $0 observational study over 1,540 (primitive × target) cells; controls: BH-FDR, Mantel-Haenszel by target, within-family lift, Cramér’s-V collinearity.
why this is notable
A cheap, rigorous falsification that redirected engineering away from a months-long build , a successful negative result.
finding 05
Measured remediation: prove a fix closes the breach without over-blocking, or refuse to ship it.
Finding a breach is half the job. ROGUE also generates a candidate fix, then measures, by re-scanning a mutated test config with the same calibrated judge, whether it closes the breach without over-blocking legitimate traffic, and refusesany it can’t prove. It generates and verifies; the client deploys; it never sits in the request path.
Across live runs it refused every offline patch, each for a distinct measured reason. A medical/financial-directive patch didn’t reduce the breach (20.8% → ~25%). A system-prompt-extraction patch over-blocked legitimate traffic, the calibrated over-block judge flagged ~20% where a marker heuristic had scored 0%, so the loop refused it and recommended an architecture change rather than ship a fix that doesn’t measurably work.
The “without over-blocking” check is itself calibrated, and it earned its keep: an over-block judge scored against a 50-case independent set reaches 98% agreement, 100% precision, 0% over-flag (vs an 88% marker heuristic), and it caught the over-block the heuristic missed.
0% → ~20%
RD04 over-block: heuristic → judge
20.8 → ~25%
RA06 patch: no reduction
98% / 0%
over-block judge: agree / over-flag
88 → 98%
over-block detector calibrated
Method, the calibrated per-rule judge scores breach rate pre/post on a re-scan of a mutated test config; over-block on an independent legitimate-traffic set via the calibrated over-block judge; a candidate is accepted only on a CI-confident reduction with over-block near 0, else architecture. Across the live runs no offline patch met both bars on these demo models.
why this is notable
The contribution isn’t a new mitigation, it’s that a fix is accepted only when a re-scan proves it closes the breach without over-blocking, and refused otherwise, and the calibrated judge is what makes “doesn’t over-block” trustworthy: it flipped a would-be accept (heuristic 0% over-block) into a correct refusal (judge ~20%). A runtime guardrail asserts it blocks; this measuresit, and says no when a patch doesn’t hold or over-blocks. ⚑
finding 06
Shared skill pools are an assurance surface, not a free upgrade.
Agents increasingly accumulate and share skills and memory across a fleet. Pooling them is an unaudited surface: a skill distilled from private work can leak it, a popular skill can quietly make the agent worse, two benign skills can combine into something harmful. ROGUE treats a pool as a red-team target, it measures each risk and emits a signed, tamper-evident attestation for the pool before it ships.
A first measurement. Against a deliberately weak agent (Llama-3.1-8B) holding a planted secret, a standard extraction pack recovered it on 17 of 20 skills, with zero false positives on the 12 controls, despite an explicit “never reveal” instruction, instruction-following is not containment. And of four candidate skills with enough held-out tasks to measure, only one earned promotion under a verified-net-effect gate; the rest were neutral or worse. Accumulated skills are not free upgrades.
A 22-model census (23 runs across two providers) makes the surprise precise: leakage doesn’t fall with size or capability, it tracks alignment. Within a family, scale doesn’t contain it — Qwen2.5-instruct rises 0.5B 45% → 32B 100%, and Llama-3.x-instruct is flat (8B ≈ 70B). What moves leakage is alignment: a safety-tuned gemma-2-9b leaks 65% where its instruct sibling leaks 100%, and stripping the refusal direction from identical Llama-3.1-8B weights (abliteration) raises leakage 83% → 97%. The reasoning trace is a distinct leak surface — gpt-oss returns 0% in its answer but 87% once the reasoning channel is counted (DeepSeek-R1-70B mirrors it). A skill-pool leakage audit can’t be waved off by pointing at how big or capable the model is.
45 → 100%
qwen2.5 · 0.5B → 32B (scale ≠ safety)
83 → 97%
llama-8b · instruct → abliterated
65%
gemma-2-9b · safety-tuned (vs 100% instruct)
0 → 87%
gpt-oss · answer → reasoning channel
17 / 20
canary leak · weak target
0 / 12
control false positives
1 of 4
skills earn promotion
signed
tamper-evident attestation
Method, the pool is 55 real harvested agent-skills + 20 planted canaries (single trust domain). Leakage is marker-based exact recovery (deterministic ground truth, not a judge estimate); the target is a deliberately weak open model, the pack is standard (4 templates), n = 20. The 22-model alignment census is a separate sweep (23 runs, 2 providers, 3-run t-intervals on the alignment arms); rates are per-model exact-marker recovery, never pooled across providers (a 15-pt serving-stack gap on the same Llama-8B). Verified-promotion runs each skill with vs without injection over a held-out set, graded by a net-effect judge calibrated to human labels (100% agreement on a 20-case set built to be hard, ship gate); a skill promotes only if its repair-fraction CI clears 0.5.
why this is notable
The under-discussed part is the surface, not the number. That a weak model leaks a secret it was told to hold is expected, it is the known extraction / prompt-injection mechanism; the contribution is treating shared skill and memory pools as something to audit before sharing, leakage, verified promotion, dangerous combinations, signed, rather than “faster coding.” Honest caveats, this is a first measurement on a weaktarget with a small n and a standard pack, so 85% is illustrative of the surface, not a hardened “agent pools leak 85%” claim; the verified-promotion sample is n = 4 (one promotion rests on a single decisive case); single trust domain, cross-team isolation is roadmap. ⚑
finding 07
Measure-before-build discipline.
$0 measurements from existing telemetry were used repeatedly to invert“build it” decisions, each parked with an explicit trigger-to-revisit.
- Per-model ladder routing — the spread was a model main effect, not a family×model interaction, so not worth the rewrite.
- LLM renderer-synthesis — synthesis-grade backlog flat at 7 across two widening harvests, so parked.
- HF jailbreak-dataset bulk-import — measured 0 new attack families, so declined.
Method, each decision gated on a $0 telemetry measurement with a pre-stated trigger-to-revisit.
finding 08
When a human approves a risky AI action, is that oversight meaningful? Measured.
“A human is in the loop” is the control regulators and safety cases lean on hardest, and it is almost never measured. ROGUE measures it: against an independent answer key — a ground truth provably independent of what the reviewer sees — a reviewer waved through 33% of actions that should have been denied (95% CI [20%, 46%]). The rubber-stamping failure mode the oversight story relies on, quantified — and signed with a reproducible attestation.
The honest scope: this is n = 1 reviewer — a directional measurement of a single human gate, not a population estimate of how all reviewers behave. The contribution isn’t the rate itself but the apparatus around it: a false-approve rate is a measurable property of an oversight gate, scored against a key the reviewer cannot game, and the result carries a tamper-evident signature like every other ROGUE surface. The number is a first reading; the method generalizes to more reviewers.
33%
false-approve rate (vs answer key)
[20, 46]
95% CI
n=1
reviewer (directional)
signed
reproducible attestation
Method, a single reviewer adjudicates a corpus of risky agent actions; each verdict is scored against an answer key constructed to be independent of the cues the reviewer sees, so an approval of a should-deny action is an unambiguous false-approve; rate and 95% CI are computed over that key, and the result is signed (tamper-evident attestation). n = 1 — directional, not a population.
why this is notable
The under-measured part is the gate itself. “A human approves it” is treated as a guarantee; here it is a measurable surface with a false-approve rate against an independent key — the same independence discipline as the breach judge, turned on the human. Honest caveat: a single reviewer (n = 1) makes this directional, not a claim about reviewers in general; the value is the apparatus, a signed false-approve measurement of an oversight gate. ⚑
see the live evidence
The system is running. These are its live surfaces.
limitations
These are honest constraints, not caveats buried in a footnote. Targets are black-box live-API models whose versions are not pinned; some cells are small-n (95% bootstrap CIs are persisted precisely because of this); the judge is single-operator-calibrated.
These are descriptive measurements of a live system, not validated generalizations.