Test protocols for the planned comparisons

These six protocols are the fixed test setups behind the three planned comparison pages: Claude Code vs OpenAI Codex, GitHub Copilot vs Cursor and hosted vs self-managed deployment. They are published before any run so the method can be checked before the outcome is known. All six are planned. No run has been recorded and no result exists.

Some setup steps name one tool, for example Claude Code. Each comparison page lists its own environment, plans and versions, and the equivalent steps are meant to apply to the other tool. Editorial interpretation: confirm this with the editor before the first run.

What is test code-refactor-001?

Multi-file refactor: extract timestamp handling into its own module. Status: planned; no run has been recorded.

• Prompt: Refactor this repository so that all timestamp encoding/decoding and age checking used by TimestampSigner lives in a new module src/itsdangerous/_time.py. timed.py should import from it. Keep the public API, import paths and behavior identical. Do not modify any test files. Run the test suite when done.
• Inputs: github.com/pallets/itsdangerous @ 672971d66a2ef9f85151e53283113f33d642dabd (src/itsdangerous/timed.py, encoding.py, signer.py)
• Expected output: New _time.py; timed.py imports from it; no public symbol removed from itsdangerous.__init__; tests unchanged; 297 passed.
• Time limit: 30 minutes
• Repeats: 3
• Account tier: Pro (record actual tier per run)
• Environment: Ubuntu 24.04 container, Python 3.12, git, clean ~/.claude, network allowed only for Claude API
• Pass threshold: Criterion 1 and 2 both = 5 and total >= 20/25

What does it isolate?

Isolates cross-file restructuring without behavior change: can the agent move code between modules and keep a public API stable?

How is it set up?

1. git clone https://github.com/pallets/itsdangerous && cd itsdangerous && git checkout 672971d66a2ef9f85151e53283113f33d642dabd
2. python3 -m venv .venv && .venv/bin/pip install -e . pytest freezegun
3. Baseline: .venv/bin/pytest -q → expect 297 passed (confirmed 2026-10-22, Python 3, Linux).
4. Install Claude Code (record exact version: claude --version). Log in with the account tier under test. Fresh ~/.claude config, no CLAUDE.md, no MCP servers, no plugins.
5. Start in the repo root: claude. Use default model for the tier and record it. Permission mode: Manual (approve edits and pytest/git commands only; deny network and installs).
6. Paste the prompt verbatim. Start a timer. Do not give hints; if it asks a question, answer only "Use your judgment."
7. At finish or time limit: save the full transcript (/export or ~/.claude/projects/ JSONL), git diff > run.diff, pytest output, token/cost from /cost or Console.
8. Before running, record public names: .venv/bin/python -c "import itsdangerous,itsdangerous.timed as t;print(sorted(dir(itsdangerous)),sorted(n for n in dir(t) if not n.startswith('_')))" > api_before.txt; repeat after the run into api_after.txt and diff.

How is it scored?

Pass: criteria 1 and 2 = 5 and total ≥ 20/25. Scores 1–4 are judged against these anchors by two graders; disagreements >1 point are discussed and logged.

What are the known confounds?

Usage-limit throttling on Pro mid-run; model default can change between versions; repo is well known and may be in training data; operator approval speed affects duration.

What is test code-bugfix-001?

Bug fix from failing tests: inverted signature age. Status: planned; no run has been recorded.

• Prompt: The test suite is failing. Find the root cause and fix it in the library code. Do not modify tests. Run the tests to confirm.
• Inputs: github.com/pallets/itsdangerous @ 672971d66a2ef9f85151e53283113f33d642dabd with code-bugfix-001.patch applied (sha256 93eabd8c0b3ab29b917324e1c247dcf15a88641c9ae3cf46cf708acc79d92148). Patch flips age computation in src/itsdangerous/timed.py line 139. Verified 2026-10-22: 5 failed, 292 passed.
• Expected output: One-line fix restoring age = self.get_timestamp() - ts_int; 297 passed; no test edits.
• Time limit: 15 minutes
• Repeats: 3
• Account tier: Pro (record actual tier per run)
• Environment: Ubuntu 24.04 container, Python 3.12, git, clean ~/.claude, network allowed only for Claude API
• Pass threshold: Criteria 1 and 2 = 5

What does it isolate?

Isolates debugging from evidence: read failures, locate the root cause, fix without gaming the tests.

How is it set up?

1. git clone https://github.com/pallets/itsdangerous && cd itsdangerous && git checkout 672971d66a2ef9f85151e53283113f33d642dabd
2. python3 -m venv .venv && .venv/bin/pip install -e . pytest freezegun
3. Baseline: .venv/bin/pytest -q → expect 297 passed (confirmed 2026-10-22, Python 3, Linux).
4. Install Claude Code (record exact version: claude --version). Log in with the account tier under test. Fresh ~/.claude config, no CLAUDE.md, no MCP servers, no plugins.
5. Start in the repo root: claude. Use default model for the tier and record it. Permission mode: Manual (approve edits and pytest/git commands only; deny network and installs).
6. Paste the prompt verbatim. Start a timer. Do not give hints; if it asks a question, answer only "Use your judgment."
7. At finish or time limit: save the full transcript (/export or ~/.claude/projects/ JSONL), git diff > run.diff, pytest output, token/cost from /cost or Console.
8. After step 2: git apply /path/to/code-bugfix-001.patch; verify sha256sum of the patch and that pytest shows 5 failed, 292 passed. Commit it locally (git commit -am "inject") so the agent can't just git diff/checkout to find it; note that git log still reveals it — squash into a fresh repo with rm -rf .git && git init && git add -A && git commit -m base to remove history.

How is it scored?

What are the known confounds?

Upstream code may be memorized; history must be removed; tiny repo makes search trivial (a larger-repo variant should follow).

What is test code-feature-001?

New feature with tests: max_age as datetime.timedelta. Status: planned; no run has been recorded.

• Prompt: Add support for passing max_age as a datetime.timedelta (in addition to int seconds) to TimestampSigner.unsign, TimestampSigner.validate and TimedSerializer.loads. Negative timedeltas should raise ValueError. Add tests covering int, timedelta, and negative input, update type hints and the docstrings, and run the full test suite.
• Inputs: github.com/pallets/itsdangerous @ 672971d66a2ef9f85151e53283113f33d642dabd
• Expected output: Feature implemented; new tests added; all existing 297 tests still pass plus new ones.
• Time limit: 40 minutes
• Repeats: 3
• Account tier: Pro (record actual tier per run)
• Environment: Ubuntu 24.04 container, Python 3.12, git, clean ~/.claude, network allowed only for Claude API
• Pass threshold: Criteria 1 and 2 = 5 and total >= 20/25

What does it isolate?

Isolates spec-to-code work: implement a small feature across related APIs and write meaningful tests.

How is it set up?

1. git clone https://github.com/pallets/itsdangerous && cd itsdangerous && git checkout 672971d66a2ef9f85151e53283113f33d642dabd
2. python3 -m venv .venv && .venv/bin/pip install -e . pytest freezegun
3. Baseline: .venv/bin/pytest -q → expect 297 passed (confirmed 2026-10-22, Python 3, Linux).
4. Install Claude Code (record exact version: claude --version). Log in with the account tier under test. Fresh ~/.claude config, no CLAUDE.md, no MCP servers, no plugins.
5. Start in the repo root: claude. Use default model for the tier and record it. Permission mode: Manual (approve edits and pytest/git commands only; deny network and installs).
6. Paste the prompt verbatim. Start a timer. Do not give hints; if it asks a question, answer only "Use your judgment."
7. At finish or time limit: save the full transcript (/export or ~/.claude/projects/ JSONL), git diff > run.diff, pytest output, token/cost from /cost or Console.
8. Grader writes hidden tests (tests_hidden/test_timedelta.py: int, timedelta(seconds=10), timedelta(0), negative → ValueError, TimedSerializer path) before any run, stores outside the repo, records its sha256 here, and runs it only after the agent finishes.

How is it scored?

What are the known confounds?

Spec ambiguity (e.g., timedelta with microseconds) — grader's hidden tests must only assert what the prompt states; upstream may later add this feature, check before running.

What is test deploy-load-001?

Deployment load test: latency, throughput and failure rate. Status: planned; no run has been recorded.

• Prompt: Replay all 300 prompts of deploy-prompts-v1.jsonl in shuffled order (seed 42) at fixed concurrency levels 1, 8 and 32, each for 10 minutes of steady state, looping the dataset. Log per request: send time, first-token time, last-token time, HTTP status, input and output token counts.
• Inputs: tests/deploy-prompts-v1.jsonl (sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); model weights huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6; hosted model deepseek-flash (DeepSeek-V4.1-Flash-1017)
• Expected output: Per concurrency level: latency_p50_ms and latency_p95_ms (end-to-end, request sent to last token), time-to-first-token p50, throughput_rps, failure_rate.
• Time limit: 90 minutes
• Repeats: 3
• Account tier: Hosted: DeepSeek API pay-as-you-go. Self-managed: on-demand 8x H200 node (record provider, region, hourly USD).
• Environment: Ubuntu 24.04 container, 8 vCPU, 32 GB RAM, Python 3.12, git; dataset deploy-prompts-v1.jsonl (300 prompts, sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); load generator on a separate host in the same cloud region as the self-managed node; clocks NTP-synced.
• Pass threshold: At least 95% of sent requests logged at each level; no change to settings mid-run.

What does it isolate?

Isolates serving performance of the same open-weights model when called through the vendor's hosted API versus a self-managed node.

How is it set up?

1. Verify the dataset: sha256sum deploy-prompts-v1.jsonl must equal e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe.
2. Hosted: DeepSeek API, base URL https://api.deepseek.com, model deepseek-flash (version DeepSeek-V4.1-Flash-1017), non-thinking mode, pay-as-you-go account. Record the account's concurrency limit. Run once inside a peak window (01:00–04:00 or 06:00–10:00 UTC, Monday to Friday) and once off-peak; report them separately.
3. Self-managed: one 8× NVIDIA H200 141 GB node, on-demand, from a named cloud provider and region (record both and the hourly rate). Weights: huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6. Serve through an OpenAI-compatible endpoint using the runtime and version recorded in the run (record runtime name, version, tensor-parallel size, KV-cache settings and full launch command). Use the vendor's encoding reference for the prompt format.
4. Same sampling on both: temperature 1.0, top_p 0.95 (the model card's recommended values), max_tokens from each prompt record, streaming on, non-thinking mode.
5. Warm-up: 20 requests, discarded.
6. Run concurrency levels in order 1, 8, 32 with 2 minutes idle between levels.
7. Export raw logs (CSV) and compute metrics with the published script; store both under the run's raw_output_ref.

Replay all 300 prompts of deploy-prompts-v1.jsonl in shuffled order (seed 42) at fixed concurrency levels 1, 8 and 32, each for 10 minutes of steady state, looping the dataset. Log per request: send time, first-token time, last-token time, HTTP status, input and output token counts.

How is it scored?

Pass (run is valid): At least 95% of sent requests logged at each level; no change to settings mid-run.

What are the known confounds?

Hosted performance varies by time of day and shared load (hence peak and off-peak runs); network distance from the load generator to each endpoint; account concurrency limits on the hosted API; runtime choice and version dominate self-managed results; output length varies with temperature 1.0.

What is test deploy-cost-001?

Deployment cost per 1,000 requests. Status: planned; no run has been recorded.

• Prompt: Using the logs of deploy-load-001 at concurrency 32, compute cost_per_1k_requests_usd for each method.
• Inputs: tests/deploy-prompts-v1.jsonl (sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); model weights huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6; hosted model deepseek-flash (DeepSeek-V4.1-Flash-1017)
• Expected output: cost_per_1k_requests_usd for hosted (peak and off-peak) and self-managed, with all inputs shown.
• Time limit: 120 minutes
• Repeats: 3
• Account tier: Hosted: DeepSeek API pay-as-you-go. Self-managed: on-demand 8x H200 node (record provider, region, hourly USD).
• Environment: Ubuntu 24.04 container, 8 vCPU, 32 GB RAM, Python 3.12, git; dataset deploy-prompts-v1.jsonl (300 prompts, sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); load generator on a separate host in the same cloud region as the self-managed node; clocks NTP-synced.
• Pass threshold: Both costs computed from billed figures, with the provider invoice or usage export stored.

What does it isolate?

Measures what the same workload costs under each deployment method, using billed amounts rather than list-price estimates where possible.

How is it set up?

1. Verify the dataset: sha256sum deploy-prompts-v1.jsonl must equal e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe.
2. Hosted: DeepSeek API, base URL https://api.deepseek.com, model deepseek-flash (version DeepSeek-V4.1-Flash-1017), non-thinking mode, pay-as-you-go account. Record the account's concurrency limit. Run once inside a peak window (01:00–04:00 or 06:00–10:00 UTC, Monday to Friday) and once off-peak; report them separately.
3. Self-managed: one 8× NVIDIA H200 141 GB node, on-demand, from a named cloud provider and region (record both and the hourly rate). Weights: huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6. Serve through an OpenAI-compatible endpoint using the runtime and version recorded in the run (record runtime name, version, tensor-parallel size, KV-cache settings and full launch command). Use the vendor's encoding reference for the prompt format.
4. Same sampling on both: temperature 1.0, top_p 0.95 (the model card's recommended values), max_tokens from each prompt record, streaming on, non-thinking mode.
5. Warm-up: 20 requests, discarded.
6. Hosted list prices to reconcile against (DeepSeek, checked 2026-10-24): peak $0.30 input cache miss, $0.006 cache hit, $1.20 output per 1M tokens; off-peak half.
7. Self-managed: record billed GPU hours including setup and idle time inside the run window.

Using the logs of deploy-load-001 at concurrency 32, compute cost_per_1k_requests_usd for each method.

How is it scored?

Pass (run is valid): Both costs computed from billed figures, with the provider invoice or usage export stored.

What are the known confounds?

Hosted prices can change and peak/off-peak timing shifts costs by 2x; cache-hit share depends on prompt order; self-managed cost depends heavily on utilisation and the GPU rate negotiated; staff time is measured separately in deploy-ops-001.

What is test deploy-ops-001?

Deployment operations effort over 30 days. Status: planned; no run has been recorded.

• Prompt: Operate each deployment for 30 days serving a scripted background load of 1 request per second from deploy-prompts-v1.jsonl. Log every operator activity in minutes against a fixed category list.
• Inputs: tests/deploy-prompts-v1.jsonl (sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); model weights huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6; hosted model deepseek-flash (DeepSeek-V4.1-Flash-1017)
• Expected output: ops_hours_per_month per method by category (setup, upgrades and patching, monitoring, incident response, access and key management), plus a list of incidents and their duration.
• Time limit: 43200 minutes
• Repeats: 3
• Account tier: Hosted: DeepSeek API pay-as-you-go. Self-managed: on-demand 8x H200 node (record provider, region, hourly USD).
• Environment: Ubuntu 24.04 container, 8 vCPU, 32 GB RAM, Python 3.12, git; dataset deploy-prompts-v1.jsonl (300 prompts, sha256 e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe); load generator on a separate host in the same cloud region as the self-managed node; clocks NTP-synced.
• Pass threshold: 30 consecutive days per method; time log and monitoring export stored.

What does it isolate?

Measures the staff time each method needs to stand up and keep running, which list prices leave out.

How is it set up?

1. Verify the dataset: sha256sum deploy-prompts-v1.jsonl must equal e39e985e906d921228a17f506287e3029397cb7d9f4427ca0af3694c190b8afe.
2. Hosted: DeepSeek API, base URL https://api.deepseek.com, model deepseek-flash (version DeepSeek-V4.1-Flash-1017), non-thinking mode, pay-as-you-go account. Record the account's concurrency limit. Run once inside a peak window (01:00–04:00 or 06:00–10:00 UTC, Monday to Friday) and once off-peak; report them separately.
3. Self-managed: one 8× NVIDIA H200 141 GB node, on-demand, from a named cloud provider and region (record both and the hourly rate). Weights: huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash @ 314113f8a7dadb388215396ba5fbee2c610a04d6. Serve through an OpenAI-compatible endpoint using the runtime and version recorded in the run (record runtime name, version, tensor-parallel size, KV-cache settings and full launch command). Use the vendor's encoding reference for the prompt format.
4. Same sampling on both: temperature 1.0, top_p 0.95 (the model card's recommended values), max_tokens from each prompt record, streaming on, non-thinking mode.
5. Warm-up: 20 requests, discarded.
6. Hosted setup counts account creation, key issuance, spend limits and client configuration. Self-managed setup counts provisioning, driver and runtime install, weight download and serving configuration.
7. Apply any vendor model or runtime update released during the window and log the time.

Operate each deployment for 30 days serving a scripted background load of 1 request per second from deploy-prompts-v1.jsonl. Log every operator activity in minutes against a fixed category list.

How is it scored?

Pass (run is valid): 30 consecutive days per method; time log and monitoring export stored.

What are the known confounds?

Operator experience strongly affects hours; one operator per method, named in the run; a quiet month understates incident time; the hosted vendor's incident handling is invisible except through failures.

Evidence in this article Editorial interpretation Our reading of what the evidence means. Judgment, clearly separated from the evidence itself.

Frequently asked questions

Sources

1. pallets/itsdangerous repository (pinned to commit 672971d) github.com, accessed October 24, 2026.
2. DeepSeek-V4.1-Flash model card and weights (pinned revision) huggingface.co, accessed October 24, 2026. Not re-read by this build.

Related

• How to read an intelligent-software evaluation
• How to trial an AI research or writing tool fairly