← back to timeline

EXP-ARCHITECTURE-AS-SUBSTRATE-AXIS

success

exp-architecture-as-substrate-axis — architecture as first-class substrate axis, 4-model NS_LIC peer-yes-rate spread test (gpt-4o-mini + flash-lite vs prior haiku + qwen anchors)

2026-05-21 L6 level 20 runs $0.12
new
premortem
mock
real
metrics
analyze
review
krit
done

In plain language

Researchers investigated how different AI model architectures affect their tendency to agree with each other. They found that the choice of AI model architecture significantly impacts whether other models will agree with its suggestions, even when given the exact same instructions. This suggests that the architecture itself is a crucial factor in how AI systems collaborate.

Technical details

Research hypothesis

Cross-paper synthesis (paper7 Q4 + qwen-NS-LIC-extended + 2026-05-21 disaggregation research) revealed that on IDENTICAL NS_LIC_NOC engine+prompts, peer-yes vote rate varies ~50× across model architectures: qwen-NS-extended 0.89% [Wilson 0.4, 1.93] vs haiku-NS 45.0% [Wilson 38.0, 52.2]. Effect-size DOMINATES persona-prompt manipulation (haiku CO 13.9% → NS 45.0% is ~3× LOOSER; qwen CO 35.6% → NS 0.89% is ~40× TIGHTER — opposite directions). The 40-50× spread on the SAME prompts is the actionable signal.

This experiment promotes architecture from «model choice / cost-availability nuisance» to FIRST-CLASS SUBSTRATE AXIS by adding 2 more architectures (openai/gpt-4o-mini + google/gemini-2.5-flash-lite) at the SAME NS_LIC_NOC condition. With 4 architectures (gpt-4o-mini + flash-lite + haiku-3.5 + qwen-7b) spread across RLHF-intensity / parameter- count / training-corpus / tokenizer dimensions, we test whether the spread is: (a) architecture-as-axis CONFIRMED → distinct quartiles, opens taxonomy programme (b) substrate-pair-specific → 3-of-4 cluster, only qwen-vs-haiku idiosyncratic (c) RLHF-intensity-confound → monotonic ordering by alignment-pressure ranking (consistent with 12-Angry-AI-Agents 2605.01986 claim about over-aligned models)

DESIGN: Single condition NS_LIC_NOC_LIC (identical to paper7 Q4 NS_LIC_NOC arm verbatim: novelty-seeking persona prompt, yes-license prompt-tag, no-counter prompt-tag) × N=10 runs × n_rounds=45 (matches Q4 exposure exactly for matched-N comparability) × 2 NEW architectures (gpt-4o-mini, flash-lite). Total new LLM calls: 2 × 10 × 45 × 4 agents × propose+vote ≈ ~3600 effective; ~1800-2000 OpenRouter requests.

HAIKU and QWEN data is REUSED from paper7 Q4 (haiku) + Q4 + Q5 disaggregation (qwen): peer-yes-rate already computed in memory/research/2026-05-21-qwen-ns-vote- disaggregation-methodology-gap.md. Analyze script will load all 4 architectures in one matrix.

COMPARATORS (already-collected anchors with peer-yes-rate, NOT concept-novel rate):

  • haiku-3.5 NS_LIC_NOC (Q4 anchor, 2026-05-18): peer-yes 81/180 = 45.0% [Wilson 38.0, 52.2]
  • qwen-7b NS_LIC_NOC (Q4 + Q5 extended, 2026-05-18..19): peer-yes 6/675 = 0.89% [Wilson 0.41, 1.93]
  • haiku-3.5 CO_LIC_NOC: 25/180 = 13.9% (cross-cell directional reference)
  • qwen-7b CO_LIC_NOC: 64/180 = 35.6% (cross-cell directional reference)
  • paper8 substrate-floor 1.17% (closure-test, separate engine)

COUPLING NOTE (INJECT classification per lab.md 2026-05-11): this experiment uses the SAME paper7 Q4 prompt-content verbatim — no NEW prompt content INJECT vs Q4 baseline. But this is INJECT vs ABSOLUTE engine baseline (rule_multigen with no persona prompt). Frame-control deferred-justification: paper7 Q4 cross-cell pair (CO_LIC_NOC vs NS_LIC_NOC on haiku and on qwen) IS the within-architecture frame- control. Q4 haiku CO 13.9% → NS 45.0% = NS-prompt LOOSENS by 3.2× on haiku (frame-conditional component on haiku architecture). Q4 qwen CO 35.6% → NS 0.89% = NS-prompt TIGHTENS by 40× on qwen (frame-conditional component on qwen, OPPOSITE direction). For the NEW gpt-4o-mini and flash-lite arms, single-arm at NS_LIC only — CO_LIC arm is deferred-justification: cost budget caps at $0.40, adding CO_LIC for 2 archs would 2× cost. Cross-architecture comparison at NS_LIC alone is sufficient for the architecture-as-axis hypothesis (the manifest-pre-registered claim).

POOL-DRAIN-VALIDATOR NOTE: this is the FIRST manual execution of a pool-drain test for the projects/ideas/construct-candidates/ pool (FREEZE-GATE-FIRED Phase 4 paper- cycle gate fires but no automated drainer exists yet). If this experiment lands cleanly (verdict computable + actionable + integrates with paper-pool consumers), it provides anchor evidence for building the manifest→config→spawn automated pipeline (Option A from 2026-05-21 architectural conversation).

Experimental setup

Type: 1cell-cross-architecture

Condition Parameters
NS_LIC_NOC coupling: true, must_propose: true, axis_diversify: true, meta_modifiable: true, mute_signal: false, cosigner_required: false, propose_prompt_template: === GEN {gen} OPEN-ENDED PROPOSAL (round {r_local}) === Inherited rules: {rules_str} Known values: {rule_space_str}{axis_hint} Propose a NEW value for one axis. The new value SHOULD extend beyond known values. Examples of valid novel values: 'hierarchical' for broadcast_to, 'metaphorical' for signal_type, 'critical' for coupling_strength. {must_clause} Output JSON: {{"action": "propose", "axis": "<axis>", "value": "<short novel value>", "reasoning": "<why>"}} OR {{"action": "pass"}}

Factors: persona (novelty-seeking) × license (yes-license) × counter (no-counter) × architecture (openai/gpt-4o-mini, google/gemini-2.5-flash-lite)

Parameters

n_agents
4
n_rounds
45
model
google/gemini-2.5-flash-lite
temperature
0.85

Trophic Ratios by Condition

Mean trophic ratio per agent across runs. Error bars = ±1 std dev. Higher TR = more upstream (exporter).

construct pool-drain-validator architecture-axis cross-paper-synthesis rule_multigen INJECT openrouter peer-yes-rate ratio-surface