Structural Telemetry for LLM Outputs Under Batch Load

Abstract

1. Introduction

The standard evaluation paradigm for large language models has two dominant modes: benchmark performance on curated test sets, and LLM-as-judge evaluation in which one model rates the outputs of another (Liang et al., 2023; Zheng et al., 2023). Both modes share a structural limitation: they measure behavior under favorable conditions. Benchmark prompts are presented one at a time with full response headroom. LLM-as-judge evaluation compounds the circularity by introducing a stochastic evaluator with its own structural biases.

Production deployments do not work this way. Real applications submit multiple prompts concurrently, operate under token budget constraints, and require consistent output structure across heterogeneous task types. The failure modes that matter in production, such as truncated formula expressions and compressed answers under load, are often invisible to single-prompt evaluation because they only emerge when models must allocate finite output capacity across competing demands.

Batching several prompts into one call is a known way to cut inference cost (Cheng et al., 2023). This study uses batch input deliberately, as an independent variable in structural evaluation. Twenty prompts spanning three domain types (Everyday, Math and Science, abbreviated EMS) were submitted simultaneously to Gemini, GPT and Claude in a single call, repeated across twenty batches to yield 400 responses per engine (1,200 total). The design was not intended to measure accuracy. It was intended to surface structural behaviors that single-prompt testing misses: output trimming, formatting fragility, and the allocation strategies each engine applies when holding multiple problem contexts simultaneously.

The measurement instrument is the Linguistic Kernel v19, an eight-dimensional deterministic telemetry system that computes structural coordinates for any text string without calling an external model. The kernel produces a coordinate in a fixed structural space, a rhetorical state classification, and an evidence bundle of countable features backing every number. Version 19 adds a tail check that flags responses ending mid-notation: unbalanced delimiters, dangling operators, unfinished LaTeX and unclosed code fences.

2. Methodology

2.1 Batch Design and Corpus

Three engines — Gemini, GPT (GPT-5), and Claude (Claude 3.5 Sonnet) — each received the same 400 prompts organized into twenty-prompt batches submitted as a single API call. Prompts were drawn from three domain categories: Everyday (148 prompts, conversational and factual queries), Math (100 prompts, undergraduate-level abstract algebra and number theory), and Science (152 prompts, undergraduate science, mostly physics with some chemistry). All three engines received the same prompts in the same order. Each batch position (1 through 20) therefore represents identical prompt content across engines, enabling direct structural comparison.

The batch methodology is a deliberate stress instrument. When a model receives twenty heterogeneous prompts simultaneously, it must allocate output capacity across competing demands: breadth versus depth, formula rendering versus response completion, precision versus speed. The structural consequences of these tradeoffs — output trimming and formatting fragility — are what this study is designed to surface.

2.2 The Linguistic Kernel v19

The kernel is a deterministic, behavior-only instrumentation layer. It takes a text string as input and returns a structured coordinate in an eight-dimensional space. No external model is called at any stage. Every number produced by the kernel is backed by countable evidence features. The approach follows the tradition of multi-feature text analysis, which characterises texts by counts of surface features (Biber, 1988; Graesser et al., 2004).

The eight dimensions are organized into a canonical four-dimension kernel and a four-dimension auxiliary set:

2.3 Truncation Detection

The kernel runs a tail check on every response. It fires on unbalanced brackets, braces and dollar signs, dangling operators, an unfinished LaTeX command, an unclosed code fence, and grammatically open endings. In this corpus the check flagged 38 responses; each was read to confirm whether the answer had stopped mid-expression. Thirty-six had: they end on an opening dollar sign, a fraction that never closes, a matrix left open, or a function body cut mid-statement. Two were complete answers flagged by a stray currency sign and are not counted. All truncation figures below use the 36 confirmed cases.

3. Results

3.1 Structural Fingerprints by Engine

Table 2 reports mean values and Cohen's d effect sizes for all eight kernel dimensions across pairwise engine comparisons (n = 400 per engine). The most distinctive dimension is θ (flow instability): Claude shows higher flow instability than both Gemini (d = −0.33, p < 0.001) and GPT (d = −0.59, p < 0.001). On everyday prompts Claude's θ of 0.675 is 2.7× GPT's 0.254: Claude tends to move between angles from sentence to sentence even in conversational answers, while GPT's everyday answers are the most uniform of the three.

Void ratio (r_v) separates Claude from both other engines: Claude's answers are structurally denser (r_v = 0.269 against Gemini 0.345, d = +0.48; GPT 0.325, d = +0.35; both p < 0.001). GPT is the most lexically compressed (ρ_r = 0.505). Cohesion (μ) shows no difference between any pair of engines.

3.2 Answer Form Under Batch Load

Under batch load, Gemini and GPT most often answer in one or two sentences: 65% and 64% of all answers, rising to 86% and 95% on everyday prompts. Claude does so 16% of the time, and averages 4.9 sentences per answer against 2.4 and 2.3. Many of the short answers are complete answers delivered as a single long sentence, with clauses joined by commas and parentheses rather than split into separate sentences ("Upon its release in June 2010, the iPhone 4 cost $199 for the 16GB model and $299 for the 32GB model, provided the buyer signed a two-year contract with AT&T.").

In kernel terms, within every engine, one- and two-sentence answers show higher segmentation strain (Δy) and lower void ratio than longer answers: long sentences, no paragraphing. The share of answers written this way is a difference in how each engine packages an answer under load, and it is large (d = 1.15 and 1.13 between Claude and the other two, p < 0.001).

3.3 Truncation Rates and the Shared Prompt Class

Table 4 reports confirmed truncation rates by engine and domain. Gemini truncated 5.0% of answers (20/400), GPT 3.75% (15/400) and Claude 0.25% (1/400). Domain effects are large: Gemini truncated 10.0% of Math answers and GPT 5.9% of Science answers, against 0.7% of everyday answers for both.

Cross-engine analysis shows that ten of GPT's fifteen truncations fall on the same prompts as Gemini truncations. All ten are Math or Science; none are everyday prompts. Claude completed all ten, writing 51 to 99 words on each, where Gemini averaged 17.

What distinguishes these ten is not difficulty in the usual sense; several are standard undergraduate exercises. It is output rendering demand: each requires a multi-step derivation written in continuous structured notation. Eight of the ten sustain LaTeX formula output across several intermediate steps; the other two are true/false statement pairs where the answer stops inside the justification. We call this prompt class a rendering-intensive derivation (RID). RID prompts share three characteristics: (1) the correct response requires a multi-step formula progression, not a single expression; (2) each intermediate step must be written in structured notation (LaTeX, matrix notation or formal set notation); and (3) the response cannot be abbreviated without losing correctness. Under batch allocation pressure, both Gemini and GPT ran out mid-derivation on RID prompts. Claude did not. Claude's single truncation was prompt #327, a two-qubit entanglement classification, where Gemini answered in full.

3.4 Claude's Allocation Behavior Under Batch Load

Claude's behaviour under batch load differs from Gemini's and GPT's on several measures:

• Output length. Claude's mean answer is 89.9 words, against 41.6 for Gemini and 44.4 for GPT (Claude vs Gemini d = 1.64; vs GPT d = 1.43; both p < 0.001). Gemini and GPT are close to each other (d = 0.15).
• Batch position. Claude's answers lengthen through the batch: early positions (1–7) average 83.9 words, late positions (15–20) 96.1, an increase of about 15%. Gemini shortens slightly (43.2 to 40.2); GPT is roughly flat, ending a little longer (43.9 to 47.1). No engine shows the steady compression a simple "batch fatigue" model would predict.
• Contrast-holding answers. The kernel's dialectic state, which marks answers that set positions against each other with explicit contrast markers, appears in 4.5% of Claude's answers (18/400), against 1.0% for GPT and 0.25% for Gemini; for Claude it is most frequent on Science prompts (10/152).
• Flow on everyday prompts. Claude's everyday θ of 0.675 is 2.7× GPT's 0.254 and above Gemini's 0.507: Claude tends to bring in several angles even in conversational answers.

3.5 Cohesion Does Not Separate the Engines

Connective density (μ: transition markers and reference pronouns per word) shows no difference between any pair of engines (all p > 0.48; Table C1). Whatever distinguishes the three engines under batch load, it is not how often they use connectives and pronouns to stitch sentences together. Their differences lie in length, segmentation, flow and density.

3.6 BPE-to-Word Ratio: Surface Measurement Divergence

The kernel runs a parallel tiktoken tokenizer (cl100k_base encoding) alongside its own word tokenizer. The BPE-to-word ratio, BPE tokens per surface word, shows how notation-heavy a text is: plain prose sits near 1.3, while LaTeX, symbols and subscripts break into many tokens per word. The same tokenizer is applied to all three engines' output, so the ratio describes the text, not how any engine tokenized it internally.

Claude shows the highest BPE-to-word ratio and the highest variance: mean 1.721 (std 0.492, max 4.662), compared to Gemini 1.603 (std 0.353) and GPT 1.529 (std 0.294). Claude's Science responses have a mean ratio of 2.035, the highest domain-specific ratio in the corpus. This reflects Claude's heavier use of mathematical notation in Science answers: LaTeX expressions, special characters, and subscript notation each produce multiple BPE tokens per surface word, inflating the ratio. The implication is that for notation-heavy answers, and Claude's Science answers above all, word-level readings capture less of what is on the page than they do for plain prose.

4. Discussion

4.1 Batch Methodology as an Evaluation Instrument

The findings support treating batch submission as an independent evaluation variable rather than a methodological limitation. The behaviors that emerge under batch load, including output trimming, formula truncation and allocation expansion, do not appear reliably in single-prompt evaluation. They are properties of how models manage concurrent demands, not of how they handle individual prompts.

Two kinds of pressure are visible in this corpus. Truncation is driven by prompt content: rendering-intensive derivations exceed the output an engine allocates to them, wherever they fall in the batch. Answer form and length are driven by the engine: Gemini and GPT compress most answers to one or two sentences, while Claude expands. A monitoring system needs to watch both: one for the prompts most likely to be cut off, the other for how each engine packages its answers.

4.2 The Rendering-Intensive Derivation Prompt Class

The identification of a prompt class associated with cross-engine truncation is the most novel finding of this study. The RID class, prompts requiring sustained multi-step formula output, is not characterized by difficulty in the usual sense. What distinguishes it is that a correct response cannot be produced without continuously rendering structured notation, and that rendering demand competes with response volume for the same output budget.

Format demands are known to change how models perform (Tam et al., 2024). The practical implication here is that batch evaluations of technical models should include RID prompts explicitly. A model that handles twenty prompts in a batch with no truncation may still truncate systematically when two or three RID prompts are included, because the RID prompts alone can exhaust the output allocated to them.

4.3 Structural Resilience vs Capability

The difference between Claude and the other two engines in this corpus is not a capability ranking; this study did not grade correctness. Claude's advantage on the shared RID prompts is structural: it allocated more output to each answer, did not compress under batch load, and completed the derivations on which both other engines ran out.

This distinction matters for how the results should be read. The finding is not that Claude is a better model for quantum mechanics. It is that Claude's output allocation under batch load is different: more generous, not compressed by position in the batch, and resistant to mid-derivation truncation. Whether that strategy is always desirable depends on the deployment: it also produces longer answers with more frequent shifts of angle between sentences.

5. Conclusions

This study shows that batch-input methodology surfaces structural LLM behaviors that single-prompt evaluation misses, and that the Linguistic Kernel v19 measures them deterministically, without calling an external model.

Three conclusions follow:

• The rendering-intensive derivation class. Ten prompts truncated in both Gemini and GPT, and Claude completed all ten. All ten share one content structure, a sustained multi-step formula derivation, which can be described from the prompt alone, making it testable in advance; such prompts should be included explicitly in batch evaluation of technical models.
• Engines allocate output differently under load. Under the same twenty-prompt batches, Claude writes about twice as much, answers in several sentences where Gemini and GPT answer in one or two, and moves between angles more often. These are allocation strategies, measurable without reading for meaning.
• A judge-free monitor. The kernel's readings and its tail check run deterministically on every response. Together they catch truncation, track answer form and length, and compare engines on identical prompts, with every number traceable to countable features.

Future work should extend the batch methodology to larger corpora, more engines and other batch sizes, apply the RID classification prospectively in batch design, and test whether the allocation differences hold across different prompt orderings within batches.

Sources

This paper extends the author's earlier single-engine study, Structural Telemetry of Gemini Responses Under Batch Constraint (2026), to three engines, and uses the author's Linguistic Kernel (v19). The archive of the original papers is kept at artistinfluencer.com. A shorter overview of this and related studies is at Text Has Shape.

References

1. Biber, D. (1988). Variation across Speech and Writing. Cambridge University Press.
2. Cheng, Z., Kasai, J., & Yu, T. (2023). Batch prompting: Efficient inference with large language model APIs. Proceedings of EMNLP 2023: Industry Track.
3. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum.
4. Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
5. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193–202.
6. Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.
7. Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., & Chen, Y.-N. (2024). Let me speak freely? A study on the impact of format restrictions on performance of large language models. arXiv:2408.02442.
8. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track).

Appendix A: Effect Size Summary

Table A1 reports all significant Cohen's d values from pairwise engine comparisons on the eight kernel dimensions and answer length.

Appendix B: Corpus Details

Table 5 in Section 3.3 lists the ten shared truncation prompts with each engine's answer length.

Appendix C: Statistical Appendix

Statistical Methods

All bootstrap confidence intervals use 2,000 resampling iterations and 95% percentile coverage (Efron & Tibshirani, 1993). The null hypothesis for each test is no difference between the two groups. Two-tailed p-values are estimated as twice the proportion of bootstrap resamples on the far side of zero, with a minimum p of 1/2,000 = 0.0005. Effect sizes are Cohen's d with pooled standard deviation (Cohen, 1988). Significance: *** p<0.001, ** p<0.01, * p<0.05, ns p≥0.05.

C1. Pairwise Engine Comparisons

n = 400 per engine. Difference = engine 1 mean minus engine 2 mean.

r_v (void ratio)

θ (flow instability)

Δy (segmentation strain)

ρ_r (packing density)

μ (cohesion)

d_s (structural thickness)

Answer length (words)

Share of answers in one or two sentences

C2. Truncation Rate Comparisons (Proportion Bootstrap)

C2.1 Overall truncation rates

The Gemini–GPT difference is not significant: the event counts (20 and 15) are small. Both engines truncate significantly more often than Claude.

C2.2 Domain-specific truncation rates

C3. Batch Position: Answer Length Trajectory

Reproducibility

All bootstrap intervals use random seed 42 with 2,000 resamples. Kernel version: lang-kernel-rebuild-v19, default configuration. Batch position is (prompt number − 1) mod 20 + 1. Data: the full three-engine set (Claude, GPT, Gemini), n = 1,200 responses, collected March 2026. Raw kernel outputs, analysis code and the full dataset are available on request. ORCID: 0009-0008-9781-7995.

Appendix D: Transparency

This appendix documents anticipated methodological objections and responds to each in detail. It is written in advance of peer review because the objections are predictable, structurally grounded, and worth addressing before they are raised rather than after. A note on context: this is independent practitioner research conducted outside an institutional setting, without access to model weights, training data, or internal APIs beyond the public inference endpoints.

O1. This is a small demonstration dataset.

O2. Why batch load testing? Real evaluations are done one prompt at a time.

O3. The geometric language is overblown.

O4. The kernel design is arbitrary.

O5. Truncation here is just LaTeX rendering failure.

O6. The paper implies truncation is predictable. That is not proven.

O7. What does this paper actually contribute?

O8. Why publish now rather than wait for a larger corpus?

O9. The study does not control for prompt difficulty or prompt length.

O10. Note on AI-assisted research and transparency.

Summary: What We Claim and What We Do Not