The Empirical Relational Benchmark: A Hybrid Character, Word, Shard, and Provenance Test Architecture for Language Models and Knowledge Systems: A Secretary Suite Project
DOI: [To be assigned]
John Swygert
August 25, 2026
Abstract
This paper converts the relational architecture developed in Papers 200–207 into an explicit empirical program. The central proposal is that symbolic and knowledge systems should not be evaluated at only one representational scale. Characters, subword sequences, words, phrases, sentences, shards, relational structures, provenance paths, and reconstruction histories can each carry useful information. A hybrid system should therefore compare multiple analytical channels and select or combine those that provide the highest discriminative value at acceptable computational cost.
The paper introduces a multiscale benchmark spanning character n-grams, lexical n-grams, relational shards, structural similarity, provenance, retrieval utility, and controlled perturbations of order, boundary, scope, and context. It also reports a preliminary corpus demonstration using the 294-page novel AInunnaki (67,023 extracted words). Exact lexical recurrence was sparse at long sequence lengths: 15 repeated 8-word sequences, 5 repeated 10-word sequences, 2 repeated 12-word sequences, and no repeated 15-word sequences. Character-level analysis, by contrast, produced many more recurring candidates, demonstrating both its sensitivity and its need for statistical calibration. The benchmark therefore treats character and word analysis as complementary rather than competing methods.
The broader objective is to test whether explicit relational information improves retrieval, provenance analysis, originality control, code diagnosis, agent error localization, and shard reconstruction beyond ordinary semantic or lexical baselines. The architecture remains deliberately falsifiable: mappability is not validity, and hybrid complexity is justified only when it produces measurable advantage.
1. Purpose
Papers 200–207 developed a relational account of language, LLMs, agents, code failure, knowledge shards, provenance, reconstruction, and orchestration. The next requirement is empirical discrimination. A useful theory must survive contact with data and must be compared against simpler alternatives.
Theory → Architecture → Prototype → Experiment → Validation or Revision
This paper specifies that experimental step. It is designed not to prove the relational framework correct, but to create conditions under which it can succeed, fail, or require revision.
2. Why One Resolution Is Not Enough
Most text analysis systems privilege one or two representational levels. Exact matching operates on strings; token systems operate on words or subwords; embedding systems operate on vectorized semantic similarity; retrieval systems often treat document chunks as the basic unit. Each level captures some information and discards other information.
characters → subwords → words → phrases → sentences → shards → relational structures
A short character sequence may reveal reuse that word-level matching misses. A word sequence may identify a distinctive phrase more efficiently than a high-dimensional semantic comparison. A relational shard may preserve provenance, scope, dependency, and function that neither character nor word similarity can recover.
The benchmark therefore rejects the assumption that one analytical layer must replace all others.
3. The Hybrid Principle
The proposed system contains parallel analytical channels rather than a single detector. Let F_c represent character-level evidence, F_w lexical evidence, F_s relational-shard evidence, F_r structural evidence, and F_p provenance evidence.
F_hybrid = w_c F_c + w_w F_w + w_s F_s + w_r F_r + w_p F_p
The weights need not remain fixed. A plagiarism-prevention task may value character and phrase recurrence heavily. A conceptual retrieval task may weight shard role and structural relation more strongly. A provenance task may give origin and reconstruction history dominant weight.
The guiding optimization criterion is practical:
Use the representation, or combination of representations, that produces the best useful discrimination for the required cost.
This prevents the framework itself from becoming doctrinaire. If a simple exact matcher performs best for a task, the hybrid system should use it.
4. Character-Sequence Analysis
Character n-grams capture recurring letter sequences without requiring word boundaries. They can detect spelling variants, inflectional overlap, fragments crossing word boundaries, punctuation-adjacent patterns, and lightly altered copying.
C_n(X) = {x_i x_(i+1) ... x_(i+n-1)}
Character analysis is highly sensitive. That sensitivity is an advantage and a liability. Common morphemes, spaces, punctuation patterns, and overlapping windows can generate large numbers of repeated candidates.
Consequently, character evidence must be calibrated against expected background frequency. A repeated character sequence is not automatically meaningful; its distinctiveness depends on length, corpus frequency, location, overlap, and relation to higher-level structure.
5. Word-Sequence Analysis
Word n-grams are less sensitive to minor spelling and formatting changes but more interpretable. They can identify recurring phrases, repeated sentence fragments, standardized terminology, deliberate motifs, and potentially overused prose.
W_n(X) = {(w_i, w_(i+1), ..., w_(i+n-1))}
Long exact word recurrences are particularly useful because their accidental probability generally decreases as sequence length grows, although common formulaic language remains a confound.
6. Shard-Level Analysis
A shard is not defined only by length. It is a bounded relational unit carrying content together with relevant metadata such as provenance, scope, state, dependency, authority, and history.
Shard = Content + Relation + Provenance + Scope + State
Two shards may be lexically dissimilar while instantiating the same relational architecture. Conversely, two shards may contain similar language while serving entirely different roles.
Shard analysis therefore asks not only 'how similar are these passages?' but also 'what relationship do they perform?'
7. Provenance, Inheritance, and Independent Convergence
The AInunnaki test corpus exposed an important distinction that should be explicit in the architecture. Structural similarity does not establish historical derivation.
Structural Similarity ≠ Shared Provenance
Shared Structure ≠ Historical Derivation
A relational library should therefore support edges such as derived_from, generalizes, instantiates, modifies, contradicts, supersedes, recurs_as, quotes, preserves, transforms, and independently_converges_on.
Independent convergence is especially important. Similar relational solutions may emerge separately because different systems encounter the same constraints. A plagiarism or provenance system that collapses similarity into derivation will generate false accusations.
8. Preliminary Corpus Demonstration: AInunnaki
A preliminary corpus demonstration was conducted on AInunnaki, a 294-page novel. PDF text extraction produced 517,250 characters and 67,023 whitespace-delimited words before normalization.
After normalization and tokenization, exact overlapping lexical n-gram recurrence was measured across the manuscript.
The dominant long recurrence was the repeated principle built around teaching receivers enough to choose wisely without choosing their future for them. The other substantive 10-word recurrence was the recognition sequence 'I see you. You see me. No threat. A beginning.' These are narrative motifs rather than evidence of accidental duplication.
9. Character-Level Demonstration
The same corpus was analyzed using rolling normalized character n-grams. Because character windows overlap heavily, the number of repeated candidates was much larger:
These counts should not be interpreted as thousands of meaningful repeated expressions. Overlapping windows around one repeated phrase can generate many neighboring n-grams, and ordinary linguistic fragments recur naturally. The result instead demonstrates why character analysis must be clustered, de-overlapped, frequency-normalized, and interpreted jointly with word- and shard-level evidence.
This is precisely the reason to retain both channels. Character analysis maximizes sensitivity; word and shard analysis improve interpretability and specificity.
10. De-Overlapping and Clustering
A robust character detector should merge overlapping repeated windows into maximal recurring spans. Without this step, one repeated 60-character passage may appear as dozens of repeated 20-, 24-, 32-, and 40-character n-grams.
overlapping repeated windows → maximal recurrent span → one candidate fingerprint
Cluster-level reporting prevents a high-sensitivity detector from exaggerating evidence through window multiplication.
11. Distinctiveness and Background Frequency
Every fingerprint requires a background model. Common expressions, affixes, technical terms, boilerplate, punctuation habits, and genre conventions should receive lower distinctiveness scores than rare combinations.
Distinctiveness(s) ∝ -log P_background(s)
The exact statistical model can vary. The essential rule is that recurrence must be interpreted relative to expected recurrence.
12. A Multiscale Fingerprint Vector
Rather than collapse all evidence immediately into one score, each artifact can be assigned a multiscale fingerprint vector:
F = [C, W, P, S, R, V, H]
C: character-sequence signature
W: word- and phrase-sequence signature
P: punctuation and formatting signature
S: shard-level signature
R: relational-structure signature
V: provenance signature
H: reconstruction or interaction-history signature
This preserves diagnostic information. Two artifacts may have low lexical overlap but high relational similarity, or high lexical similarity but clearly independent provenance.
13. Retrieval Benchmark
The retrieval experiment should compare at least four systems: lexical retrieval, semantic embedding retrieval, relational retrieval, and hybrid retrieval.
Evaluation should include top-1 accuracy, top-k recall, mean reciprocal rank, target rank, latency, computational cost, and downstream task success.
14. Retrieval Utility Versus Retrieval Frequency
A shard that surfaces often is not necessarily useful. Let f_i be the number of times shard i is surfaced and u_i the number of times it contributes to a successful output.
Utility Ratio_i = u_i / f_i
Frequency measures exposure. The utility ratio measures acceptance or functional contribution. Both should be retained because a rarely surfaced but nearly always essential shard differs fundamentally from a frequently surfaced but consistently discarded shard.
15. Controlled Relational Perturbations
The benchmark should deliberately alter one relational dimension while holding others constant. Candidate interventions include punctuation, word order, grouping boundary, operator scope, instruction hierarchy, context position, provenance label, and shard role.
ΔR_i → measure ΔY
A true relational diagnostic should predict which interventions matter and where the failure will occur, rather than merely observing that outputs changed.
16. Agent and Code Tests
The same benchmark can be extended beyond prose. Agent tasks can test instruction precedence, retrieved-text boundaries, tool-result scope, and memory contamination. Code tasks can test delimiters, nesting, scope, order, state, interfaces, and dependency boundaries.
The minimum corrective intervention principle should be measured directly: does a relational diagnosis lead to smaller successful repairs than unconstrained regeneration?
δC_min = smallest correction that restores target behavior
17. Efficiency as a First-Class Variable
A more accurate system may still be inferior if its computational cost is disproportionate. Each analytical channel should therefore be scored for useful information gained relative to cost.
E_i = ΔI_i / Cost_i
The system can route easy tasks to inexpensive detectors and invoke deeper relational analysis only when the expected information gain justifies it.
This adaptive routing principle is essential if the architecture is ultimately extended to very large text, music, video, code, and multimodal archives.
18. Toward Music and Video
The present benchmark is linguistic, but its architecture is intentionally modality-neutral. A shard is better understood as a bounded relational information unit at a selected scale, not inherently as a piece of prose.
signal → primitive units → sequences → structures → relational units → higher-order structures
In music, candidate units may include samples, spectral events, notes, intervals, chords, rhythms, motifs, phrases, sections, and compositions. In video, candidate units may include pixels, frame features, objects, motion, shots, transitions, scenes, sequences, and narrative structures.
All eventually become encoded digital information, but the most useful analytical scale will depend on the task. Binary representation is universal at the storage layer; meaningful relational boundaries emerge at higher levels.
19. Falsification Criteria
The hybrid framework should be considered unsuccessful or unnecessarily complex if simpler baselines equal or exceed it on accuracy, utility, cost, provenance discrimination, or error localization.
If character analysis adds no useful detection beyond word analysis, remove or down-weight it.
If relational metadata fails to improve retrieval or reconstruction, simplify the shard schema.
If provenance edges do not reduce false attribution, revise the provenance model.
If hybrid routing costs more than it saves, use a simpler fixed strategy.
If TSTOEAO-derived categories do not predict failure better than conventional taxonomies, they should not be privileged.
20. TSTOEAO Relation
The benchmark is strongly aligned with TSTOEAO because it operationalizes states, boundaries, pathways, transformations, costs, corrections, targets, and residuals. However, this alignment is not itself evidence.
Mappability ≠ Validity
The specific empirical question is whether a relational decomposition predicts or corrects outcomes better than competing approaches.
21. Proposed Experimental Sequence
Phase I: establish character-, word-, punctuation-, and shard-level baselines.
Phase II: de-overlap and calibrate fingerprints against background frequency.
Phase III: compare lexical, semantic, relational, and hybrid retrieval.
Phase IV: introduce controlled order, boundary, scope, provenance, and hierarchy perturbations.
Phase V: measure minimum corrective interventions in agent and code tasks.
Phase VI: test provenance inheritance versus independent convergence.
Phase VII: extend the representation hierarchy to music and video.
Phase VIII: compare all relational methods against simpler baselines and publish negative as well as positive results.
22. Conclusion
The central empirical lesson of the preliminary work is that no single representational scale should be assumed sufficient. Long word sequences provide sparse, interpretable fingerprints. Character sequences provide much greater sensitivity but require clustering and statistical normalization. Shard and provenance representations capture relationships that surface similarity cannot distinguish.
The appropriate architecture is therefore hybrid and adaptive. It should use character, word, structural, shard, and provenance evidence in whatever combination produces the greatest reliable value for the task.
The best representation is the one that preserves the relation required by the problem.
The purpose of Paper 208 is not to declare the relational architecture validated. It is to establish a benchmark through which validation or rejection can occur. The next work is straightforward in principle: implement the detectors, record the costs, run controlled perturbations, compare against baselines, and allow the measurements to change the architecture.
If the relational framework survives those tests, Papers 200–207 become more than a conceptual series. They become the specification for an experimentally supported computational system.
References
Broder, A. Z. (1997). On the resemblance and containment of documents. Compression and Complexity of Sequences.
Jurafsky, D., & Martin, J. H. Speech and Language Processing. Stanford University.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27.
Swygert, J. (2026). AInunnaki. Ivory Tower Publishing.
Swygert, J. (2026). 200 From Language to Computation: Linguistics, Punctuation, Mathematics, and Programming as a Unified Relational Architecture in Large Language Models. Ivory Tower Publishing.
Swygert, J. (2026). 201 The Relational Agent: How Symbolic Structure Shapes LLM Reasoning, Analysis, and Action. Ivory Tower Publishing.
Swygert, J. (2026). 202 Relational Debugging: A Unified Architecture for Diagnosing Failure in Code and Computational Systems. Ivory Tower Publishing.
Swygert, J. (2026). 203 The Shard as a Relational Unit: Deconstruction, Reconstruction, and Knowledge Architecture. Ivory Tower Publishing.
Swygert, J. (2026). 204 The Statistical Shard Library: Measuring Retrieval, Reuse, Frequency, and Relational Importance. Ivory Tower Publishing.
Swygert, J. (2026). 205 Symbolic Fingerprints: Provenance, Reuse Detection, and Plagiarism Prevention through Relational Analysis. Ivory Tower Publishing.
Swygert, J. (2026). 206 Relational Reconstruction: Building Novel Outputs from Provenance-Aware Knowledge Shards. Ivory Tower Publishing.
Swygert, J. (2026). 207 The Relational Intelligence Engine: From Language Structure to Adaptive Machine Reasoning. Ivory Tower Publishing.
Vaswani, A., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30.
No comments:
Post a Comment