# Dataset accuracy The instruction records here were extracted from vendor specifications by a language model and then repaired over many passes. `scripts/validate_data.py` checks that the result is internally consistent: no duplicates, syntax tokens that agree with their own mnemonic, cross-references that resolve. None of that is evidence the records are *true*. For a reference work, correctness is the product. So it is measured here rather than asserted, with a reproducible script and a stated method. Run it yourself: ```bash python3 scripts/accuracy_audit.py --sample 300 ``` ## Method A seeded random sample (`SEED = 20260901`) is drawn per architecture and each record is checked against the text of the specification it was extracted from. | Check | Question it answers | Catches | |---|---|---| | `mnemonic` | Does this instruction appear in the spec at all? | invented instructions | | `opcode_quoted` | Does the opcode string appear as the manual prints it? | wrong x86 encodings | | `opcode_derivation` | Does the computed instruction word match the record's own bit pattern? | wrong derived encodings | | `format` | Is the encoding form a real form in this spec? | wrong or invented forms | Proportions are reported with Wilson 95% confidence intervals, which behave at these sample sizes where the normal approximation does not. ## Results Sample of 300 records per architecture, 2026-09-01. Each architecture is sampled with its own generator from the shared seed, so adding an architecture never moves another one's figures. | Architecture | n | mnemonic | opcode | format | |---|---|---|---|---| | x86 | 287 | **97.9%** [95.5, 99.0] | **97.2%** [94.6, 98.6] | **98.6%** [96.5, 99.5] | | ARM | 300 | **96.3%** [93.6, 97.9] | not yet checked | not yet checked | | RISC-V | 300 | **96.3%** [93.6, 97.9] | **100%** [98.3, 100] | **89.7%** [85.7, 92.6] | | PowerISA | 300 | **94.0%** [90.7, 96.2] | not checkable | **98.7%** [96.6, 99.5] | | PTX | 190 | **99.5%** [97.1, 99.9] | n/a, virtual ISA | not yet checked | | AMDGPU | 300 | **100%** [98.7, 100] | not yet checked | not yet checked | x86 opcodes are checked as quoted strings; RISC-V opcodes are checked by derivation (226 of 300 records had a rebuildable bit pattern). PowerISA stores computed opcodes whose bit patterns use decimal field values with form-dependent widths, so neither method applies to it yet. ### PTX and AMDGPU: read these differently These two are checked against NVIDIA's PTX ISA document and LLVM's AMDGPU TableGen, neither vendored here (`GPU_REF_DIR`). Their numbers are the highest in the table and mean the least, because the pipeline is different: unlike the other four, these records were **imported programmatically from the very sources they are checked against**, by `import_ptx.py` and `import_amdgpu.py`. So this measures **fidelity to source**, not independent corroboration. It answers "did the importer drop, mangle or invent anything, and has the data drifted from an upstream that moved" - a real question, and the check that would catch a fabricated instruction. It is not evidence of the same kind as PowerISA at 94.0%, and 100% here is worth less than that 94%. PTX carries no binary encoding at all (`"format": "Virtual (PTX)"`), being a virtual ISA, so the opcode checks do not apply to it in principle rather than for want of effort. ### ARM ARM is checked against Arm's machine-readable architecture specification, the same artifact the ARM records were built from, rather than against the 39-page overview PDF in `database/arm/`. The reference defines 1,670 mnemonics across the A64 and AArch32 profiles. That XML is deliberately **not vendored**: it is roughly 70 MB of Arm's material, and not shipping vendor documents is the whole point of `DATA-LICENSE.md`. Fetch and extract it, then: ```bash ARM_MRA_DIR=/path/to/extracted-arm-mra python3 scripts/accuracy_audit.py ``` Without `ARM_MRA_DIR` the audit skips ARM and says so, rather than quietly reporting on three architectures as though that were the whole corpus. ## What this does not say **Corroboration is not correctness.** Finding a mnemonic in the spec proves the instruction is real, not that its description or pseudocode is right. Judging those requires a human reading both documents, and this audit does not attempt it. Treat every number above as a **lower bound on fidelity**. **Coverage is now every architecture**, 5,986 records, though not on equal terms: see the PTX and AMDGPU note above for why two of them prove less than the others. What is still thin is depth rather than breadth. ARM, PTX and AMDGPU have only the mnemonic check; ARM's encodings live in the XML register diagrams and can be checked exactly as RISC-V's now are, which is the obvious next increment. **The ARM reference is dated 2022-12.** Anything Arm has added since will look uncorroborated without being wrong. `gcspopm` (FEAT_GCS) and `vbfmmla` are exactly that, and they are among the residual ARM misses. **A miss is not automatically an error.** `pypdf` flattens the tables these manuals keep encodings in, so a correct value can be unfindable in extracted text. ## Six corrections made to the method These are recorded because they change how the numbers should be read, and because each one initially looked like a catastrophic data problem. **Computed opcodes are not quoted opcodes.** Searching the manuals for PowerISA and RISC-V opcodes scored 2.0% and 0.5%. The cause was the method, not the data: those architectures store a fully assembled instruction word (`0x04000057`) that no manual ever prints, while x86 stores the opcode as the manual writes it (`FF /0`). Computed words are now checked against the record's own bit pattern instead. **Intel's manual is the wrong reference for AMD instructions.** Nine of the first ten apparent x86 failures were XOP, 3DNow! and TBM instructions, which are AMD extensions that Intel correctly never documented. They are now reported as "no reference available" rather than counted as misses. This is a real limit: roughly a tenth of the x86 set cannot be checked until AMD's manual is vendored. **An A64 specification does not contain AArch32 instructions.** Checking ARM against the A64 profile alone scored 63.7%, and the misses were overwhelmingly AArch32 and T32: `vtbx`, `smulbb`, `qadd16`, `rsb.w`. Adding the AArch32 profile and treating the T32 `.w`/`.n` width hint as the assembler directive it is, rather than as part of the mnemonic, moved the figure to 96.3%. This is the same mistake as the AMD one, made twice, which is why the method now names its reference for every architecture. **Format labels carry editorial glosses.** RISC-V scored 79.3% because this corpus labels some encodings `R-type (atomic)` or `I-type (float)`: the spec's own form plus a clarifying parenthetical of the project's own. That is house style, not error. Matching the base form lifted it into the high 80s and low 90s depending on the sample; the run reported above scores 89.7%. **A capitalised plural is not the only heading NVIDIA writes.** PTX first scored 96.8%, and the misses were `ldmatrix`, `stmatrix` and `movmatrix`, whose headings read "Warp-level matrix load instruction: ldmatrix", singular and lowercase. A case-sensitive match for "Instructions:" lost them. PTX mnemonics are also dotted qualifier chains, and the document sections some variants inside their base's section rather than giving them their own, so `bar.cta` is documented under `bar`. Matching case-insensitively and falling back to the base mnemonic gives 99.5%. **One AMDGPU miss was a prefix missing from a regex.** `cluster_load_b128` looked fabricated until it turned out to be defined in `FLATInstructions.td`; the mnemonic-shape pattern simply had no `cluster_` prefix. Worth recording because broadening a pattern until a failure disappears is exactly how an audit talks itself into a clean result, so the definition was checked in the TableGen source before the prefix was added. ## Confirmed defects, since repaired The derivation check found 18 RISC-V records whose stored opcode contradicted their own documented bit pattern. All 18 were adjudicated by hand against the specification and repaired by `scripts/fix_riscv_encoding_defects.py`; the census now reports **406 of 406 in agreement**. They had failed in both directions: - **`NOT`** stores `0x00004013`, which encodes `xori rd, rs, 0`, a register move. `not rd, rs` is `xori rd, rs, -1`, so the correct template is `0xFFF04013`. The opcode is wrong; the bit pattern is right. - **The `VAMO*` family** (`VAMOADDW.V`, `VAMOANDW.V` and siblings) carries bit patterns with a spurious leading 6-bit field and no `vs2`, so the pattern does not describe a 32-bit RISC-V AMO encoding. Here the stored opcode is right and the pattern is wrong. Two more groups turned up once the family was examined properly. The four `.vv` forms of `VMSGT`/`VMSGE` **do not exist in the specification**: it provides `vmsgt.vx` and `vmsgt.vi` but no vector-vector form, noting "Following two instructions are not provided directly", and the assembler builds them by swapping the operands of `vmslt`/`vmsle`. The strings `vmsgt.vv` and `vmsge.vv` occur nowhere in the specification text. All four records carried the funct6 of `vmsgt.vi/vx`, so all four patterns were identical while their opcodes correctly differed. And `VWMACCUS.VX` described itself as OPIVV with a `vs1` operand and funct3 000, when a `.VX` form takes a scalar register and uses OPMVX, funct3 110, which is what its opcode already said. This check is cheap enough to run over the whole RISC-V set rather than a sample. Before the repair, all 406 records with a rebuildable pattern yielded **18 disagreements (4.4%)**: ``` NOT VMSGE.VV VMSGEU.VV VMSGT.VV VMSGTU.VV VWMACCUS.VX VAMOADDD.V VAMOADDW.V VAMOANDD.V VAMOANDW.V VAMOMAXD.V VAMOMAXW.V VAMOMIND.V VAMOMINW.V VAMOORD.V VAMOORW.V VAMOXORD.V VAMOXORW.V ``` That was a work list rather than a statistic, and it has been worked: `python3 scripts/fix_riscv_encoding_defects.py --check` now reports nothing to do. The repair script refuses to write unless every record it touches agrees with itself afterwards, which caught a real bug in the repair itself: the first version handed the 32-bit `W` variants the 64-bit width code. ## Honest summary Across four architectures and 64% of the corpus, 94% to 98% of sampled records are corroborated by their own source specification on every mechanically testable claim. The known failure modes are concentrated and enumerable rather than diffuse. Four apparent problems this audit surfaced turned out to be flaws in its own method, three of them the same mistake in different clothes: checking records against a document that could not contain them. That is worth stating plainly: the headline risk in work like this is measuring the reference instead of the data. What remains unmeasured is the semantic correctness of prose and pseudocode, ARM encodings, and PTX and AMDGPU entirely. An expert review of a stratified sample is the next step, and this audit is the sampling frame for it.