Copies/docs/orientation_heatmap_benchmark.md

Writing-density orientation experiments — 2026-11-04

Historical experiment. Start with the orientation decision record for the current recommendation, rejected approaches and links to later validation.

Artifacts and runnable scripts: tmp/orientation-heatmaps/. This extends the previous experiment in tmp/orientation-benchmark/.

The density-map classifier recovers all three rotations missed by the previous image classifier. It is complementary rather than a replacement: at the selected thresholds it catches fewer rotations overall. A simple union catches all 64 paired test rotations, but exceeds the desired referral volume.

Accuracy and time

Detection time means measured feature extraction and prediction for all 347 logical pages, sequential CPU, excluding rendering, image-file decoding, training and application startup. The fusion times sum independently measured components and conservatively count their ink extraction twice. They are not measured full-pipeline times.

Method	Detected /64 rotations	False alarms /64 upright	Sparse rotations /24	Flags /347 original pages	Detection time for 347 pages
Coarse heatmap, linear	47	4	16	15	0.97 s
Fine heatmap, linear	40	4	14	14	0.72 s
Two-scale heatmaps, nonlinear	55	3	19	12	1.52 s
Two-scale heatmaps, grid-line cleanup	54	3	19	12	2.01 s
Heatmap + line endpoints, nonlinear	49	7	16	13	2.31 s
Heatmap + line endpoints, trees	54	4	21	15	2.07 s
Previous image classifier	61	1	24	16	11.17 s
Flag if either image or heatmap model flags	64	4	24	28	12.70 s (sum)
Combined rule, recall-oriented calibration	63	3	23	19	12.70 s (sum)
Combined rule, stricter calibration	60	1	22	14	12.70 s (sum)

The original-page flags correspond to 3.46% for the nonlinear heatmap alone, 4.61% for the previous image screen, 8.07% for their union, 5.48% for recall-oriented fusion, and 4.03% for stricter fusion. An intentionally inverted population naturally produces many more flags; these rates describe the presumed-upright originals.

Full splitting runtime

Run	Input	Time including splitting, previews, exports and startup
Previous baseline without orientation	47 PDFs / 214 source scans	36.55 s
Previous image orientation screen	Same batch	39.62 s
New heatmap screen	Same batch	40.80 s

The first heatmap timing was 40.32 s but overlapped a separate neural benchmark, so it was superseded by the isolated 40.80 s run. Timings are individual runs, not confidence intervals; the previous baseline and image-screen times come from the preceding experiment. The new screen has not demonstrated a lower complete splitting time despite its cheaper detection kernel. File decoding during the feature experiment was an additional 2.63 s; separate neural-screen file loading was 2.68 s. Normal integration reuses already rendered previews. No LLM requests were made.

Features and evaluation

A fixed min-channel <100 ink mask excludes 15-pixel image borders. This is cheap dark-ink extraction, not the application's expensive full content analysis. Normalized square-root ink-density grids at 8×12 and 16×24 form the best heatmap representation. Coarse/fine linear models use 8×12 and 24×32 grids. Normalization removes total-ink variation while retaining relative spatial distribution. Grid cleanup removes long horizontal/vertical strokes; it slightly worsened quality. Layout models add 48 horizontal bands describing left/right ink limits, occupied columns and ink density, plus 16×24 histograms of line start/end positions after inexpensive horizontal component grouping.

Models: standardized logistic regression C=.1; standardized RBF SVM C=3, gamma=scale; ExtraTrees with 160 trees, min leaf size 3, max_features=.6. Models are intentionally small. Paired orientation scoring was also tested (results.json): it gave no improvement for heatmap-only models and worsened the layout SVM.

All 347 saved logical pages and their synthetic 180-degree rotations supply training/evaluation features. Five folds separate entire student copies. Inside each outer training fold, a separate 25% of student groups calibrates the threshold; the remaining groups fit the classifier. Outer test students appear in neither fitting nor threshold calibration. The primary 64-page test subset is the previously visually reviewed sample, containing the 24 largest vertical whitespace gaps plus 40 random pages; each tested both ways. Thus every primary reported heatmap prediction is from a model that has never trained on that student.

Calibration for individual models targets at most about 4% upright flags on the inner calibration subset (finite-sample rounding applies). Fusion tries a small fixed grid of neural thresholds and heatmap calibration quantiles, selecting maximum calibration recall under a 4% or 2% upright-flag constraint. Calibration controls the observed inner subset only; it does not guarantee a held-out rate. fusion-target4.json and fusion.json record the fold-specific selected thresholds. In one fold the recall-oriented fusion selected heatmap-only screening.

Quality results are out-of-fold. The separately saved timing-model.joblib is fit on all pages solely to measure integration speed; none of its in-sample predictions contributes to the quality results. Training a heatmap model takes a fraction of a second; all five SVM fold fits together took 0.27 s.

On the larger, provisionally labelled 347-page population, the nonlinear heatmap catches 299/347 rotations; the previous image screen 313/347; union 342/347; recall-oriented fusion 338/347; stricter fusion 319/347. These figures rely on the original orientation assumption. The 64 selected pages and previous model flags were visually checked in the preceding experiment, but every other page was not independently annotated.

Interpretation and limitations

Heatmaps carry orientation information even when letters are unreadable. They recover Copie01-p03, Copie03-p07, and Copie36-p08, all three previous misses. They also miss other sparse pages: their 19/24 sparse recall is below the prior image screen's 24/24. No tested rule simultaneously improves the prior primary-sample recall and stays below 5% referrals over the entire original corpus: recall-oriented fusion is close at 5.48%, while the stricter variant has lower sample recall.

These are exploratory comparisons on one evaluation, with relatively few students per inner calibration set and many tried feature variants. Student grouping prevents direct student leakage, but choosing a preferred method after looking at these results still needs validation on another evaluation. Training/evaluation on the same assessment can learn its paper/layout conventions; there is no evidence yet for generalization to other assessments. The original synthetic pairing makes layout reversal systematic; real scanner artifacts and naturally inverted scans need later validation.

Artifacts and reproduction

experiment.py extracts all features and runs six primary models; paired.py evaluates symmetric scoring; compare_neural.py recomputes full neural scores and complete preprocessing/prediction runtime; fusion.py calibrates combined rules (currently target 2%); features.npz holds fixed feature vectors; results.json holds every out-of-fold score, fold assignment, flag, threshold and timing. fusion-target4.json retains the 4% fusion experiment. visualize.py creates the heatmap comparison. integration.py heatmap-isolated reproduces the full splitter timing into a new directory (existing-output refusal applies). The model is loaded with joblib; only load trusted locally generated models.

No application source was changed. Full-run boundaries and all keep/remove decisions were verified against the prior splitter reference; all 47 source hashes still match.