Copies/docs/orientation_expanded_benchmark.md

Expanded orientation experiments — 2026-11-04

Historical experiment. Start with the orientation decision record for the current recommendation, rejected approaches and links to later validation.

Artifacts, models, scripts and raw scores: tmp/orientation-expanded/.

The main improvement is substantially more labelled-by-construction training data and a richer spatial representation. The useful heatmap records both absolute page layout and layout after cropping around the writing; a second dark-ink threshold exposes fainter strokes. A learned combination of heatmap scores and the pretrained orientation model's full output distribution is stronger than the previous maximum-of-two-scores rule.

Data and evaluation

Added Interro05's 406 processed pages and Interro06's 390 processed pages (796 total). Their existing CropBoxes were reset to the full MediaBox in memory before rendering at 75 dpi, preserving the stored PDF rotations. The files themselves were not modified. This recovers blank margins needed for layout analysis. Interro07 uses the same 347 saved automatic-split pages as previous experiments. Every page supplies an original and a synthetic 180° rotation: 2,286 examples total.

Primary experiment: fit models on Interro05, select thresholds using Interro06, evaluate Interro07. Thus Interro07 is absent from fitting and threshold calibration in this round. It is not an untouched final benchmark: earlier experiments and model-family choices already used Interro07. The assessments also come from the same cohort, so this is not an unseen-writer evaluation.

For fusion training, the heatmap classifier produces five-fold out-of-copy predictions within Interro05. The fusion model trains on those held-out heatmap scores and the pretrained neural outputs. The heatmap classifier is then refitted to all Interro05 pages. This avoids training the fusion on unrealistically perfect in-sample heatmap scores. A separate full assessment calibrates thresholds.

The original images are presumed upright. The prior 64-page sample and 24 sparse examples were visually reviewed previously. A new 24-page random sample from each added assessment was visually checked; all 17 flags of the earlier fusion candidate were also reviewed and appear upright. This includes the final fusion candidate's ten separate-image flags. Every page in the expanded corpus has not been independently annotated, so full-population counts retain this ground-truth limitation. Formula-only pages can be intrinsically ambiguous.

Separate-image results and runtime

Times below are measured detection time for all 347 Interro07 previews, excluding image loading, PDF rendering, model loading and one-time training. The complete feature extraction and per-page prediction calls are included. Image-file loading added 2.77 s in this standalone benchmark; integration reuses the existing splitter previews. The previous fixed image-rule time is represented by the neural feature cost; it has no material learned-head cost.

Method	Calibration false-alarm target	Rotations detected /347	Upright flags /347	Sparse detected /24	Prior sample detected /64	Detection time /347 pages
Previous fixed image score	fixed .52 score	313	16 (4.6%)	24	61	8.16 s
Global heatmaps	4%	325	12 (3.5%)	20	55	2.28 s
Global + writing-crop heatmaps	4%	343	16 (4.6%)	22	61	2.85 s
Global + writing-crop + faint-ink heatmaps	4%	343	8 (2.3%)	23	62	4.14 s
Learned combination of neural outputs	2%	328	15 (4.3%)	24	64	8.16 s
Rich heatmap + image outputs, tree fusion	2%	346	10 (2.9%)	24	64	13.51 s

The new tree fusion detects 346/347 synthetic rotations (99.7%) with ten upright flags (2.9%). Its remaining separate-image miss is Interro07 Copie05, logical page 1. The rich heatmap alone detects 343/347 with eight flags; it is the inexpensive alternative.

Both richer heatmaps and the fusion recover the three misses of the original classifier on separate images. This is stronger evidence than the earlier 64-page experiment, although most of the larger population still relies on presumed orientation labels.

The second ink threshold is 140 rather than 100 on the minimum RGB channel. Every map uses normalized square-root ink density on 8×12 and 16×24 grids. The rich vector concatenates the two grids for the full dark-ink page, the dark-ink writing crop, and the full page at threshold 140. It has 1,440 features. Long-line grid removal was not used.

The heatmap classifier is a standardized RBF SVM (C=3). Neural features retain all four orientation probabilities for full-page and writing-crop views, each supplied and rotated: sixteen numbers. Fusion appends one out-of-fold heatmap score to those sixteen outputs. The final classifier is ExtraTrees with 200 trees, minimum leaf size 5, random seed 514. It predicts a screening score, not a calibrated probability. The primary fusion threshold is 0.2769861111, selected from the upright Interro06 score tail for a 2% calibration target; the rich heatmap threshold is -0.0486107703, from the 4% calibration target.

A learned logistic head over neural outputs alone also improves over the previous arbitrary maximum rule, at essentially the same inference cost. The complete exploratory threshold curves and other rejected candidates are in results.json and multi-fusion-results.json; no unsuccessful models were removed from the raw logs.

Full splitting trial

The experimental wrapper ran the standard splitter on all 47 original PDFs / 214 scanned pages, adding both the rich heatmap and fused scores to each retained output. It also scored a simulated 180° version of every retained page without repeating neural inference: the four neural view outputs are reused in reversed order. Thus the full timing includes orientation auditing for both directions. No LLM calls are included.

Full Interro07 run	Rotations detected /347	Upright flags /347	Total runtime
Earlier splitting baseline	—	—	36.55 s
Earlier fixed image screen	—	15	39.62 s
New combined trial, rich heatmap scores	343	8	48.58 s (shared trial)
New combined trial, fused scores	345	11	48.58 s (shared trial)

These are individual timing runs, not statistical estimates. The two new rows come from one run that calculates both screens; no standalone rich-heatmap-only full-pipeline timing was made in this round. The earlier rows come from preceding experiments, so the time difference should not be treated as a precise isolated overhead measurement.

In the actual splitter previews, the fused screen catches 345/347 rotations with 11 upright flags (3.2%); it catches 63/64 of the prior sample and all 24 sparse rotations. The two misses are Copie01-p03 and Copie05-p01. Separate-image rendering detects 346/347 with ten flags. Small raster phase/crop differences change borderline outcomes. The rich heatmap is stable at 343 detected and eight upright flags across the two rendering paths.

All 420 keep/remove decisions and all output rectangle coordinates match the prior baseline exactly. All 47 original source hashes still match; 347 logical pages remain, with 73 blank halves removed. Scores were recorded only in isolated experimental outputs. Application code and evaluation PDFs were not changed.

A simple union of rich-heatmap and fusion flags on live previews catches all 347 rotations, but gives 18 upright flags (5.19%), just above the requested 5% volume. A stricter union using the heatmap's 2% calibration threshold catches 346 with 15 flags (4.32%). These post-hoc unions are diagnostic comparisons, not independently selected deployment thresholds. A grid search calibrated entirely on Interro06 did not improve on the fusion screen at useful held-out referral levels (union-results.json); it selected effectively fusion-only rules.

Rotating the assessment roles

The richer fusion was additionally tested with each assessment excluded from fitting and threshold selection in turn. Interro07 calibration was used for the Interro05 and Interro06 test runs. The following table uses the same 2% calibration-target policy. Interro05/06 detection times are estimates from the measured Interro07 per-page cost; full-pipeline runtimes were not measured for those two assessments.

Test assessment	Training assessment	Threshold assessment	Rotations caught	Upright flags	Detection time
Interro05	Interro06	Interro07	402/406	5/406 (1.2%)	≈15.8 s (estimate)
Interro06	Interro05	Interro07	389/390	4/390 (1.0%)	≈15.2 s (estimate)
Interro07	Interro05	Interro06	346/347	10/347 (2.9%)	13.51 s (measured)

This supports generalization across these three assessments, but not across unrelated paper layouts, cohorts, scanners or handwriting populations. Rotating examination roles is not the same as an untouched final test after model selection. All training, calibration and test roles and raw scores are retained for audit.

Reproduce and inspect

prepare.py renders the two additional assessments. extract.py creates features.npz and timings.json. evaluate.py fits the initial candidates; evaluate_multi.py fits richer fusion. reverse.py and reverse_multi.py rotate assessment roles. benchmark.py measures complete per-preview screening code paths; this corrected an omitted faint-map resize cost in the initial extraction instrumentation. Use benchmark-timings.json, not early multi_heat timing estimates, for quoted final detection times. integration.py fused_multi_trees runs the full experimental splitter into a fresh output directory. It refuses to overwrite an existing result directory.

Model files are local trusted joblib artifacts; features, logits, thresholds, complete predictions, runtime logs and split reports remain under tmp/orientation-expanded. The representative assessment contact sheets are Interro05-sample.jpg and Interro06-sample.jpg. integrated-scores.json records each live-preview upright and rotated score. The separate-image missed page is Interro07-Copie05-p01-missed.png. Previous false-positive review contact sheets are fused-flags-0.jpg and fused-flags-1.jpg.

Recommendation: the rich heatmap is an attractive fast local screen. The learned fusion gives the strongest measured accuracy while staying below one minute and below 5% upright referrals in the actual Interro07 splitter trial. Use LLM confirmation on flagged pages; keep unresolved cases visible rather than rotating solely on this heuristic. A further assessment with reviewed orientation labels would be the appropriate next validation step.