sebastien/Copies
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Detect and confirm page orientation during automatic splitting

Browse Source 
Bundle the trained heatmap/image orientation model and run it after provisional
blank-page and split analysis. Confirm candidates with Gemini using low thinking;
resolve mixed A3 sheet ordering with the complete statement and medium thinking.
Keep A4 scans independent and export paired A3 scans in booklet reading order.

Reanalyse uniformly inverted source sheets, validate ordering permutations, and
preserve source provenance, corrected rotations and reading slots in manual review.
Document the experiments, rejected approaches, calibration and runtime limits.

Validation: focused splitting/orientation regression suites pass; bundled model
checksum and wheel assets verified. Full Interro07 offline run: 44.26 seconds,
eight candidates, mocked upright confirmations, source hashes and geometry intact.
This commit is contained in:
sebastien committed 2026-11-04 21:07:20 +01:00
1 parent 78f3e4b2f0
commit 0a4f4a80d0
21 files changed
+1574 -51
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/page_orientation.md
+283
View File
@@ -0,0 +1,283 @@
# Detecting upside-down handwritten pages
Decision record, 2026-11-04. The recommended experiment combines writing-density
heatmaps with a small pretrained image-orientation classifier, then sends flagged
pages for LLM confirmation. The trained bundle defaults to a conservative
threshold. It is now enabled in the automatic evaluation splitting step; see
[the integration rules](auto_page_split.md#orientation-and-physical-sheets).
The experiment results below predate live confirmation integration and exclude
Gemini latency.
## Requirements and result
The target is to process all of Interro07 in less than a minute, including normal
splitting. Recognition of the handwriting is unnecessary. Some missed rotations
are acceptable; false alarms can be checked individually, provided referrals stay
around or below 5% on an ordinarily upright batch. That percentage is a workload
target, not a cap to enforce by silently dropping low-ranked candidates. A batch
with many genuinely inverted pages should produce more referrals.
The final experimental splitting run processed 47 original PDFs / 214 scans,
retaining 347 logical pages and removing 73 blank halves, in **50.92 seconds**.
The conservative screen flagged eight upright logical pages (2.3%). Its simulated
inversion audit detected 345/347 rotations, including all 24 sparse examples and
all 64 pages from the initial test sample. Output rectangles, keep/remove decisions
and original source hashes matched the baseline.
Direct checks of the reusable predictor on actual rotated RGB arrays give:
| Profile | Assessment | Rotations detected | Upright false alarms | Screening time for that assessment |
|---|---|---:|---:|---:|
| Conservative, recommended | Interro04 | 372/373 | 11/373 (2.9%) | 14.91 s |
| Conservative, recommended | Interro07 | 345/347 | 6/347 (1.7%) | 13.48 s |
| Balanced | Interro04 | 373/373 | 19/373 (5.1%) | 14.91 s |
| Balanced | Interro07 | 346/347 | 16/347 (4.6%) | 13.48 s |
Screening times average one upright and one rotated pass, excluding file loading,
PDF rendering and model startup. Both profiles use the same weights and computation.
The 50.92-second full-run timing includes splitting, previews, exports, startup
and an inexpensive inverted-feature audit, but excludes LLM confirmation. It was
measured once on a Ryzen 7 PRO 7840U with four copy workers and one ONNX inference
thread per worker; it is not a latency guarantee. Separate-image and splitter
preview counts differ because their raster alignment and clip boundaries differ.
## Recommended algorithm
Operate on each retained logical A4 page independently, reusing the splitter's
existing **75 dpi RGB overview**. A complete A3 spread makes writing too small and
can mix orientation evidence from its two halves. Preserve the source-page
grouping for later confirmation and repair.
1. Respect the PDF's displayed rotation and retain the full logical page area.
Existing margin crops would remove useful layout evidence. During dataset
preparation, full MediaBoxes were restored in memory without saving changes
to the PDFs.
2. Extract a cheap ink mask: a pixel counts as ink when its minimum RGB channel
is below 100. Exclude the outer 15 pixels on each edge. This is not the full
content/blank detector; do not use this orientation mask to discard pages.
3. Define a writing crop using the 0.5th and 99.5th percentiles of ink coordinates,
with 12 pixels of padding. The implementation floors both percentile endpoints
and uses an exclusive upper bound of `endpoint + 13`. If fewer than five ink
pixels exist, use the full page. See the rounding caveat below.
4. Build density grids at **8×12 and 16×24** for three views: the full threshold-100
mask, its writing crop, and the full-page mask at threshold 140 for faint ink.
Area-resize each mask, take the square root of the densities, and divide each
grid by its L2 norm plus `1e-6`. Concatenate the six grids: **1,440 features**.
5. A standardized RBF SVM (`C=3`, `gamma="scale"`) converts those features into
one layout score. Global maps preserve margins and blank space; crop-relative
maps preserve line/block arrangement when writing occupies a small area.
6. Run **PP-LCNet_x1_0_doc_ori** on the RGB full page and writing crop, each in its
supplied orientation and rotated 180°. Resize each view directly to 224×224;
normalize RGB channels with means `[.485, .456, .406]` and standard deviations
`[.229, .224, .225]`. Retain all four orientation probabilities from each view:
**16 neural features**. No text recognition is performed.
7. Append the SVM score to those 16 features. An ExtraTrees classifier with
200 trees, minimum leaf size 5, `max_features=1.0`, and seed 514 predicts the
final screening score. Use `score > threshold`, not `>=`.
8. The conservative threshold is **0.3044920634920634**; the balanced threshold is
**0.17323015873015868**. These values belong to the saved weights and feature
pipeline. Retraining or changing preprocessing requires recalibration.
The final score is not a calibrated probability. Confirm flagged pages before
rotating them. Formula-only, blank and unreadable pages can remain unresolved.
Do not let an uncertain orientation score change blank-page decisions.
An upside-down A3 source reverses its two halves as well as their orientation.
Retain provenance and use the existing orientation-repair logic when applying
confirmed corrections; rotating each exported half alone can leave their order
wrong. The automatic splitter now applies the confirmed-sheet rules before export.
## Training and validation
All seven original interros contributed to preparation: **2,762 logical pages**,
each with a synthetic 180° counterpart. Duplicate Interro03 pilot folders were
excluded.
- Fit on Interro01 (392), Interro02 (423), Interro03 (431), and Interro05 (406):
**1,652 original pages**.
- Set numeric thresholds using Interro06: **390 pages**.
- Reserve Interro04: **373 pages**. Check Interro07 separately: **347 pages**.
To train the fusion, fit the heatmap SVM on three training assessments and score
the fourth, repeating for all four. Train ExtraTrees using these held-out-assessment
SVM scores and the pretrained neural features. Finally refit the SVM on all four
training assessments. In-sample SVM scores would give the fusion an unrealistically
easy training problem. Complete learned-model fitting took **20.76 seconds**;
training is not repeated during splitting.
The balanced weights and threshold were frozen before predicting Interro04.
The conservative fallback was evaluated after the balanced profile slightly
exceeded the holdout referral budget. Its numeric threshold still comes solely
from Interro06, but its policy selection is a follow-up experiment, not a second
untouched holdout validation. Interro07 had already informed earlier architecture
choices. These assessments share a cohort, so the result does not establish
unseen-writer generalization.
Originals are presumed upright from the corrected workflow. Representative
samples, the initial 64-page set, its 24 sparse examples, and all nineteen balanced
Interro04 false alarms were visually reviewed. The entire corpus was not
independently labelled. Synthetic rotations do not cover every real scanner fault.
## What was tried and why it was not selected
These stages used different training populations and thresholds. Their figures
are diagnostic records, not one common leaderboard. Historical rotated-feature
results are also subject to the crop-rounding caveat below.
### OCR, whitespace and pretrained classifiers
- **Tesseract OSD:** caught only 9/64 rotations and flagged 23/64 upright pages.
It returned no orientation for 29 upright and 28 rotated examples. Four workers
processed all 128 calls in **6.07 seconds**. It was fast enough but poorly matched
to sparse handwritten mathematics; unknown outputs must count as misses.
- **Top/bottom whitespace alone:** caught 49/64 rotations, but flagged 16/64 upright
pages and 139/347 corpus originals. Its full-corpus computation took **3.91 s**.
A single margin-asymmetry number discards line layout and mistakes legitimate
bottom-heavy writing for inverted pages.
- **Pretrained whole-page orientation:** the official short-side resize/centre-crop
preprocessing caught 51/64 rotations with four false alarms; resizing the entire
page to a square caught 50 with one. Their early measured wrapper times were
**1.43 s** and **1.35 s** for 347 pages, excluding shared file loading. Centre
cropping can hide sparse writing. These early partial timings must not be
compared directly with the final complete screening times.
- **Small text-line classifier:** voting over up to five cheaply extracted writing
components caught 30/64 rotations with seven false alarms; extraction and
classification took **5.69 s** for 347 originals. Equation fragments and ruling
make these imperfect text-line crops. This rejects that cheap baseline, not all
possible line classifiers.
- **Fixed whole-page/crop combination:** averaging `P180(image)` with
`P0(rotated image)` for each view, then taking the larger view score, caught
61/64 rotations with one false alarm. A complete later screen took **11.17 s**
for 347 pages. It recovered all sparse examples but missed three formula-heavy
rotations. Learning from all 16 neural outputs and a layout score worked better
than this hand-chosen maximum rule.
See [the first benchmark](orientation_benchmark.md) for all early variants.
### Initial heatmap experiments
Five-fold out-of-copy evaluation within Interro07 used the original 64-page sample.
Each time below includes feature extraction and prediction for 347 pages, excluding
file loading, rendering, training and startup.
| Variant | Rotations caught /64 | Upright false alarms /64 | Sparse rotations /24 | Screening time /347 pages |
|---|---:|---:|---:|---:|
| Coarse heatmap, linear classifier | 47 | 4 | 16 | 0.97 s |
| Finer heatmap, linear classifier | 40 | 4 | 14 | 0.72 s |
| Two scales, nonlinear SVM | 55 | 3 | 19 | 1.52 s |
| Two scales plus long-grid-line removal | 54 | 3 | 19 | 2.01 s |
| Heatmap plus line endpoints, nonlinear SVM | 49 | 7 | 16 | 2.31 s |
| Heatmap plus line endpoints, tree classifier | 54 | 4 | 21 | 2.07 s |
Finer grids alone did not compensate for a weak classifier. Removing long ruling
strokes slightly hurt quality and cost more time. Explicit line starts/ends did
not improve on the simpler two-scale SVM. Symmetric scoring of both orientations
gave no gain for heatmap-only models and worsened the line-layout SVM.
Heatmaps recovered all three original image-classifier misses, establishing useful
complementarity. However, a simple OR of both screens flagged **28/347 originals
(8.1%)** despite catching all 64 sample rotations. Its summed measured component
time was **12.70 s**. Inner-calibrated OR rules either retained too many referrals
or lost recall; a grid search on Interro06 later selected effectively fusion-only
rules. Catching every sample rotation was not sufficient to select a method.
See [the heatmap benchmark](orientation_heatmap_benchmark.md) for those folds and
threshold experiments.
### Adding other assessments and richer features
Training on Interro05 and calibrating on Interro06 gave the following Interro07
results. Times are complete measured per-page screening loops for 347 pages,
excluding file loading, rendering and startup.
| Variant | Rotations caught /347 | Upright flags /347 | Screening time /347 pages |
|---|---:|---:|---:|
| Global heatmaps | 325 | 12 | 2.28 s |
| Global plus writing-crop heatmaps | 343 | 16 | 2.85 s |
| Global, crop-relative and faint-ink heatmaps | 343 | 8 | 4.14 s |
| Learned neural-output combination alone | 328 | 15 | 8.16 s |
| Rich heatmap plus neural outputs, tree fusion | 346 | 10 | 13.51 s |
The crop-relative maps and second ink threshold were retained. Pure heatmaps were
an attractive inexpensive alternative, but neural evidence recovered additional
cases. Logistic and SVM fusion candidates did not offer a better overall
recall/referral tradeoff than the selected tree fusion at the tested calibration
policies. Their complete threshold curves remain in the raw experiment results.
That narrower tree-fusion model produced **345 detections and 11 upright flags**
in a **48.58 s** full splitting trial. Adding the richer heatmap flags by simple
OR caught all 347 simulated rotations but produced **18 flags (5.19%)**, just above
the workload target. Such post-hoc unions were not independently validated.
More training data improved the held-out-assessment result, but did not uniformly
reduce every error rate. The final balanced model has more Interro07 referrals
than this narrower model. Prefer the final conservative profile for the measured
workload, rather than assuming that more training always wins at a fixed threshold.
See [the expanded benchmark](orientation_expanded_benchmark.md) and
[the final training record](orientation_final_training.md) for the complete evidence.
## Measurement lessons and reproducibility
- **Actually rotate and recrop test images.** Flooring percentile bounds makes
crop creation non-equivariant by a pixel under 180° rotation. Swapping cached
views therefore does not exactly reproduce a fresh detector call on a rotated
page. The final direct-image results above supersede that approximation for
rotation accuracy; the full splitter audit remains explicitly labelled simulated.
- **Do not mix render paths silently.** Rendering an exported clipped PDF and
slicing the source overview can shift pixels. Measure the integrated path too.
- **Separate timing scopes.** Model-only inference, full screening, image decoding,
training and full splitting are different measurements. An initially omitted
faint-map resize cost was corrected in the expanded standalone benchmark.
Interrupted runs and a run overlapping another CPU benchmark were excluded from
the selected timings. Faster classifier kernels did not always produce faster
complete splitting runs.
- **Preserve full-page geometry.** Training on already cropped content would hide
the margins and blank vertical spaces the heatmap is intended to analyse.
- **Keep assessment/copy groups together.** A page and its rotated counterpart
must never straddle training and test sets. Use held-out scores when stacking
classifiers. Report which data selected thresholds and which selected policies.
- **Confirm before repair.** Keep source-page provenance, A3 half ordering and
unresolved verdicts intact; orientation screening is not proof of a safe repair.
The detailed records above are repository documentation. Large generated datasets,
models, galleries and most experiment scripts are under ignored `tmp/` and are
local artifacts, not assets guaranteed by a fresh checkout:
- `tmp/orientation-benchmark/`: initial sample, actual randomly rotated PDF copies,
paired scores, Tesseract results and reference full splitting runs.
- `tmp/orientation-heatmaps/`: first heatmap features, grouped folds and rejected
layout/fusion variants.
- `tmp/orientation-expanded/`: Interro05/06 features, cross-assessment candidates,
live-preview scores and timing corrections.
- `tmp/orientation-final/`: corpus, frozen design/weights, profiles, direct rotated
predictions, package checks, final full-run report and source verification.
The final folder's `prepare.py`, `extract.py`, `train.py`, `profiles.py`,
`benchmark.py`, `direct_rotations.py` and `integration.py` document the executable
workflow. They rely on earlier experiment artifacts and local assessment files;
they are not a clean-checkout training command. Existing-output refusal applies to
the full splitting experiments. `frozen-model.json` and `bundle/sha256.json`
identify the saved weights; `profiles.json` records the conservative follow-up.
The standalone bundle contains `predict.py`, `model.joblib`, `doc.onnx`,
`settings.json`, dependencies and its validation record. Instantiate one `Detector`
per worker and reuse it; pass full logical-page RGB `uint8` images at 75 dpi.
The returned fields are `flag`, `score`, `threshold` and `heatmap_score`. The
pretrained ONNX model is PaddlePaddle's
[PP-LCNet_x1_0_doc_ori](https://huggingface.co/PaddlePaddle/PP-LCNet_x1_0_doc_ori_onnx),
Apache-2.0. The recorded runtime uses scikit-learn 1.9.1 and ONNX Runtime 1.31.1.
## Integration into the maintained workflow
The runtime is maintained in `copienator/page_orientation_detector.py`, with assets
under `copienator/data/page_orientation/` and declared wheel dependencies. It no
longer depends on the local `tmp/` experiment folders. `copienator/split_orientation.py`
handles Gemini verdicts, physical-sheet grouping and validated ordering. The GUI
and `page-split --automatic` enable this after provisional geometry/blank analysis.
The standalone geometry tool enables it through `--orientation --statement ...`.
See [automatic splitting](auto_page_split.md#orientation-and-physical-sheets) for
A4/A3 rules, failure handling, provenance and manual-review behaviour. Unit tests
use mocked Gemini responses; local full-batch benchmarks can do the same. Such
benchmarks establish local computation time, not live API latency or LLM accuracy.