Copies/docs/auto_page_split.md

# Experimental automatic page splitting

## Python GUI

The “Séparer et réordonner les pages” step now offers **Manuellement** and
**Automatiquement**. Automatic processing runs in the background using the
standard splitter, publishes PDFs in `Copies`, and preserves source scans in
`Copies Originales`. A4 scans keep their positions; consecutive A3 scans are paired
and exported in booklet reading order (outer right, inner left, inner right,
outer left). The parallel-copy setting
appears only for automatic processing. Existing single-copy and marked-copy
selection work in both modes.

Choose Manuellement afterward to review the saved proposals against the
preserved original. The editor displays independent overlapping output crops
and crosses out removed blank pages. Advancing accepts the displayed decision;
the existing keep/discard shortcuts override it. Moving a splitter resumes
manual splitting on that page. Reviewing automatic proposals preserves their
saved reading order, source rotations and individual-half corrections.
Proposals are reused only while both source and output fingerprints still match.

Each automatic run saves its inputs, report/gallery, and previous outputs under
`.copienator/runs/page-split-*`. All PDFs are analysed before publication, and
PDFs and provenance are published together with rollback on installation errors.
An entirely blank copy keeps its preserved source and receives a manual-review
flag instead of an invalid empty output PDF.

The equivalent CLI command is:

```sh
python -m copienator page-split EVALUATION --automatic --workers 4
```

Omit `--automatic` for the interactive editor. The standalone review command
below remains available for experiments without installing results into `Copies`.

Run from the repository root, on original scans rather than already split copies:

```sh
python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/split-review --workers 4
```

The output directory must not already exist. Input PDFs are never overwritten.
This command is deliberately separate from the GUI and from the interactive
`page-split` command. It does not reorder booklets or use secondary splitters.
Outputs follow source-page order, left then right. They must not replace the
current processed evaluation without review and the appropriate later ordering.

## Outputs

- `index.html`: source-page previews, proposed boundaries, blank decisions,
  uncertainty reasons, and links to exported PDFs.
- `pdfs/`: clipped PDFs retaining source scan data, without rasterizing output.
- `report.json`: decisions, original file hashes, original rotations/CropBoxes,
  source page numbers, displayed clip coordinates, and output page numbers.
- `review/`: JPEG previews. Green marks the left output, purple the right/whole
  output, orange the detected centre groups; red shading marks a blank removal.

A wholly blank copy has no output PDF, but still has report entries. An interrupted
run may leave partial outputs; use a new output directory for a subsequent run.

## Decisions

### Orientation and physical sheets

After provisional blank/middle detection, each nonblank logical page is screened
using the bundled conservative orientation model. Model weights and dependencies
are included in the installed package; no runtime model download is required.
Gemini confirms only candidates, returning `upright`, `inverted` or `inconclusive`.
An inconclusive reply is treated as upright for the remainder of the run.
`MODEL_FOR_PAGE_ORIENTATION` can select the confirmation model; its default `None`
inherits `MODEL_FOR_LABEL_ID`. Orientation confirmations use low thinking;
page-ordering requests use medium thinking. Both use structured responses.

A4 scans are independent, even when adjacent: correct a confirmed inversion and
keep its position. A3 scans are paired only when consecutive; an unpaired A3 scan
is flagged for manual review. Once an A3 inversion is confirmed, Gemini checks
all other nonblank pages on that sheet, including pages the local model did not
flag. Blank logical slots remain available until sheet ordering is resolved.

- If every logical page is blank or confirmed inverted, turn both source faces
  180°, recompute blank/middle/overlap decisions, then apply normal booklet order.
- Otherwise turn only the inverted logical pages. Send the full evaluation
  `enonce.pdf` (including supported filename aliases) and all sheet images,
  including blanks, to Gemini for a reading-order permutation. Validate that every
  slot occurs exactly once, then remove blank pages from the final export.

A request failure, malformed orientation reply, invalid permutation or missing
énoncé for mixed-sheet ordering aborts analysis before publication. These errors
are not interpreted as inconclusive handwriting. Existing copies and originals
remain untouched. The report stores screen scores, replies, source/logical
rotations, physical-sheet grouping and final page numbers. Manual acceptance
preserves that order; reinstating a blank proposal restores its reading-order slot.

The standalone analysis tool remains a geometry-only baseline by default. To
exercise orientation and booklet ordering without publishing into `Copies`:

```sh
python -m copienator.auto_page_split 'EVALUATION/Copies Originales' tmp/oriented-review \
  --workers 4 --orientation --statement EVALUATION/enonce.pdf
```

See [the decision record](page_orientation.md) for the trained model, rejected
approaches and calibration limits. Its earlier timings exclude live Gemini latency.

### Integration verification (2026-11-04)

The maintained implementation passes 65 focused tests, including A4 independence,
A3 booklet order, blank siblings, inconclusive replies, full-source reanalysis,
mixed-sheet image rotation and full-statement ordering payloads, invalid response
rejection, failure preservation, and manual acceptance/reinstatement/secondary cuts.
Wheel inspection confirms that weights, settings, ONNX model and license ship with
the package.

An isolated full Interro07 run took **44.26 seconds** with four workers: 47 PDFs,
214 scans, 347 kept pages, 73 blank halves removed, eight detector candidates.
Gemini replies were mocked as upright; no live requests were sent. Source hashes,
blank decisions and boundaries match the reference; output page mappings match
the new reading order. This checks local computation and request routing, not
live Gemini accuracy or latency. The evidence is under
`tmp/orientation-implemented-interro07-verified/` (`timing.json`, `report.json`,
`verification.json`).


A subsequent one-off live check of precisely those eight candidates used
`gemini-3.8-flash`, thinking `low`, four concurrent workers and no retries.
Gemini classified **all eight as upright**, with no inconclusive or inverted
verdicts. Request wall time was **9.64 s**, with individual calls taking
**3.33–5.96 s**. No sibling investigations or ordering calls were needed, and no
evaluation PDFs were modified. This was a separate confirmation-only run, not a
measured complete splitting run with live latency included. Exact source crops,
per-page scores/verdicts/times and unchanged-source verification are saved under
`tmp/orientation-eight-live-flash/` (`results.json`).

Displayed physical dimensions distinguish A3 landscape spreads from A4 sheets,
with a 5% tolerance. A4 landscape sheets are also retained whole. Unexpected
sizes and portrait A3 pages are retained whole and flagged. PDF rotation is
respected. Automatic evaluation processing additionally applies the orientation
and physical-sheet rules above; standalone analysis requires `--orientation`.

The standard splitter renders a 75 dpi overview, accepts clearly written pages,
and analyses a central 10% strip at the requested 200 dpi for overflow ownership.
Sparse logical pages use `ink_detection.detect_page_content` at full resolution.
Ambiguous ownership, overflow reaching the strip edge, and unexpected geometry
use the internal full-page fallback. Coloured-ruling cleanup prevents grid
fragments from linking text across the fold.

That shared entry point runs normal rognage with its default artifact filtering.
On confirmed ruled paper it exposes the existing large-blank refinement's retained
content, including an empty result. Ordinary cropping deliberately falls back to
the original mask when refinement is empty; the new entry point does not. Normal
cropping behaviour and defaults remain unchanged. The same coherent-stroke filter
checks seedless pages for plausible pencil writing. On plain paper, the existing
adaptive foreground extractor recovers letter-sized pencil components; faint
page-spanning scanner borders do not qualify. It does not retain a page
merely because unrelated faint pixels survive. There is no separate weak-pixel
veto or splitter-specific hole-removal algorithm.

Small initial stroke groups near the midpoint are followed towards supported
writing inside either page. Connected faint strokes can recover beyond the
cropper's local 2 mm radius, while page-wide ruling networks are rejected. The
ownership search allows larger horizontal gaps between strong pen regions,
penalises vertical gaps, and stops at a page anchor rather than joining entire
lines across both pages. Pale thin ruling is filtered before word grouping.

An isolated group with no page anchor is treated as a fold artifact. Comparable
support from both pages still produces an explicit review flag. Output rectangles
cover the actual terminal strokes plus 2 mm padding, not the expanded search
area. Each side's boundary remains independent. The regression suite and reviewed
real examples are documented in `cropping-test/page-splitting/README.md`.

Blank decisions use logical-page evidence after subtracting actual strokes
attributed to the opposite side. The report includes the crop analysis and the
content-refinement result for each output. A spread artifact that is absent from
the logical page's credible content cannot by itself veto blank removal.

## Limitations and interpretation

The report's flags are heuristic reasons, not calibrated confidence scores.
Repeated-hole detection and ruling suppression can be ambiguous around unusual
handwriting. Extremely faint isolated writing can escape detection. Horizontal
grouping can miss disconnected parts of long formulae or merge adjacent blocks.
Ambiguous cases intentionally duplicate content rather than choosing an owner.
Full-height rectangular overlap can include neighbouring text at other heights.

Blank removal is intentionally conservative. Do not interpret a retained page
as proof it contains student work. In particular, residual scan marks can keep
otherwise blank pages. Interro07's saved page-origin mappings provide a useful
comparison with earlier processing, but are not independent handwritten ground
truth and do not validate overflow attribution.

## Verification

```sh
python -m unittest tests.test_auto_page_split tests.test_ink_detection tests.test_crop_blank_margins tests.test_page_splitter_reverse tests.test_page_splitter_secondary -q
```

Tests cover both overflow directions at once, disconnected letters, ambiguous
ownership, opposite-side blank attribution, weak scan noise, pencil-only writing,
blank grids with binding holes, faint grid notes and fraction bars, A4/unknown
sizes, all four PDF rotations with CropBox
offsets, rendered export fidelity, input preservation, existing-output refusal,
and wholly blank copies.

## Updated full Interro07 batch (2026-11-03)

The current batch is `tmp/auto-page-split-interro07-support-20261103/`.
All 47 original PDFs were processed: 214 scanned pages, with 206 A3 spreads
and 8 A4 sheets. The output contains **347 logical pages**, removing **73 blank
halves**. All 345 pages kept by the earlier workflow remain kept. The two
additional retained halves contain exercise headings (Copie08 source 6/left and
Copie11 source 4/left), consistent with the reviewed content policy.

Exactly seven keep/remove decisions change from the preceding 66-removal batch,
matching the seven reviewed difficult blank fixtures. Their source halves were
visually checked; the two heading exports were rendered with Poppler and checked.
Original hashes, output page counts and displayed dimensions were verified.
`comparison.json` records the comparisons. The 26 source-page overlap review
flags remain; blank detection results do not validate all overflow attribution.

## Shared rognage trial (2026-11-02)

The current results are in `tmp/auto-page-split-interro07-shared-rognage/`.
Blank detection now uses the shared crop content analysis on each logical page.
The splitter-specific weak-pixel veto and hole cleanup have been removed.
Normal cropping still uses its original empty-refinement fallback.

Against Interro07's saved page-origin mappings, the updated detector removes
66 of the 75 previously omitted halves (88%), compared with 5 in the initial
prototype. All 345 previously retained pages are preserved. Nine previously
omitted halves remain kept, including two very pale pages retained by the
pencil-writing recovery check. These are conservative keep decisions; no
previously retained page was removed. The output has 354 logical pages.

All proposed removals were reviewed in source-page contact sheets. The final
PDF page counts, dimensions, and source hashes are checked, with results in
`comparison.json`. The 44 regression tests pass, covering normal crop behaviour,
blank ruled sheets with holes, faint writing, scanner borders, large faint
diagrams, overlap geometry, and rotation. This validates blank detection against
the existing Interro07 workflow; it is not an exhaustive annotation of handwriting
or overflow ownership. Centre-binding artifacts can still cause false overlap
proposals. No GUI integration or change to the existing processed PDFs was made.

## Initial Interro07 trial (2026-11-02; superseded)

The final trial is in `tmp/auto-page-split-interro07-final/` (ignored by Git).
It processed all 47 original PDFs: 214 scanned pages, comprising 206 A3 spreads
and 8 A4 sheets. Original-file hashes were verified unchanged. The exported
PDFs contain 415 logical pages. There are 47 extended spreads, including 21 with
both boundaries extended, and 69 source pages flagged for review. These counts
describe proposals, not validated correct overflow detections.

Comparison with the saved `Interro07/.copienator/page_origins/` mappings:

| Decision | Logical pages |
| --- | ---: |
| Previously retained, still retained | 345 |
| Previously retained, now removed | 0 |
| Previously omitted, now removed | 5 |
| Previously omitted, retained as uncertain | 58 |
| Previously omitted, classified as content | 12 |

All five proposed removals were visually inspected and appear blank. Selected
overflow previews and Poppler-rendered exports were also inspected. The coloured
ruling cleanup fixed a false 88.65 mm left extension on Copie36, source page 4:
the final 4.45 mm extension follows the equation at the centre. Binding artifacts
still generate false extensions elsewhere. No independent exhaustive annotation
of overflow ownership was made; preserving previously retained pages does not
prove that every stroke is correctly assigned or unclipped.

The 38 focused and existing regression tests pass, including pixel comparisons
of exported pages with rotated, offset-CropBox sources. The prototype is **not
ready for unattended GUI integration**. The next priorities are better blank-page
recall without erasing faint work, and better separation of binding artifacts
from centre handwriting. `comparison.json` lists the 70 keep/remove disagreements
for targeted follow-up. Neither the GUI nor the existing processed copies changed.

## Standard coarse-to-fine analysis

The former fast mode is now the only command-line analysis path. It renders a
75 dpi overview and only a central 10% strip at the requested 200 dpi. A positive
content gate accepts obvious writing; sparse or ambiguous pages still use full
resolution content detection. Ambiguous ownership or overflow reaching the strip
boundary uses the complete reference analysis. Unexpected geometry keeps the
reference path. Exported scan data remains at its original resolution.

```sh
python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/fast-review \
  --workers 4
```

The full Interro07 run in `tmp/auto-page-split-interro07-fast-final-20261103/`
takes 37.87 s including exports/previews, retains 347 pages and removes 73 blank
halves. All 420 keep/remove decisions agree with a fresh reference analysis.
Seven output boundaries differ: five by one pixel, and two inspected differences
in ruling/padding without observed loss of writing. This mode is not pixel-identical
to the reference. There is no analysis-mode selector; full-spread analysis remains an internal fallback.
See `cropping-test/page-splitting/experiments/FAST_ANALYSIS.md` for matched timing
comparisons, rejected low-resolution variants, quality checks and limitations.


## Interro06 blank-detection follow-up

The reviewed Interro06 suite originally passed 15/19 with fast analysis. All four
failures were unwritten halves retained for isolated ink marks. Compact solid
spots smaller than 2 mm no longer qualify as glyphs by themselves; their convex
hull must be filled, so small x-shaped glyphs remain eligible. A short almost
collinear smear within 15 mm of the top/bottom edge also needs neighbouring
writing support. The smear rule has additional area, size and density limits.
Both filters affect blank classification only; normal margin cropping is unchanged.

The four targets now pass, along with all 19 Interro06 cases, all 15 Interro07
blank cases and the writing-preservation controls. Inputs and reviewed expectations
are unchanged. The exact marks and initial measurements are saved under
`cropping-test/interro06/blank-investigation/`.