# Experimental automatic page splitting ## Python GUI The “Séparer et réordonner les pages” step now offers **Manuellement** and **Automatiquement**. Automatic processing runs in the background using the standard splitter, publishes PDFs in `Copies`, and preserves source scans in `Copies Originales`. A4 scans keep their positions; consecutive A3 scans are paired and exported in booklet reading order (outer right, inner left, inner right, outer left). The parallel-copy setting appears only for automatic processing. Existing single-copy and marked-copy selection work in both modes. Choose Manuellement afterward to review the saved proposals against the preserved original. The editor displays independent overlapping output crops and crosses out removed blank pages. Advancing accepts the displayed decision; the existing keep/discard shortcuts override it. Moving a splitter resumes manual splitting on that page. Reviewing automatic proposals preserves their saved reading order, source rotations and individual-half corrections. Proposals are reused only while both source and output fingerprints still match. Each automatic run saves its inputs, report/gallery, and previous outputs under `.copienator/runs/page-split-*`. All PDFs are analysed before publication, and PDFs and provenance are published together with rollback on installation errors. An entirely blank copy keeps its preserved source and receives a manual-review flag instead of an invalid empty output PDF. The equivalent CLI command is: ```sh python -m copienator page-split EVALUATION --automatic --workers 4 ``` Omit `--automatic` for the interactive editor. The standalone review command below remains available for experiments without installing results into `Copies`. Run from the repository root, on original scans rather than already split copies: ```sh python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/split-review --workers 4 ``` The output directory must not already exist. Input PDFs are never overwritten. This command is deliberately separate from the GUI and from the interactive `page-split` command. It does not reorder booklets or use secondary splitters. Outputs follow source-page order, left then right. They must not replace the current processed evaluation without review and the appropriate later ordering. ## Outputs - `index.html`: source-page previews, proposed boundaries, blank decisions, uncertainty reasons, and links to exported PDFs. - `pdfs/`: clipped PDFs retaining source scan data, without rasterizing output. - `report.json`: decisions, original file hashes, original rotations/CropBoxes, source page numbers, displayed clip coordinates, and output page numbers. - `review/`: JPEG previews. Green marks the left output, purple the right/whole output, orange the detected centre groups; red shading marks a blank removal. A wholly blank copy has no output PDF, but still has report entries. An interrupted run may leave partial outputs; use a new output directory for a subsequent run. ## Decisions ### Orientation and physical sheets After provisional blank/middle detection, each nonblank logical page is screened using the bundled conservative orientation model. Model weights and dependencies are included in the installed package; no runtime model download is required. Gemini confirms only candidates, returning `upright`, `inverted` or `inconclusive`. An inconclusive reply is treated as upright for the remainder of the run. `MODEL_FOR_PAGE_ORIENTATION` can select the confirmation model; its default `None` inherits `MODEL_FOR_LABEL_ID`. Orientation confirmations use low thinking; page-ordering requests use medium thinking. Both use structured responses. A4 scans are independent, even when adjacent: correct a confirmed inversion and keep its position. A3 scans are paired only when consecutive; an unpaired A3 scan is flagged for manual review. Once an A3 inversion is confirmed, Gemini checks all other nonblank pages on that sheet, including pages the local model did not flag. Blank logical slots remain available until sheet ordering is resolved. - If every logical page is blank or confirmed inverted, turn both source faces 180°, recompute blank/middle/overlap decisions, then apply normal booklet order. - Otherwise turn only the inverted logical pages. Send the full evaluation `enonce.pdf` (including supported filename aliases) and all sheet images, including blanks, to Gemini for a reading-order permutation. Validate that every slot occurs exactly once, then remove blank pages from the final export. A request failure, malformed orientation reply, invalid permutation or missing énoncé for mixed-sheet ordering aborts analysis before publication. These errors are not interpreted as inconclusive handwriting. Existing copies and originals remain untouched. The report stores screen scores, replies, source/logical rotations, physical-sheet grouping and final page numbers. Manual acceptance preserves that order; reinstating a blank proposal restores its reading-order slot. The standalone analysis tool remains a geometry-only baseline by default. To exercise orientation and booklet ordering without publishing into `Copies`: ```sh python -m copienator.auto_page_split 'EVALUATION/Copies Originales' tmp/oriented-review \ --workers 4 --orientation --statement EVALUATION/enonce.pdf ``` See [the decision record](page_orientation.md) for the trained model, rejected approaches and calibration limits. Its earlier timings exclude live Gemini latency. ### Integration verification (2026-11-04) The maintained implementation passes 65 focused tests, including A4 independence, A3 booklet order, blank siblings, inconclusive replies, full-source reanalysis, mixed-sheet image rotation and full-statement ordering payloads, invalid response rejection, failure preservation, and manual acceptance/reinstatement/secondary cuts. Wheel inspection confirms that weights, settings, ONNX model and license ship with the package. An isolated full Interro07 run took **44.26 seconds** with four workers: 47 PDFs, 214 scans, 347 kept pages, 73 blank halves removed, eight detector candidates. Gemini replies were mocked as upright; no live requests were sent. Source hashes, blank decisions and boundaries match the reference; output page mappings match the new reading order. This checks local computation and request routing, not live Gemini accuracy or latency. The evidence is under `tmp/orientation-implemented-interro07-verified/` (`timing.json`, `report.json`, `verification.json`). A subsequent one-off live check of precisely those eight candidates used `gemini-3.8-flash`, thinking `low`, four concurrent workers and no retries. Gemini classified **all eight as upright**, with no inconclusive or inverted verdicts. Request wall time was **9.64 s**, with individual calls taking **3.33–5.96 s**. No sibling investigations or ordering calls were needed, and no evaluation PDFs were modified. This was a separate confirmation-only run, not a measured complete splitting run with live latency included. Exact source crops, per-page scores/verdicts/times and unchanged-source verification are saved under `tmp/orientation-eight-live-flash/` (`results.json`). Displayed physical dimensions distinguish A3 landscape spreads from A4 sheets, with a 5% tolerance. A4 landscape sheets are also retained whole. Unexpected sizes and portrait A3 pages are retained whole and flagged. PDF rotation is respected. Automatic evaluation processing additionally applies the orientation and physical-sheet rules above; standalone analysis requires `--orientation`. The standard splitter renders a 75 dpi overview, accepts clearly written pages, and analyses a central 10% strip at the requested 200 dpi for overflow ownership. Sparse logical pages use `ink_detection.detect_page_content` at full resolution. Ambiguous ownership, overflow reaching the strip edge, and unexpected geometry use the internal full-page fallback. Coloured-ruling cleanup prevents grid fragments from linking text across the fold. That shared entry point runs normal rognage with its default artifact filtering. On confirmed ruled paper it exposes the existing large-blank refinement's retained content, including an empty result. Ordinary cropping deliberately falls back to the original mask when refinement is empty; the new entry point does not. Normal cropping behaviour and defaults remain unchanged. The same coherent-stroke filter checks seedless pages for plausible pencil writing. On plain paper, the existing adaptive foreground extractor recovers letter-sized pencil components; faint page-spanning scanner borders do not qualify. It does not retain a page merely because unrelated faint pixels survive. There is no separate weak-pixel veto or splitter-specific hole-removal algorithm. Small initial stroke groups near the midpoint are followed towards supported writing inside either page. Connected faint strokes can recover beyond the cropper's local 2 mm radius, while page-wide ruling networks are rejected. The ownership search allows larger horizontal gaps between strong pen regions, penalises vertical gaps, and stops at a page anchor rather than joining entire lines across both pages. Pale thin ruling is filtered before word grouping. An isolated group with no page anchor is treated as a fold artifact. Comparable support from both pages still produces an explicit review flag. Output rectangles cover the actual terminal strokes plus 2 mm padding, not the expanded search area. Each side's boundary remains independent. The regression suite and reviewed real examples are documented in `cropping-test/page-splitting/README.md`. Blank decisions use logical-page evidence after subtracting actual strokes attributed to the opposite side. The report includes the crop analysis and the content-refinement result for each output. A spread artifact that is absent from the logical page's credible content cannot by itself veto blank removal. ## Limitations and interpretation The report's flags are heuristic reasons, not calibrated confidence scores. Repeated-hole detection and ruling suppression can be ambiguous around unusual handwriting. Extremely faint isolated writing can escape detection. Horizontal grouping can miss disconnected parts of long formulae or merge adjacent blocks. Ambiguous cases intentionally duplicate content rather than choosing an owner. Full-height rectangular overlap can include neighbouring text at other heights. Blank removal is intentionally conservative. Do not interpret a retained page as proof it contains student work. In particular, residual scan marks can keep otherwise blank pages. Interro07's saved page-origin mappings provide a useful comparison with earlier processing, but are not independent handwritten ground truth and do not validate overflow attribution. ## Verification ```sh python -m unittest tests.test_auto_page_split tests.test_ink_detection tests.test_crop_blank_margins tests.test_page_splitter_reverse tests.test_page_splitter_secondary -q ``` Tests cover both overflow directions at once, disconnected letters, ambiguous ownership, opposite-side blank attribution, weak scan noise, pencil-only writing, blank grids with binding holes, faint grid notes and fraction bars, A4/unknown sizes, all four PDF rotations with CropBox offsets, rendered export fidelity, input preservation, existing-output refusal, and wholly blank copies. ## Updated full Interro07 batch (2026-11-03) The current batch is `tmp/auto-page-split-interro07-support-20261103/`. All 47 original PDFs were processed: 214 scanned pages, with 206 A3 spreads and 8 A4 sheets. The output contains **347 logical pages**, removing **73 blank halves**. All 345 pages kept by the earlier workflow remain kept. The two additional retained halves contain exercise headings (Copie08 source 6/left and Copie11 source 4/left), consistent with the reviewed content policy. Exactly seven keep/remove decisions change from the preceding 66-removal batch, matching the seven reviewed difficult blank fixtures. Their source halves were visually checked; the two heading exports were rendered with Poppler and checked. Original hashes, output page counts and displayed dimensions were verified. `comparison.json` records the comparisons. The 26 source-page overlap review flags remain; blank detection results do not validate all overflow attribution. ## Shared rognage trial (2026-11-02) The current results are in `tmp/auto-page-split-interro07-shared-rognage/`. Blank detection now uses the shared crop content analysis on each logical page. The splitter-specific weak-pixel veto and hole cleanup have been removed. Normal cropping still uses its original empty-refinement fallback. Against Interro07's saved page-origin mappings, the updated detector removes 66 of the 75 previously omitted halves (88%), compared with 5 in the initial prototype. All 345 previously retained pages are preserved. Nine previously omitted halves remain kept, including two very pale pages retained by the pencil-writing recovery check. These are conservative keep decisions; no previously retained page was removed. The output has 354 logical pages. All proposed removals were reviewed in source-page contact sheets. The final PDF page counts, dimensions, and source hashes are checked, with results in `comparison.json`. The 44 regression tests pass, covering normal crop behaviour, blank ruled sheets with holes, faint writing, scanner borders, large faint diagrams, overlap geometry, and rotation. This validates blank detection against the existing Interro07 workflow; it is not an exhaustive annotation of handwriting or overflow ownership. Centre-binding artifacts can still cause false overlap proposals. No GUI integration or change to the existing processed PDFs was made. ## Initial Interro07 trial (2026-11-02; superseded) The final trial is in `tmp/auto-page-split-interro07-final/` (ignored by Git). It processed all 47 original PDFs: 214 scanned pages, comprising 206 A3 spreads and 8 A4 sheets. Original-file hashes were verified unchanged. The exported PDFs contain 415 logical pages. There are 47 extended spreads, including 21 with both boundaries extended, and 69 source pages flagged for review. These counts describe proposals, not validated correct overflow detections. Comparison with the saved `Interro07/.copienator/page_origins/` mappings: | Decision | Logical pages | | --- | ---: | | Previously retained, still retained | 345 | | Previously retained, now removed | 0 | | Previously omitted, now removed | 5 | | Previously omitted, retained as uncertain | 58 | | Previously omitted, classified as content | 12 | All five proposed removals were visually inspected and appear blank. Selected overflow previews and Poppler-rendered exports were also inspected. The coloured ruling cleanup fixed a false 88.65 mm left extension on Copie36, source page 4: the final 4.45 mm extension follows the equation at the centre. Binding artifacts still generate false extensions elsewhere. No independent exhaustive annotation of overflow ownership was made; preserving previously retained pages does not prove that every stroke is correctly assigned or unclipped. The 38 focused and existing regression tests pass, including pixel comparisons of exported pages with rotated, offset-CropBox sources. The prototype is **not ready for unattended GUI integration**. The next priorities are better blank-page recall without erasing faint work, and better separation of binding artifacts from centre handwriting. `comparison.json` lists the 70 keep/remove disagreements for targeted follow-up. Neither the GUI nor the existing processed copies changed. ## Standard coarse-to-fine analysis The former fast mode is now the only command-line analysis path. It renders a 75 dpi overview and only a central 10% strip at the requested 200 dpi. A positive content gate accepts obvious writing; sparse or ambiguous pages still use full resolution content detection. Ambiguous ownership or overflow reaching the strip boundary uses the complete reference analysis. Unexpected geometry keeps the reference path. Exported scan data remains at its original resolution. ```sh python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/fast-review \ --workers 4 ``` The full Interro07 run in `tmp/auto-page-split-interro07-fast-final-20261103/` takes 37.87 s including exports/previews, retains 347 pages and removes 73 blank halves. All 420 keep/remove decisions agree with a fresh reference analysis. Seven output boundaries differ: five by one pixel, and two inspected differences in ruling/padding without observed loss of writing. This mode is not pixel-identical to the reference. There is no analysis-mode selector; full-spread analysis remains an internal fallback. See `cropping-test/page-splitting/experiments/FAST_ANALYSIS.md` for matched timing comparisons, rejected low-resolution variants, quality checks and limitations. ## Interro06 blank-detection follow-up The reviewed Interro06 suite originally passed 15/19 with fast analysis. All four failures were unwritten halves retained for isolated ink marks. Compact solid spots smaller than 2 mm no longer qualify as glyphs by themselves; their convex hull must be filled, so small x-shaped glyphs remain eligible. A short almost collinear smear within 15 mm of the top/bottom edge also needs neighbouring writing support. The smear rule has additional area, size and density limits. Both filters affect blank classification only; normal margin cropping is unchanged. The four targets now pass, along with all 19 Interro06 cases, all 15 Interro07 blank cases and the writing-preservation controls. Inputs and reviewed expectations are unchanged. The exact marks and initial measurements are saved under `cropping-test/interro06/blank-investigation/`.