Automatic page splitting
This commit is contained in:
sebastien committed
1 parent
d4404e4c27
commit
78f3e4b2f0
13 files changed
+1959
-40
No files matched your search
@@ -0,0 +1,241 @@
|
||||
# Experimental automatic page splitting
|
||||
|
||||
## Python GUI
|
||||
|
||||
The “Séparer et réordonner les pages” step now offers **Manuellement** and
|
||||
**Automatiquement**. Automatic processing runs in the background using the
|
||||
standard splitter, publishes PDFs in `Copies`, and preserves source scans in
|
||||
`Copies Originales`. It retains source-page order. The parallel-copy setting
|
||||
appears only for automatic processing. Existing single-copy and marked-copy
|
||||
selection work in both modes.
|
||||
|
||||
Choose Manuellement afterward to review the saved proposals against the
|
||||
preserved original. The editor displays independent overlapping output crops
|
||||
and crosses out removed blank pages. Advancing accepts the displayed decision;
|
||||
the existing keep/discard shortcuts override it. Moving a splitter resumes
|
||||
manual splitting on that page. Reviewing automatic proposals preserves their
|
||||
source-page order; the original manual workflow retains its booklet ordering.
|
||||
Proposals are reused only while both source and output fingerprints still match.
|
||||
|
||||
Each automatic run saves its inputs, report/gallery, and previous outputs under
|
||||
`.copienator/runs/page-split-*`. All PDFs are analysed before publication, and
|
||||
PDFs and provenance are published together with rollback on installation errors.
|
||||
An entirely blank copy keeps its preserved source and receives a manual-review
|
||||
flag instead of an invalid empty output PDF.
|
||||
|
||||
The equivalent CLI command is:
|
||||
|
||||
```sh
|
||||
python -m copienator page-split EVALUATION --automatic --workers 4
|
||||
```
|
||||
|
||||
Omit `--automatic` for the interactive editor. The standalone review command
|
||||
below remains available for experiments without installing results into `Copies`.
|
||||
|
||||
Run from the repository root, on original scans rather than already split copies:
|
||||
|
||||
```sh
|
||||
python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/split-review --workers 4
|
||||
```
|
||||
|
||||
The output directory must not already exist. Input PDFs are never overwritten.
|
||||
This command is deliberately separate from the GUI and from the interactive
|
||||
`page-split` command. It does not reorder booklets or use secondary splitters.
|
||||
Outputs follow source-page order, left then right. They must not replace the
|
||||
current processed evaluation without review and the appropriate later ordering.
|
||||
|
||||
## Outputs
|
||||
|
||||
- `index.html`: source-page previews, proposed boundaries, blank decisions,
|
||||
uncertainty reasons, and links to exported PDFs.
|
||||
- `pdfs/`: clipped PDFs retaining source scan data, without rasterizing output.
|
||||
- `report.json`: decisions, original file hashes, original rotations/CropBoxes,
|
||||
source page numbers, displayed clip coordinates, and output page numbers.
|
||||
- `review/`: JPEG previews. Green marks the left output, purple the right/whole
|
||||
output, orange the detected centre groups; red shading marks a blank removal.
|
||||
|
||||
A wholly blank copy has no output PDF, but still has report entries. An interrupted
|
||||
run may leave partial outputs; use a new output directory for a subsequent run.
|
||||
|
||||
## Decisions
|
||||
|
||||
Displayed physical dimensions distinguish A3 landscape spreads from A4 sheets,
|
||||
with a 5% tolerance. A4 landscape sheets are also retained whole. Unexpected
|
||||
sizes and portrait A3 pages are retained whole and flagged. PDF rotation is
|
||||
respected; handwriting orientation is not inferred.
|
||||
|
||||
The standard splitter renders a 75 dpi overview, accepts clearly written pages,
|
||||
and analyses a central 10% strip at the requested 200 dpi for overflow ownership.
|
||||
Sparse logical pages use `ink_detection.detect_page_content` at full resolution.
|
||||
Ambiguous ownership, overflow reaching the strip edge, and unexpected geometry
|
||||
use the internal full-page fallback. Coloured-ruling cleanup prevents grid
|
||||
fragments from linking text across the fold.
|
||||
|
||||
That shared entry point runs normal rognage with its default artifact filtering.
|
||||
On confirmed ruled paper it exposes the existing large-blank refinement's retained
|
||||
content, including an empty result. Ordinary cropping deliberately falls back to
|
||||
the original mask when refinement is empty; the new entry point does not. Normal
|
||||
cropping behaviour and defaults remain unchanged. The same coherent-stroke filter
|
||||
checks seedless pages for plausible pencil writing. On plain paper, the existing
|
||||
adaptive foreground extractor recovers letter-sized pencil components; faint
|
||||
page-spanning scanner borders do not qualify. It does not retain a page
|
||||
merely because unrelated faint pixels survive. There is no separate weak-pixel
|
||||
veto or splitter-specific hole-removal algorithm.
|
||||
|
||||
Small initial stroke groups near the midpoint are followed towards supported
|
||||
writing inside either page. Connected faint strokes can recover beyond the
|
||||
cropper's local 2 mm radius, while page-wide ruling networks are rejected. The
|
||||
ownership search allows larger horizontal gaps between strong pen regions,
|
||||
penalises vertical gaps, and stops at a page anchor rather than joining entire
|
||||
lines across both pages. Pale thin ruling is filtered before word grouping.
|
||||
|
||||
An isolated group with no page anchor is treated as a fold artifact. Comparable
|
||||
support from both pages still produces an explicit review flag. Output rectangles
|
||||
cover the actual terminal strokes plus 2 mm padding, not the expanded search
|
||||
area. Each side's boundary remains independent. The regression suite and reviewed
|
||||
real examples are documented in `cropping-test/page-splitting/README.md`.
|
||||
|
||||
Blank decisions use logical-page evidence after subtracting actual strokes
|
||||
attributed to the opposite side. The report includes the crop analysis and the
|
||||
content-refinement result for each output. A spread artifact that is absent from
|
||||
the logical page's credible content cannot by itself veto blank removal.
|
||||
|
||||
## Limitations and interpretation
|
||||
|
||||
The report's flags are heuristic reasons, not calibrated confidence scores.
|
||||
Repeated-hole detection and ruling suppression can be ambiguous around unusual
|
||||
handwriting. Extremely faint isolated writing can escape detection. Horizontal
|
||||
grouping can miss disconnected parts of long formulae or merge adjacent blocks.
|
||||
Ambiguous cases intentionally duplicate content rather than choosing an owner.
|
||||
Full-height rectangular overlap can include neighbouring text at other heights.
|
||||
|
||||
Blank removal is intentionally conservative. Do not interpret a retained page
|
||||
as proof it contains student work. In particular, residual scan marks can keep
|
||||
otherwise blank pages. Interro07's saved page-origin mappings provide a useful
|
||||
comparison with earlier processing, but are not independent handwritten ground
|
||||
truth and do not validate overflow attribution.
|
||||
|
||||
## Verification
|
||||
|
||||
```sh
|
||||
python -m unittest tests.test_auto_page_split tests.test_ink_detection tests.test_crop_blank_margins tests.test_page_splitter_reverse tests.test_page_splitter_secondary -q
|
||||
```
|
||||
|
||||
Tests cover both overflow directions at once, disconnected letters, ambiguous
|
||||
ownership, opposite-side blank attribution, weak scan noise, pencil-only writing,
|
||||
blank grids with binding holes, faint grid notes and fraction bars, A4/unknown
|
||||
sizes, all four PDF rotations with CropBox
|
||||
offsets, rendered export fidelity, input preservation, existing-output refusal,
|
||||
and wholly blank copies.
|
||||
|
||||
## Updated full Interro07 batch (2026-11-03)
|
||||
|
||||
The current batch is `tmp/auto-page-split-interro07-support-20261103/`.
|
||||
All 47 original PDFs were processed: 214 scanned pages, with 206 A3 spreads
|
||||
and 8 A4 sheets. The output contains **347 logical pages**, removing **73 blank
|
||||
halves**. All 345 pages kept by the earlier workflow remain kept. The two
|
||||
additional retained halves contain exercise headings (Copie08 source 6/left and
|
||||
Copie11 source 4/left), consistent with the reviewed content policy.
|
||||
|
||||
Exactly seven keep/remove decisions change from the preceding 66-removal batch,
|
||||
matching the seven reviewed difficult blank fixtures. Their source halves were
|
||||
visually checked; the two heading exports were rendered with Poppler and checked.
|
||||
Original hashes, output page counts and displayed dimensions were verified.
|
||||
`comparison.json` records the comparisons. The 26 source-page overlap review
|
||||
flags remain; blank detection results do not validate all overflow attribution.
|
||||
|
||||
## Shared rognage trial (2026-11-02)
|
||||
|
||||
The current results are in `tmp/auto-page-split-interro07-shared-rognage/`.
|
||||
Blank detection now uses the shared crop content analysis on each logical page.
|
||||
The splitter-specific weak-pixel veto and hole cleanup have been removed.
|
||||
Normal cropping still uses its original empty-refinement fallback.
|
||||
|
||||
Against Interro07's saved page-origin mappings, the updated detector removes
|
||||
66 of the 75 previously omitted halves (88%), compared with 5 in the initial
|
||||
prototype. All 345 previously retained pages are preserved. Nine previously
|
||||
omitted halves remain kept, including two very pale pages retained by the
|
||||
pencil-writing recovery check. These are conservative keep decisions; no
|
||||
previously retained page was removed. The output has 354 logical pages.
|
||||
|
||||
All proposed removals were reviewed in source-page contact sheets. The final
|
||||
PDF page counts, dimensions, and source hashes are checked, with results in
|
||||
`comparison.json`. The 44 regression tests pass, covering normal crop behaviour,
|
||||
blank ruled sheets with holes, faint writing, scanner borders, large faint
|
||||
diagrams, overlap geometry, and rotation. This validates blank detection against
|
||||
the existing Interro07 workflow; it is not an exhaustive annotation of handwriting
|
||||
or overflow ownership. Centre-binding artifacts can still cause false overlap
|
||||
proposals. No GUI integration or change to the existing processed PDFs was made.
|
||||
|
||||
## Initial Interro07 trial (2026-11-02; superseded)
|
||||
|
||||
The final trial is in `tmp/auto-page-split-interro07-final/` (ignored by Git).
|
||||
It processed all 47 original PDFs: 214 scanned pages, comprising 206 A3 spreads
|
||||
and 8 A4 sheets. Original-file hashes were verified unchanged. The exported
|
||||
PDFs contain 415 logical pages. There are 47 extended spreads, including 21 with
|
||||
both boundaries extended, and 69 source pages flagged for review. These counts
|
||||
describe proposals, not validated correct overflow detections.
|
||||
|
||||
Comparison with the saved `Interro07/.copienator/page_origins/` mappings:
|
||||
|
||||
| Decision | Logical pages |
|
||||
| --- | ---: |
|
||||
| Previously retained, still retained | 345 |
|
||||
| Previously retained, now removed | 0 |
|
||||
| Previously omitted, now removed | 5 |
|
||||
| Previously omitted, retained as uncertain | 58 |
|
||||
| Previously omitted, classified as content | 12 |
|
||||
|
||||
All five proposed removals were visually inspected and appear blank. Selected
|
||||
overflow previews and Poppler-rendered exports were also inspected. The coloured
|
||||
ruling cleanup fixed a false 88.65 mm left extension on Copie36, source page 4:
|
||||
the final 4.45 mm extension follows the equation at the centre. Binding artifacts
|
||||
still generate false extensions elsewhere. No independent exhaustive annotation
|
||||
of overflow ownership was made; preserving previously retained pages does not
|
||||
prove that every stroke is correctly assigned or unclipped.
|
||||
|
||||
The 38 focused and existing regression tests pass, including pixel comparisons
|
||||
of exported pages with rotated, offset-CropBox sources. The prototype is **not
|
||||
ready for unattended GUI integration**. The next priorities are better blank-page
|
||||
recall without erasing faint work, and better separation of binding artifacts
|
||||
from centre handwriting. `comparison.json` lists the 70 keep/remove disagreements
|
||||
for targeted follow-up. Neither the GUI nor the existing processed copies changed.
|
||||
|
||||
## Standard coarse-to-fine analysis
|
||||
|
||||
The former fast mode is now the only command-line analysis path. It renders a
|
||||
75 dpi overview and only a central 10% strip at the requested 200 dpi. A positive
|
||||
content gate accepts obvious writing; sparse or ambiguous pages still use full
|
||||
resolution content detection. Ambiguous ownership or overflow reaching the strip
|
||||
boundary uses the complete reference analysis. Unexpected geometry keeps the
|
||||
reference path. Exported scan data remains at its original resolution.
|
||||
|
||||
```sh
|
||||
python -m copienator.auto_page_split 'Interro07/Copies Originales' tmp/fast-review \
|
||||
--workers 4
|
||||
```
|
||||
|
||||
The full Interro07 run in `tmp/auto-page-split-interro07-fast-final-20261103/`
|
||||
takes 37.87 s including exports/previews, retains 347 pages and removes 73 blank
|
||||
halves. All 420 keep/remove decisions agree with a fresh reference analysis.
|
||||
Seven output boundaries differ: five by one pixel, and two inspected differences
|
||||
in ruling/padding without observed loss of writing. This mode is not pixel-identical
|
||||
to the reference. There is no analysis-mode selector; full-spread analysis remains an internal fallback.
|
||||
See `cropping-test/page-splitting/experiments/FAST_ANALYSIS.md` for matched timing
|
||||
comparisons, rejected low-resolution variants, quality checks and limitations.
|
||||
|
||||
|
||||
## Interro06 blank-detection follow-up
|
||||
|
||||
The reviewed Interro06 suite originally passed 15/19 with fast analysis. All four
|
||||
failures were unwritten halves retained for isolated ink marks. Compact solid
|
||||
spots smaller than 2 mm no longer qualify as glyphs by themselves; their convex
|
||||
hull must be filled, so small x-shaped glyphs remain eligible. A short almost
|
||||
collinear smear within 15 mm of the top/bottom edge also needs neighbouring
|
||||
writing support. The smear rule has additional area, size and density limits.
|
||||
Both filters affect blank classification only; normal margin cropping is unchanged.
|
||||
|
||||
The four targets now pass, along with all 19 Interro06 cases, all 15 Interro07
|
||||
blank cases and the writing-preservation controls. Inputs and reviewed expectations
|
||||
are unchanged. The exact marks and initial measurements are saved under
|
||||
`cropping-test/interro06/blank-investigation/`.
|
||||
Reference in new issue
Block a user