Compare commits
16
Commits
06d7bad04e
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0a453403cf | ||
|
|
5b8215e7f5 | ||
|
|
9b22a8a137 | ||
|
|
0a86403ca6 | ||
|
|
5080274e8f | ||
|
|
d60d5479d6 | ||
|
|
e8b8a11c5b | ||
|
|
060859ddef | ||
|
|
dfce79f240 | ||
|
|
f7b3689f23 | ||
|
|
0a167072ed | ||
|
|
3969580e01 | ||
|
|
359c62004c | ||
|
|
13a08f6cbd | ||
|
|
bf05272797 | ||
|
|
db4ed2ef31 |
+4
-1
@@ -1,4 +1,6 @@
|
|||||||
OLD/
|
OLD/
|
||||||
|
/Interro*/
|
||||||
|
/DS*/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.py[cod]
|
*.py[cod]
|
||||||
.venv/
|
.venv/
|
||||||
@@ -7,4 +9,5 @@ dist/
|
|||||||
*.egg-info/
|
*.egg-info/
|
||||||
config.py
|
config.py
|
||||||
.copienator-gui.json
|
.copienator-gui.json
|
||||||
.copienator/
|
.copienator/
|
||||||
|
tmp/
|
||||||
|
|||||||
+58
@@ -173,6 +173,62 @@ Lancer l'assistant avec :
|
|||||||
python -m copienator gui
|
python -m copienator gui
|
||||||
#+END_SRC
|
#+END_SRC
|
||||||
|
|
||||||
|
Sous Linux ou macOS, le lanceur exécutable =./start-gui.sh= démarre aussi
|
||||||
|
l’interface, depuis n’importe quel répertoire. Il utilise le Python de
|
||||||
|
=.venv= s’il existe, sinon =python3= du PATH. On peut lui passer le dossier
|
||||||
|
d’évaluation : =./start-gui.sh Interro=. Sans argument, il ouvre le
|
||||||
|
sous-dossier immédiat non masqué le plus récemment modifié du répertoire
|
||||||
|
courant, en excluant =copienator=, =copienator_gui=, =tests=, =OLD=,
|
||||||
|
=__pycache__=, =build=, =dist=, =*.egg-info=, =venv=, =env= et
|
||||||
|
=node_modules=. S’il n’y a aucun dossier admissible, l’interface démarre
|
||||||
|
sans évaluation. Un chemin explicite reste utilisable même s’il est exclu
|
||||||
|
de la sélection automatique.
|
||||||
|
|
||||||
|
=Recharger — vérifier à nouveau= relit les fichiers d’entrée du dossier.
|
||||||
|
Pour reprendre le découpage d’une copie, sélectionnez-la dans =Copies
|
||||||
|
détectées=, cliquez sur =Refaire la copie sélectionnée=, puis sur
|
||||||
|
=Exécuter=. Le découpage repart de l’original conservé.
|
||||||
|
Dans la fenêtre de découpage, =i= inverse l’ordre de toutes les pages
|
||||||
|
(dernière vers première) et reprend à la nouvelle première page. Les choix
|
||||||
|
de découpage déjà saisis sont effacés ; les rotations globales sont conservées.
|
||||||
|
Ce raccourci est configurable via =PAGE_SPLITTER_KB["reverse_pages"]=.
|
||||||
|
=Copier la commande= copie les commandes complètes (à lancer depuis la
|
||||||
|
racine du projet). Dans la console, les boutons de copie et le clic droit
|
||||||
|
permettent de copier la sélection ou toute la sortie ; =Ctrl+C= et
|
||||||
|
=Ctrl+A= sont également disponibles (=Cmd= sous macOS).
|
||||||
|
|
||||||
|
Pendant =Découper une partie à gauche pour détection des labels=, =n= décale
|
||||||
|
la zone de 50 px vers la droite, =N= de 100 px, =t= de 50 px vers la
|
||||||
|
gauche et =l= l’élargit de 50 px. =1= utilise les pages entières. =s=
|
||||||
|
signale la copie en erreur et passe à la suivante ; =Entrée= valide et
|
||||||
|
enregistre la découpe affichée.
|
||||||
|
Une copie ignorée conserve ses anciennes découpes. Les signalements sont
|
||||||
|
conservés dans =.copienator/copy_errors.json=, même après fermeture.
|
||||||
|
Dans =Séparer et réordonner les pages=, =Traiter les copies signalées=
|
||||||
|
reprend ces copies à partir des originaux conservés. Le même bouton dans
|
||||||
|
=Découper une partie à gauche pour détection des labels= reprend leur
|
||||||
|
découpage ; chaque signalement
|
||||||
|
est effacé seulement après validation et enregistrement avec =Entrée=.
|
||||||
|
Fermer la fenêtre, appuyer à nouveau sur =s= ou rencontrer une erreur
|
||||||
|
conserve le signalement. Les commandes =page-split= et =crop-labels=
|
||||||
|
acceptent aussi =--marked= pour traiter les copies signalées de l’évaluation.
|
||||||
|
|
||||||
|
Avec =SHOW_PERSONAL_STEPS = True=, =Analyser l’énoncé= propose le choix
|
||||||
|
entre Gemini et =Énoncés et solutions personnels (SHEETINFO)=. Ce dernier
|
||||||
|
lance =python -m copienator statement-personal Interro= : il lit
|
||||||
|
=enonce.tex=, utilise le service d’exercices sur =localhost:8080= pour
|
||||||
|
générer =Text=, =Sol=, =Text2=, =Sol2= et les barèmes personnels =Persp=,
|
||||||
|
et écrit un groupe par exercice dans =label_groups=.
|
||||||
|
|
||||||
|
Deux boutons ouvrent ensuite des actions facultatives avec aperçu de
|
||||||
|
commande et bouton =Exécuter= : =Regrouper avec Gemini…= remplace seulement
|
||||||
|
=label_groups= (=statement Interro --groups-only=) ; =Remplacer Persp avec
|
||||||
|
Gemini…= régénère seulement les barèmes des groupes actuels
|
||||||
|
(=statement Interro --persp-only=), avec le même prompt que le parcours
|
||||||
|
Gemini complet. Une réponse incomplète ou un échec laisse les anciens
|
||||||
|
barèmes en place. On peut ignorer ces étapes et conserver les résultats
|
||||||
|
personnels.
|
||||||
|
|
||||||
On peut aussi ouvrir directement une évaluation avec =python -m copienator gui
|
On peut aussi ouvrir directement une évaluation avec =python -m copienator gui
|
||||||
Interro=. L'interface conserve l'état et l'historique des étapes dans
|
Interro=. L'interface conserve l'état et l'historique des étapes dans
|
||||||
=Interro/.copienator-gui.json=, et les sorties complètes dans
|
=Interro/.copienator-gui.json=, et les sorties complètes dans
|
||||||
@@ -203,6 +259,8 @@ Les chemins des étapes personnelles peuvent être adaptés avec
|
|||||||
|
|
||||||
* Documentation complémentaire
|
* Documentation complémentaire
|
||||||
|
|
||||||
|
- [[file:docs/final_output.md][Fichiers finaux dans A Rendre]] : contenu du JPEG, sélection et
|
||||||
|
pagination du PDF, JPEG par réponse, =score.json=, =info.json= et diffusion.
|
||||||
- [[file:Script.org][Référence des étapes et des scripts]] : commandes, arguments,
|
- [[file:Script.org][Référence des étapes et des scripts]] : commandes, arguments,
|
||||||
prérequis, fichiers produits et parcours alternatifs.
|
prérequis, fichiers produits et parcours alternatifs.
|
||||||
- [[file:Architecture.org][Architecture et conventions de développement]] : API commune,
|
- [[file:Architecture.org][Architecture et conventions de développement]] : API commune,
|
||||||
|
|||||||
+28
-3
@@ -290,8 +290,31 @@ OU
|
|||||||
3. =python -m copienator giving-names Interro BGnot=
|
3. =python -m copienator giving-names Interro BGnot=
|
||||||
|
|
||||||
Crée un dossier =A Rendre= avec des liens symboliques vers
|
Crée un dossier =A Rendre= avec des liens symboliques vers
|
||||||
+ La copie à rendre
|
+ =<nom>.jpg= : la correction complète concaténée (=Concat.jpg=)
|
||||||
|
+ =<nom>.pdf=, si disponible : la correction filtrée, avec contexte,
|
||||||
|
énoncés et solutions dans le parcours =BGnot= (=Concat_F.pdf=)
|
||||||
+ un fichier =score.json= qui contient les notes par question
|
+ un fichier =score.json= qui contient les notes par question
|
||||||
|
+ =info.json= : pour chaque label, =present= (réponse fournie),
|
||||||
|
=not_empty= (réponse non vide compilée), =touched= (présente dans le
|
||||||
|
PDF filtré) et =score= (même valeur que dans =score.json=)
|
||||||
|
+ =answers/*.jpg=, si =RETURN_ANSWERS_ENABLED= est activé : un JPEG
|
||||||
|
par réponse non vide, contenant toujours la réponse annotée
|
||||||
|
|
||||||
|
Le PDF n'est donc pas une conversion du JPEG. Voir la
|
||||||
|
[[file:docs/final_output.md][documentation des fichiers finaux]] pour les règles de sélection,
|
||||||
|
la pagination et les différences entre parcours.
|
||||||
|
|
||||||
|
=RETURN_JPEG_ENABLED= et =RETURN_PDF_ENABLED= dans =config.py=
|
||||||
|
permettent de désactiver ces sorties séparément (activées par défaut).
|
||||||
|
Relancer =giving-names= retire les fichiers nommés désactivés en
|
||||||
|
conservant leurs sources. =score.json= et =info.json= sont toujours inclus.
|
||||||
|
|
||||||
|
=RETURN_ANSWERS_ENABLED= est désactivé par défaut, activé dans la
|
||||||
|
configuration personnelle. =RETURN_ANSWERS_CONTEXT=,
|
||||||
|
=RETURN_ANSWERS_QUESTION= et =RETURN_ANSWERS_SOLUTION= choisissent les
|
||||||
|
documents ajoutés avant chaque réponse (seule la question est activée
|
||||||
|
par défaut). Recompiler les anciennes annotations une fois avant cet
|
||||||
|
export pour produire les images finales et leurs métadonnées.
|
||||||
|
|
||||||
Si un nom est =Unknown= : renommer à la main le dossier et le fichier dedans.
|
Si un nom est =Unknown= : renommer à la main le dossier et le fichier dedans.
|
||||||
4. Éventuellement, faire des modifications manuelles aux =score.json=.
|
4. Éventuellement, faire des modifications manuelles aux =score.json=.
|
||||||
@@ -328,12 +351,14 @@ conserve que :
|
|||||||
+ le résultat final =correction.json= ;
|
+ le résultat final =correction.json= ;
|
||||||
+ les journaux de =.copienator/logs= et les journaux placés à la
|
+ les journaux de =.copienator/logs= et les journaux placés à la
|
||||||
racine, comme =correction_log= ;
|
racine, comme =correction_log= ;
|
||||||
+ les images et fichiers =score.json= présents dans =A Rendre=.
|
+ les images (y compris =answers=), PDF et fichiers =score.json= et
|
||||||
|
=info.json= présents dans =A Rendre=.
|
||||||
|
|
||||||
Les liens symboliques conservés dans =A Rendre= sont remplacés par de
|
Les liens symboliques conservés dans =A Rendre= sont remplacés par de
|
||||||
véritables fichiers avant la suppression de leurs cibles. La commande
|
véritables fichiers avant la suppression de leurs cibles. La commande
|
||||||
refuse de démarrer si les copies traitées, =correction.json= ou une
|
refuse de démarrer si les copies traitées, =correction.json= ou une
|
||||||
image/un score d'élève sont absents. Elle affiche d'abord un résumé et
|
image/un score d'élève sont absents (l'image est facultative si
|
||||||
|
=RETURN_JPEG_ENABLED= vaut =False=). Elle affiche d'abord un résumé et
|
||||||
demande de saisir le nom de l'évaluation pour confirmer.
|
demande de saisir le nom de l'évaluation pour confirmer.
|
||||||
|
|
||||||
Dans le GUI, cette commande apparaît comme dernière étape facultative
|
Dans le GUI, cette commande apparaît comme dernière étape facultative
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ from pathlib import Path
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from .json_io import read_json
|
from .json_io import read_json
|
||||||
|
from .feedback_boxes import valid_feedback_box
|
||||||
|
|
||||||
Log = Callable[[str], None]
|
Log = Callable[[str], None]
|
||||||
|
|
||||||
@@ -25,8 +26,10 @@ def apply_checkbox_actions(
|
|||||||
continue
|
continue
|
||||||
result = labels_data[label]["result"]
|
result = labels_data[label]["result"]
|
||||||
feedbacks = result.get("feedback", [])
|
feedbacks = result.get("feedback", [])
|
||||||
global_feedbacks = [item for item in feedbacks if not item.get("box_2d")]
|
# Match the renderer's fallback for invalid boxes so checkbox indices
|
||||||
local_feedbacks = [item for item in feedbacks if item.get("box_2d")]
|
# still address the right comment when returned annotations are read.
|
||||||
|
global_feedbacks = [item for item in feedbacks if not valid_feedback_box(item.get("box_2d"))]
|
||||||
|
local_feedbacks = [item for item in feedbacks if valid_feedback_box(item.get("box_2d"))]
|
||||||
local_feedbacks.sort(key=lambda item: item["box_2d"][0])
|
local_feedbacks.sort(key=lambda item: item["box_2d"][0])
|
||||||
|
|
||||||
for action in label_actions:
|
for action in label_actions:
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ from typing import Any
|
|||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
from .json_io import read_json
|
from .json_io import read_json
|
||||||
|
from .feedback_boxes import valid_feedback_box
|
||||||
from .workspace import EvaluationWorkspace
|
from .workspace import EvaluationWorkspace
|
||||||
|
|
||||||
AnnotationData = dict[str, dict[str, dict[str, Any]]]
|
AnnotationData = dict[str, dict[str, dict[str, Any]]]
|
||||||
@@ -69,7 +70,7 @@ def _scaled_result(result: dict[str, Any], coordinates: GroupCoordinates | None)
|
|||||||
return scaled
|
return scaled
|
||||||
for feedback in scaled.get("feedback", []):
|
for feedback in scaled.get("feedback", []):
|
||||||
box = feedback.get("box_2d")
|
box = feedback.get("box_2d")
|
||||||
if not box or len(box) != 4:
|
if not box or not valid_feedback_box(box):
|
||||||
continue
|
continue
|
||||||
box[0] = int(box[0] * coordinates.height) // 1000
|
box[0] = int(box[0] * coordinates.height) // 1000
|
||||||
box[2] = int(box[2] * coordinates.height) // 1000
|
box[2] = int(box[2] * coordinates.height) // 1000
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
from collections.abc import Collection, Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def build_answer_info(
|
||||||
|
scores: Mapping[str, str],
|
||||||
|
present_labels: Collection[str],
|
||||||
|
rendered_labels: Collection[str],
|
||||||
|
touched: Mapping[str, bool] | None = None,
|
||||||
|
) -> dict[str, dict[str, Any]]:
|
||||||
|
"""Describe every question using the answers actually compiled for a copy."""
|
||||||
|
present = set(present_labels)
|
||||||
|
rendered = set(rendered_labels)
|
||||||
|
return {
|
||||||
|
label: {
|
||||||
|
"present": label in present,
|
||||||
|
"not_empty": label in rendered,
|
||||||
|
"touched": (touched or {}).get(label, False),
|
||||||
|
"score": score,
|
||||||
|
}
|
||||||
|
for label, score in scores.items()
|
||||||
|
}
|
||||||
@@ -7,11 +7,17 @@ from pathlib import Path
|
|||||||
import pandas as pd
|
import pandas as pd
|
||||||
from PIL import Image, ImageDraw, ImageFont
|
from PIL import Image, ImageDraw, ImageFont
|
||||||
|
|
||||||
from copienator.configuration import FINAL_SCORE_FONT_PATH, FINAL_SCORE_ODS_PATH, FINAL_SCORE_OUTPUT_DIR
|
from copienator.configuration import (
|
||||||
|
FINAL_SCORE_FONT_PATH,
|
||||||
|
FINAL_SCORE_HISTOGRAM_PATH,
|
||||||
|
FINAL_SCORE_ODS_PATH,
|
||||||
|
FINAL_SCORE_OUTPUT_DIR,
|
||||||
|
)
|
||||||
|
|
||||||
# Configuration constants
|
# Configuration constants
|
||||||
ODS_PATH = Path(FINAL_SCORE_ODS_PATH).expanduser()
|
ODS_PATH = Path(FINAL_SCORE_ODS_PATH).expanduser()
|
||||||
OUTPUT_DIR = Path(FINAL_SCORE_OUTPUT_DIR).expanduser()
|
OUTPUT_DIR = Path(FINAL_SCORE_OUTPUT_DIR).expanduser()
|
||||||
|
HISTOGRAM_PATH = Path(FINAL_SCORE_HISTOGRAM_PATH).expanduser()
|
||||||
|
|
||||||
|
|
||||||
def score_font(size):
|
def score_font(size):
|
||||||
@@ -33,6 +39,35 @@ def get_rounded_score(score):
|
|||||||
except (ValueError, TypeError):
|
except (ValueError, TypeError):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def copy_return_artifacts(source_dir: Path, destination_dir: Path) -> None:
|
||||||
|
"""Copy the metadata and optional individual answers for one student."""
|
||||||
|
for filename in ("score.json", "info.json"):
|
||||||
|
source = source_dir / filename
|
||||||
|
if source.is_file():
|
||||||
|
shutil.copy2(source, destination_dir / filename)
|
||||||
|
else:
|
||||||
|
print(f"Warning: Missing '{source}'.")
|
||||||
|
|
||||||
|
answers_source = source_dir / "answers"
|
||||||
|
answers_destination = destination_dir / "answers"
|
||||||
|
if answers_destination.is_symlink():
|
||||||
|
answers_destination.unlink()
|
||||||
|
elif answers_destination.is_dir():
|
||||||
|
shutil.rmtree(answers_destination)
|
||||||
|
if answers_source.is_dir():
|
||||||
|
shutil.copytree(answers_source, answers_destination)
|
||||||
|
|
||||||
|
|
||||||
|
def copy_histogram(output_dir: Path) -> None:
|
||||||
|
"""Copy the score histogram beside the per-student output folders."""
|
||||||
|
if not HISTOGRAM_PATH.is_file():
|
||||||
|
print(f"Warning: Missing histogram '{HISTOGRAM_PATH}'.")
|
||||||
|
return
|
||||||
|
destination = output_dir / "histogramme.pdf"
|
||||||
|
shutil.copy2(HISTOGRAM_PATH, destination)
|
||||||
|
print(f"Copied histogram: {destination}")
|
||||||
|
|
||||||
def process_images(base_dir, output_dir):
|
def process_images(base_dir, output_dir):
|
||||||
# 1. Load Data
|
# 1. Load Data
|
||||||
try:
|
try:
|
||||||
@@ -56,60 +91,64 @@ def process_images(base_dir, output_dir):
|
|||||||
print(f"Error: Directory '{search_path}' not found.")
|
print(f"Error: Directory '{search_path}' not found.")
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
for img_path in sorted(search_path.glob("*/*.jpg")):
|
for student_source in sorted(path for path in search_path.iterdir() if path.is_dir()):
|
||||||
student_name = img_path.stem # Filename without extension
|
image_paths = sorted(student_source.glob("*.jpg"))
|
||||||
|
pdf_paths = sorted(student_source.glob("*.pdf"))
|
||||||
|
media_paths = image_paths or pdf_paths
|
||||||
|
if not media_paths:
|
||||||
|
print(f"Error: No JPG or PDF found in '{student_source}'.")
|
||||||
|
continue
|
||||||
|
|
||||||
|
student_name = media_paths[0].stem
|
||||||
|
student_output = output_dir / student_name
|
||||||
|
student_output.mkdir(parents=True, exist_ok=True)
|
||||||
|
# Remove files produced by the former flat output layout when migrating
|
||||||
|
# an existing export directory.
|
||||||
|
for suffix in (".jpg", ".pdf"):
|
||||||
|
legacy_output = output_dir / f"{student_name}{suffix}"
|
||||||
|
if legacy_output.is_file() or legacy_output.is_symlink():
|
||||||
|
legacy_output.unlink()
|
||||||
|
copy_return_artifacts(student_source, student_output)
|
||||||
|
|
||||||
# 4. Find Score
|
# 4. Find Score
|
||||||
if student_name not in score_db:
|
if student_name not in score_db:
|
||||||
print(f"Error: Student '{student_name}' not found in ODS file.")
|
print(f"Error: Student '{student_name}' not found in ODS file.")
|
||||||
continue
|
else:
|
||||||
|
raw_score = score_db[student_name]
|
||||||
|
score = get_rounded_score(raw_score)
|
||||||
|
|
||||||
raw_score = score_db[student_name]
|
if score is None:
|
||||||
score = get_rounded_score(raw_score)
|
print(f"Error: Invalid score '{raw_score}' for '{student_name}'.")
|
||||||
|
else:
|
||||||
|
# 5. Process Images
|
||||||
|
for img_path in image_paths:
|
||||||
|
try:
|
||||||
|
with Image.open(img_path) as img:
|
||||||
|
img = img.convert("RGB")
|
||||||
|
draw = ImageDraw.Draw(img)
|
||||||
|
width, _height = img.size
|
||||||
|
|
||||||
if score is None:
|
font_size = int(width * 0.08)
|
||||||
print(f"Error: Invalid score '{raw_score}' for '{student_name}'.")
|
font = score_font(font_size)
|
||||||
continue
|
text = str(score)
|
||||||
|
|
||||||
# 5. Process Image
|
bbox = draw.textbbox((0, 0), text, font=font)
|
||||||
try:
|
text_w = bbox[2] - bbox[0]
|
||||||
with Image.open(img_path) as img:
|
|
||||||
img = img.convert("RGB")
|
|
||||||
draw = ImageDraw.Draw(img)
|
|
||||||
width, height = img.size
|
|
||||||
|
|
||||||
# Dynamic font size (15% of image height)
|
# 30px padding, top right.
|
||||||
font_size = int(width * 0.08)
|
x = width - text_w - 30
|
||||||
|
y = 30
|
||||||
|
draw.text((x, y), text, fill=(255, 0, 0), font=font)
|
||||||
|
|
||||||
font = score_font(font_size)
|
img.save(student_output / img_path.name)
|
||||||
|
print(f"Processed: {student_name} -> {score}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error processing image for '{student_name}': {e}")
|
||||||
|
|
||||||
text = str(score)
|
for pdf_path in pdf_paths:
|
||||||
|
shutil.copy2(pdf_path, student_output / pdf_path.name)
|
||||||
|
|
||||||
# Calculate text size and position (Top Right)
|
copy_histogram(output_dir)
|
||||||
bbox = draw.textbbox((0, 0), text, font=font)
|
|
||||||
text_w = bbox[2] - bbox[0]
|
|
||||||
text_h = bbox[3] - bbox[1]
|
|
||||||
|
|
||||||
# 30px padding
|
|
||||||
x = width - text_w - 30
|
|
||||||
y = 30
|
|
||||||
|
|
||||||
# Draw Text (Red)
|
|
||||||
draw.text((x, y), text, fill=(255, 0, 0), font=font)
|
|
||||||
|
|
||||||
# Save
|
|
||||||
save_path = output_dir / f"{student_name}.jpg"
|
|
||||||
img.save(save_path)
|
|
||||||
print(f"Processed: {student_name} -> {score}")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error processing image for '{student_name}': {e}")
|
|
||||||
|
|
||||||
for pdf_path in sorted(search_path.glob("*/*.pdf")):
|
|
||||||
student_name = pdf_path.stem # Filename without extension
|
|
||||||
save_path = output_dir / f"{student_name}.pdf"
|
|
||||||
|
|
||||||
shutil.copy(str(pdf_path), str(save_path))
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None):
|
def main(argv=None):
|
||||||
|
|||||||
@@ -30,7 +30,9 @@ from copienator import (
|
|||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
from copienator.annotation_data import load_annotation_data
|
from copienator.annotation_data import load_annotation_data
|
||||||
|
from copienator.answer_info import build_answer_info
|
||||||
from copienator.filesystem import staged_directory
|
from copienator.filesystem import staged_directory
|
||||||
|
from copienator.feedback_boxes import valid_feedback_box
|
||||||
from copienator.utils import natural_key
|
from copienator.utils import natural_key
|
||||||
|
|
||||||
MARGIN_LEFT = 300
|
MARGIN_LEFT = 300
|
||||||
@@ -325,6 +327,16 @@ def compose_label_image(base_img, label, result, hmin,
|
|||||||
|
|
||||||
# Filter deleted items (used by reading_annotations.py)
|
# Filter deleted items (used by reading_annotations.py)
|
||||||
feedbacks = [f for f in feedbacks if "to_delete" not in f]
|
feedbacks = [f for f in feedbacks if "to_delete" not in f]
|
||||||
|
# Never guess where an invalid rectangle belongs, or lose its comment.
|
||||||
|
# Use a copy so rendering cannot mutate saved correction data.
|
||||||
|
normalized = []
|
||||||
|
for feedback in feedbacks:
|
||||||
|
box = feedback.get("box_2d")
|
||||||
|
if box is not None and not valid_feedback_box(box):
|
||||||
|
print(f"Warning: Copie{with_id or ''} {label}: invalid feedback box {box!r}; displaying the comment without a rectangle.")
|
||||||
|
feedback = {**feedback, "box_2d": None}
|
||||||
|
normalized.append(feedback)
|
||||||
|
feedbacks = normalized
|
||||||
|
|
||||||
global_fb = [f for f in feedbacks if not f.get('box_2d')]
|
global_fb = [f for f in feedbacks if not f.get('box_2d')]
|
||||||
local_fb = [f for f in feedbacks if f.get('box_2d')]
|
local_fb = [f for f in feedbacks if f.get('box_2d')]
|
||||||
@@ -446,6 +458,7 @@ def process_student(student_id, labels_data, root_dir, all_labels, overwrite):
|
|||||||
with staged_directory(output_dir) as staging:
|
with staged_directory(output_dir) as staging:
|
||||||
d_notes = dict.fromkeys(all_labels, "")
|
d_notes = dict.fromkeys(all_labels, "")
|
||||||
label_images = []
|
label_images = []
|
||||||
|
answer_labels = []
|
||||||
sorted_labels = sorted(labels_data.items(), key=lambda item: natural_key(item[0]))
|
sorted_labels = sorted(labels_data.items(), key=lambda item: natural_key(item[0]))
|
||||||
|
|
||||||
for label, content in sorted_labels:
|
for label, content in sorted_labels:
|
||||||
@@ -475,8 +488,12 @@ def process_student(student_id, labels_data, root_dir, all_labels, overwrite):
|
|||||||
final_img.save(staging / f"{label}.jpg")
|
final_img.save(staging / f"{label}.jpg")
|
||||||
if result.get('error', "") != "empty-answer":
|
if result.get('error', "") != "empty-answer":
|
||||||
label_images.append(final_img)
|
label_images.append(final_img)
|
||||||
|
answer_labels.append(label)
|
||||||
|
|
||||||
atomic_write_json(staging / "score.json", d_notes)
|
atomic_write_json(staging / "score.json", d_notes)
|
||||||
|
atomic_write_json(staging / "info.json", build_answer_info(
|
||||||
|
d_notes, labels_data, answer_labels
|
||||||
|
))
|
||||||
if label_images:
|
if label_images:
|
||||||
max_w = max(image.width for image in label_images)
|
max_w = max(image.width for image in label_images)
|
||||||
total_h = sum(image.height for image in label_images)
|
total_h = sum(image.height for image in label_images)
|
||||||
|
|||||||
@@ -9,10 +9,13 @@ from google import genai
|
|||||||
from copienator import configuration as config
|
from copienator import configuration as config
|
||||||
from copienator import (
|
from copienator import (
|
||||||
CliError,
|
CliError,
|
||||||
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
atomic_write_bytes,
|
atomic_write_bytes,
|
||||||
execute,
|
execute,
|
||||||
|
read_json,
|
||||||
standard_parser,
|
standard_parser,
|
||||||
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -43,6 +46,41 @@ def list_jobs(*, client=None) -> ExitCode:
|
|||||||
return ExitCode.SUCCESS
|
return ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
|
def check_evaluation_jobs(workspace: EvaluationWorkspace, *, client=None) -> ExitCode:
|
||||||
|
"""Only report readiness after checking every job recorded for this evaluation."""
|
||||||
|
if not workspace.batch_jobs_file.is_file():
|
||||||
|
print("Impossible de vérifier les batchs : batch_jobs.json est absent.")
|
||||||
|
return ExitCode.PARTIAL
|
||||||
|
manifest = read_json(workspace.batch_jobs_file)
|
||||||
|
jobs = manifest.get("jobs") if isinstance(manifest, dict) else None
|
||||||
|
if not isinstance(jobs, dict):
|
||||||
|
raise CliError(f"Invalid batch manifest: {workspace.batch_jobs_file}")
|
||||||
|
if not jobs:
|
||||||
|
print("Aucun job enregistré pour cette évaluation.")
|
||||||
|
return ExitCode.PARTIAL
|
||||||
|
if any(not isinstance(entry, dict) or not isinstance(entry.get("name"), str)
|
||||||
|
or not entry["name"].strip() for entry in jobs.values()):
|
||||||
|
raise CliError(f"Invalid batch job in {workspace.batch_jobs_file}")
|
||||||
|
client = client or _client()
|
||||||
|
ready = True
|
||||||
|
for tier, entry in jobs.items():
|
||||||
|
job = client.batches.get(name=entry["name"])
|
||||||
|
state = job.state.name if hasattr(job.state, "name") else job.state
|
||||||
|
print(f"{tier} — {entry['name']}: {state}")
|
||||||
|
if state != "JOB_STATE_SUCCEEDED":
|
||||||
|
ready = False
|
||||||
|
if getattr(job, "error", None):
|
||||||
|
print(f" Erreur : {job.error}")
|
||||||
|
elif not getattr(getattr(job, "dest", None), "file_name", None):
|
||||||
|
ready = False
|
||||||
|
print(" Le fichier de résultats n’est pas encore disponible.")
|
||||||
|
if ready:
|
||||||
|
print("Tous les batchs de l’évaluation ont réussi. Les résultats sont prêts à récupérer.")
|
||||||
|
return ExitCode.SUCCESS
|
||||||
|
print("Les résultats ne sont pas tous prêts. Consultez à nouveau cette étape plus tard.")
|
||||||
|
return ExitCode.PARTIAL
|
||||||
|
|
||||||
|
|
||||||
def download_job(
|
def download_job(
|
||||||
job_name: str,
|
job_name: str,
|
||||||
*,
|
*,
|
||||||
@@ -73,6 +111,8 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
parser = standard_parser("List or download Gemini correction batch jobs")
|
parser = standard_parser("List or download Gemini correction batch jobs")
|
||||||
parser.add_argument("--download", metavar="JOB_NAME")
|
parser.add_argument("--download", metavar="JOB_NAME")
|
||||||
parser.add_argument("--output", type=Path, help="Downloaded JSONL destination")
|
parser.add_argument("--output", type=Path, help="Downloaded JSONL destination")
|
||||||
|
parser.add_argument("--evaluation", type=Path,
|
||||||
|
help="Check readiness of jobs recorded in this evaluation's batch_jobs.json")
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
@@ -80,6 +120,11 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
|
|
||||||
def handle(args: argparse.Namespace) -> ExitCode:
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
|
if args.evaluation is not None:
|
||||||
|
if args.download or args.output is not None:
|
||||||
|
raise CliError("--evaluation cannot be combined with --download or --output",
|
||||||
|
ExitCode.INVALID_ARGUMENTS)
|
||||||
|
return check_evaluation_jobs(workspace_from_args(args))
|
||||||
if args.output is not None and not args.download:
|
if args.output is not None and not args.download:
|
||||||
raise CliError("--output requires --download", ExitCode.INVALID_ARGUMENTS)
|
raise CliError("--output requires --download", ExitCode.INVALID_ARGUMENTS)
|
||||||
if args.download:
|
if args.download:
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ from copienator import (
|
|||||||
CliError,
|
CliError,
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
|
configuration,
|
||||||
evaluation_parser,
|
evaluation_parser,
|
||||||
execute,
|
execute,
|
||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
@@ -83,15 +84,18 @@ def _return_artifacts(workspace: EvaluationWorkspace) -> list[Path]:
|
|||||||
path for path in files if path.suffix.casefold() in IMAGE_SUFFIXES
|
path for path in files if path.suffix.casefold() in IMAGE_SUFFIXES
|
||||||
]
|
]
|
||||||
scores = [path for path in files if path.name.casefold() == "score.json"]
|
scores = [path for path in files if path.name.casefold() == "score.json"]
|
||||||
if not images or not scores:
|
pdfs = [path for path in files if path.suffix.casefold() == ".pdf"]
|
||||||
|
if (configuration.RETURN_JPEG_ENABLED and not images) or not scores:
|
||||||
missing = []
|
missing = []
|
||||||
if not images:
|
if configuration.RETURN_JPEG_ENABLED and not images:
|
||||||
missing.append("image")
|
missing.append("image")
|
||||||
if not scores:
|
if not scores:
|
||||||
missing.append("score.json")
|
missing.append("score.json")
|
||||||
incomplete.append(f"{directory.name} ({', '.join(missing)})")
|
incomplete.append(f"{directory.name} ({', '.join(missing)})")
|
||||||
artifacts.extend(images)
|
artifacts.extend(images)
|
||||||
artifacts.extend(scores)
|
artifacts.extend(scores)
|
||||||
|
artifacts.extend(pdfs)
|
||||||
|
artifacts.extend(path for path in files if path.name.casefold() == "info.json")
|
||||||
|
|
||||||
if incomplete:
|
if incomplete:
|
||||||
details = "\n".join(f" - {item}" for item in incomplete)
|
details = "\n".join(f" - {item}" for item in incomplete)
|
||||||
@@ -219,7 +223,7 @@ def print_plan(workspace: EvaluationWorkspace, plan: CleanupPlan, *, verbose: bo
|
|||||||
print(f" - {statement_count} textual statement files")
|
print(f" - {statement_count} textual statement files")
|
||||||
print(" - correction.json")
|
print(" - correction.json")
|
||||||
print(f" - {log_count} log files")
|
print(f" - {log_count} log files")
|
||||||
print(f" - {return_count} image/score artifacts in A Rendre")
|
print(f" - {return_count} image/PDF/score artifacts in A Rendre")
|
||||||
print(
|
print(
|
||||||
f"Will delete {len(plan.deleted_files)} files and "
|
f"Will delete {len(plan.deleted_files)} files and "
|
||||||
f"{len(plan.deleted_directories)} directories "
|
f"{len(plan.deleted_directories)} directories "
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import base64
|
|||||||
import concurrent.futures
|
import concurrent.futures
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import signal
|
||||||
import shlex
|
import shlex
|
||||||
import shutil
|
import shutil
|
||||||
import sys
|
import sys
|
||||||
@@ -18,6 +19,7 @@ from google import genai
|
|||||||
from copienator import configuration as config
|
from copienator import configuration as config
|
||||||
from copienator.commands import grouping
|
from copienator.commands import grouping
|
||||||
from copienator import prompting
|
from copienator import prompting
|
||||||
|
from copienator.feedback_boxes import valid_feedback_box
|
||||||
from copienator import (
|
from copienator import (
|
||||||
CliError,
|
CliError,
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
@@ -44,15 +46,22 @@ COPIES_DIR = Path()
|
|||||||
GROUPS_DIR = Path()
|
GROUPS_DIR = Path()
|
||||||
output_path = Path()
|
output_path = Path()
|
||||||
progress_path = Path()
|
progress_path = Path()
|
||||||
|
pending_responses_path = Path()
|
||||||
tasks: list[tuple] = []
|
tasks: list[tuple] = []
|
||||||
tasks_to_process: list[tuple] = []
|
tasks_to_process: list[tuple] = []
|
||||||
results: dict = {}
|
results: dict = {}
|
||||||
completed_tasks: list = []
|
completed_tasks: list = []
|
||||||
errors_summary: list = []
|
errors_summary: list = []
|
||||||
|
pending_responses: dict[str, str] = {}
|
||||||
overwrite = False
|
overwrite = False
|
||||||
limit = None
|
limit = None
|
||||||
client = None
|
client = None
|
||||||
start_time = 0.0
|
start_time = 0.0
|
||||||
|
stop_requested = threading.Event()
|
||||||
|
|
||||||
|
|
||||||
|
class CorrectionStopRequested(Exception):
|
||||||
|
"""Raised in a worker before it starts another Gemini request."""
|
||||||
|
|
||||||
# --- Thread-safe Logging ---
|
# --- Thread-safe Logging ---
|
||||||
log_lock = threading.Lock()
|
log_lock = threading.Lock()
|
||||||
@@ -81,9 +90,30 @@ def flush_thread_log(tid=None):
|
|||||||
f.write("\n".join(thread_logs[tid]) + "\n\n")
|
f.write("\n".join(thread_logs[tid]) + "\n\n")
|
||||||
thread_logs[tid].clear()
|
thread_logs[tid].clear()
|
||||||
|
|
||||||
|
|
||||||
|
def report_group_progress(completed: int, total: int) -> None:
|
||||||
|
"""Emit a stable, human-readable progress line for the GUI and CLI."""
|
||||||
|
print(
|
||||||
|
f"[Progression correction] Groupes traités : {completed}/{total}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def request_graceful_stop(_signum=None, _frame=None) -> None:
|
||||||
|
"""Stop scheduling Gemini calls while allowing in-flight calls to finish."""
|
||||||
|
if not stop_requested.is_set():
|
||||||
|
stop_requested.set()
|
||||||
|
print(
|
||||||
|
"\n[Interruption] Arrêt des nouveaux appels Gemini demandé. "
|
||||||
|
"Attente des appels en cours et sauvegarde de leurs résultats…",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
# --- Lock for thread-safe file writing ---
|
# --- Lock for thread-safe file writing ---
|
||||||
io_lock = threading.Lock()
|
io_lock = threading.Lock()
|
||||||
pro_lock = threading.Lock()
|
pro_lock = threading.Lock()
|
||||||
|
group_index_lock = threading.Lock()
|
||||||
|
reserved_group_indices: dict[str, int] = {}
|
||||||
pro_count = 0
|
pro_count = 0
|
||||||
flash_count = 0
|
flash_count = 0
|
||||||
pro_quota_exhausted = False
|
pro_quota_exhausted = False
|
||||||
@@ -137,6 +167,7 @@ def configure_runtime(
|
|||||||
api_client=None,
|
api_client=None,
|
||||||
) -> None:
|
) -> None:
|
||||||
global INPUT_DIR, COPIES_DIR, GROUPS_DIR, output_path, progress_path
|
global INPUT_DIR, COPIES_DIR, GROUPS_DIR, output_path, progress_path
|
||||||
|
global pending_responses_path, pending_responses
|
||||||
global tasks, tasks_to_process, results, completed_tasks, errors_summary
|
global tasks, tasks_to_process, results, completed_tasks, errors_summary
|
||||||
global overwrite, limit, client, start_time
|
global overwrite, limit, client, start_time
|
||||||
global pro_count, flash_count, pro_quota_exhausted
|
global pro_count, flash_count, pro_quota_exhausted
|
||||||
@@ -146,17 +177,21 @@ def configure_runtime(
|
|||||||
GROUPS_DIR = workspace.groups_dir
|
GROUPS_DIR = workspace.groups_dir
|
||||||
output_path = workspace.correction_file
|
output_path = workspace.correction_file
|
||||||
progress_path = workspace.correction_progress_file
|
progress_path = workspace.correction_progress_file
|
||||||
|
pending_responses_path = workspace.correction_pending_responses_file
|
||||||
tasks = list(discovered_tasks)
|
tasks = list(discovered_tasks)
|
||||||
overwrite = bool(args.overwrite)
|
overwrite = bool(args.overwrite)
|
||||||
limit = args.limit
|
limit = args.limit
|
||||||
start_time = time.time()
|
start_time = time.time()
|
||||||
errors_summary = []
|
errors_summary = []
|
||||||
|
pending_responses = {}
|
||||||
completed_tasks = []
|
completed_tasks = []
|
||||||
results = {label: [] for _file, label in tasks}
|
results = {label: [] for _file, label in tasks}
|
||||||
thread_logs.clear()
|
thread_logs.clear()
|
||||||
|
reserved_group_indices.clear()
|
||||||
pro_count = 0
|
pro_count = 0
|
||||||
flash_count = 0
|
flash_count = 0
|
||||||
pro_quota_exhausted = False
|
pro_quota_exhausted = False
|
||||||
|
stop_requested.clear()
|
||||||
|
|
||||||
if not overwrite:
|
if not overwrite:
|
||||||
if progress_path.is_file():
|
if progress_path.is_file():
|
||||||
@@ -170,6 +205,20 @@ def configure_runtime(
|
|||||||
raise TypeError("correction.json must contain a JSON object")
|
raise TypeError("correction.json must contain a JSON object")
|
||||||
results = loaded_results
|
results = loaded_results
|
||||||
|
|
||||||
|
# A response saved during a graceful interruption is not a completed
|
||||||
|
# correction. Resume its auxiliary checks even with --overwrite instead of
|
||||||
|
# paying for the same primary request again.
|
||||||
|
if pending_responses_path.is_file():
|
||||||
|
loaded_pending = read_json(pending_responses_path)
|
||||||
|
if not isinstance(loaded_pending, dict) or not all(
|
||||||
|
isinstance(key, str) and isinstance(value, str)
|
||||||
|
for key, value in loaded_pending.items()
|
||||||
|
):
|
||||||
|
raise TypeError(
|
||||||
|
"correction_pending_responses.json must contain a JSON object"
|
||||||
|
)
|
||||||
|
pending_responses = loaded_pending
|
||||||
|
|
||||||
completed_set = {(str(file_path), label) for file_path, label in completed_tasks}
|
completed_set = {(str(file_path), label) for file_path, label in completed_tasks}
|
||||||
tasks_to_process = [
|
tasks_to_process = [
|
||||||
task for task in tasks if (str(task[0]), task[1]) not in completed_set
|
task for task in tasks if (str(task[0]), task[1]) not in completed_set
|
||||||
@@ -180,7 +229,11 @@ def configure_runtime(
|
|||||||
def reset_workspace(workspace: EvaluationWorkspace) -> None:
|
def reset_workspace(workspace: EvaluationWorkspace) -> None:
|
||||||
"""Apply the explicitly requested correction reset."""
|
"""Apply the explicitly requested correction reset."""
|
||||||
print("--- Running Reset ---")
|
print("--- Running Reset ---")
|
||||||
for path in (workspace.correction_file, workspace.correction_progress_file):
|
for path in (
|
||||||
|
workspace.correction_file,
|
||||||
|
workspace.correction_progress_file,
|
||||||
|
workspace.correction_pending_responses_file,
|
||||||
|
):
|
||||||
if path.exists():
|
if path.exists():
|
||||||
path.unlink()
|
path.unlink()
|
||||||
print(f"Deleted: {path}")
|
print(f"Deleted: {path}")
|
||||||
@@ -209,6 +262,8 @@ def call_gemini_with_retries(model_id, contents, config,
|
|||||||
delays = [60, 300]
|
delays = [60, 300]
|
||||||
|
|
||||||
for attempt in range(3):
|
for attempt in range(3):
|
||||||
|
if stop_requested.is_set():
|
||||||
|
raise CorrectionStopRequested
|
||||||
# Switch to fallback immediately if quota was exhausted by another thread
|
# Switch to fallback immediately if quota was exhausted by another thread
|
||||||
if model_id == MODEL_ID_pro and pro_quota_exhausted and fallback_model_id:
|
if model_id == MODEL_ID_pro and pro_quota_exhausted and fallback_model_id:
|
||||||
model_id = fallback_model_id
|
model_id = fallback_model_id
|
||||||
@@ -224,6 +279,8 @@ def call_gemini_with_retries(model_id, contents, config,
|
|||||||
full_response_text += chunk.text
|
full_response_text += chunk.text
|
||||||
return full_response_text
|
return full_response_text
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
if stop_requested.is_set():
|
||||||
|
raise CorrectionStopRequested from e
|
||||||
error_msg = str(e).lower()
|
error_msg = str(e).lower()
|
||||||
is_quota_error = "429" in error_msg or "quota" in error_msg or "exhausted" in error_msg
|
is_quota_error = "429" in error_msg or "quota" in error_msg or "exhausted" in error_msg
|
||||||
is_minute_limit = "minute" in error_msg or "rpm" in error_msg or "tpm" in error_msg
|
is_minute_limit = "minute" in error_msg or "rpm" in error_msg or "tpm" in error_msg
|
||||||
@@ -235,7 +292,8 @@ def call_gemini_with_retries(model_id, contents, config,
|
|||||||
wait_time = float(retry_match.group(1)) + 1.0 if retry_match else delays[attempt]
|
wait_time = float(retry_match.group(1)) + 1.0 if retry_match else delays[attempt]
|
||||||
|
|
||||||
tprint(f"\tGemini Pro minute limit hit. Waiting {wait_time:.1f}s...")
|
tprint(f"\tGemini Pro minute limit hit. Waiting {wait_time:.1f}s...")
|
||||||
time.sleep(wait_time)
|
if stop_requested.wait(wait_time):
|
||||||
|
raise CorrectionStopRequested
|
||||||
continue # Retry same model
|
continue # Retry same model
|
||||||
|
|
||||||
# Immediately fallback to Flash without waiting if it's a Pro quota error
|
# Immediately fallback to Flash without waiting if it's a Pro quota error
|
||||||
@@ -247,7 +305,8 @@ def call_gemini_with_retries(model_id, contents, config,
|
|||||||
|
|
||||||
if attempt < 2:
|
if attempt < 2:
|
||||||
tprint(f"\tGemini API failure: {e}. Retrying in {delays[attempt]} seconds...")
|
tprint(f"\tGemini API failure: {e}. Retrying in {delays[attempt]} seconds...")
|
||||||
time.sleep(delays[attempt])
|
if stop_requested.wait(delays[attempt]):
|
||||||
|
raise CorrectionStopRequested
|
||||||
else:
|
else:
|
||||||
tprint(f"\tGemini API failure: {e}. Maximum retries reached.")
|
tprint(f"\tGemini API failure: {e}. Maximum retries reached.")
|
||||||
raise
|
raise
|
||||||
@@ -267,6 +326,9 @@ def correct_boxes_with_gemini(pid, label, pdf_path, original_feedbacks,
|
|||||||
for f in corrected_feedbacks:
|
for f in corrected_feedbacks:
|
||||||
b = f.get("box_2d")
|
b = f.get("box_2d")
|
||||||
if b:
|
if b:
|
||||||
|
if not valid_feedback_box(b) or any(value < 0 or value > 1000 for value in b):
|
||||||
|
f["box_2d"] = None
|
||||||
|
continue
|
||||||
ymin_s, xmin_s, ymax_s, xmax_s = b
|
ymin_s, xmin_s, ymax_s, xmax_s = b
|
||||||
|
|
||||||
# Y mapping: Add the group Y-offset (yming), then normalize to total_height
|
# Y mapping: Add the group Y-offset (yming), then normalize to total_height
|
||||||
@@ -290,6 +352,16 @@ def get_next_group_idx(label):
|
|||||||
if not existing: return 0
|
if not existing: return 0
|
||||||
return max([int(f.stem.split("_")[1]) for f in existing])
|
return max([int(f.stem.split("_")[1]) for f in existing])
|
||||||
|
|
||||||
|
|
||||||
|
def reserve_next_group_idx(label: str) -> int:
|
||||||
|
"""Reserve a unique zero-based group index for this correction run."""
|
||||||
|
with group_index_lock:
|
||||||
|
if label not in reserved_group_indices:
|
||||||
|
reserved_group_indices[label] = get_next_group_idx(label)
|
||||||
|
idx = reserved_group_indices[label]
|
||||||
|
reserved_group_indices[label] = idx + 1
|
||||||
|
return idx
|
||||||
|
|
||||||
def handle_label_errors(pid, label, res, pdf_path):
|
def handle_label_errors(pid, label, res, pdf_path):
|
||||||
"""Handles Gemini labeling errors, moves/copies files, and returns new tasks."""
|
"""Handles Gemini labeling errors, moves/copies files, and returns new tasks."""
|
||||||
new_tasks = []
|
new_tasks = []
|
||||||
@@ -330,7 +402,7 @@ def handle_label_errors(pid, label, res, pdf_path):
|
|||||||
if pdf_path != old_pdf_path:
|
if pdf_path != old_pdf_path:
|
||||||
shutil.move(str(pdf_path), str(old_pdf_path))
|
shutil.move(str(pdf_path), str(old_pdf_path))
|
||||||
|
|
||||||
idx = get_next_group_idx(new_label)
|
idx = reserve_next_group_idx(new_label)
|
||||||
height = grouping.get_pdf_height(str(new_pdf_path))
|
height = grouping.get_pdf_height(str(new_pdf_path))
|
||||||
grouping.create_jpg(new_label, idx, [(pid, str(new_pdf_path), height)], GROUPS_DIR)
|
grouping.create_jpg(new_label, idx, [(pid, str(new_pdf_path), height)], GROUPS_DIR)
|
||||||
tprint(f"\t\tMaking {new_label} group {idx+1}")
|
tprint(f"\t\tMaking {new_label} group {idx+1}")
|
||||||
@@ -342,6 +414,8 @@ def handle_label_errors(pid, label, res, pdf_path):
|
|||||||
tprint(f"\tHandling additional-answer for {pid} {label}")
|
tprint(f"\tHandling additional-answer for {pid} {label}")
|
||||||
try:
|
try:
|
||||||
add_labels = json.loads(call_gemini_with_retries(MODEL_ID_flash, contents, config))
|
add_labels = json.loads(call_gemini_with_retries(MODEL_ID_flash, contents, config))
|
||||||
|
except CorrectionStopRequested:
|
||||||
|
raise
|
||||||
except Exception: # noqa: BLE001 - invalid auxiliary model response
|
except Exception: # noqa: BLE001 - invalid auxiliary model response
|
||||||
add_labels = []
|
add_labels = []
|
||||||
|
|
||||||
@@ -362,7 +436,7 @@ def handle_label_errors(pid, label, res, pdf_path):
|
|||||||
if not base_add_pdf_path.exists() and not add_pdf_path.exists():
|
if not base_add_pdf_path.exists() and not add_pdf_path.exists():
|
||||||
shutil.copy(str(pdf_path), str(add_pdf_path))
|
shutil.copy(str(pdf_path), str(add_pdf_path))
|
||||||
tprint(f"\t\tCopying Copie{pid} : {label} -> {add_label}")
|
tprint(f"\t\tCopying Copie{pid} : {label} -> {add_label}")
|
||||||
idx = get_next_group_idx(add_label)
|
idx = reserve_next_group_idx(add_label)
|
||||||
tprint(f"\t\tMaking {add_label} group {idx+1}")
|
tprint(f"\t\tMaking {add_label} group {idx+1}")
|
||||||
height = grouping.get_pdf_height(str(add_pdf_path))
|
height = grouping.get_pdf_height(str(add_pdf_path))
|
||||||
grouping.create_jpg(add_label, idx, [(pid, str(add_pdf_path), height)], GROUPS_DIR)
|
grouping.create_jpg(add_label, idx, [(pid, str(add_pdf_path), height)], GROUPS_DIR)
|
||||||
@@ -400,6 +474,9 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
total_height = group_data[-1][2]
|
total_height = group_data[-1][2]
|
||||||
use_flash = n >= 4 or total_height <= 500
|
use_flash = n >= 4 or total_height <= 500
|
||||||
|
|
||||||
|
if precomputed_response is None and stop_requested.is_set():
|
||||||
|
raise CorrectionStopRequested
|
||||||
|
|
||||||
# Only apply limits and counts if we are making a live call
|
# Only apply limits and counts if we are making a live call
|
||||||
if precomputed_response is None:
|
if precomputed_response is None:
|
||||||
if not use_flash:
|
if not use_flash:
|
||||||
@@ -420,9 +497,11 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
model_to_use = MODEL_ID_flash if use_flash else MODEL_ID_pro
|
model_to_use = MODEL_ID_flash if use_flash else MODEL_ID_pro
|
||||||
|
|
||||||
if precomputed_response:
|
if precomputed_response:
|
||||||
tprint(f"Using batched response for: {label} {group_name}")
|
tprint(f"Using saved response for: {label} {group_name}")
|
||||||
full_response_text = precomputed_response
|
full_response_text = precomputed_response
|
||||||
else:
|
else:
|
||||||
|
if stop_requested.is_set():
|
||||||
|
raise CorrectionStopRequested
|
||||||
tprint(f"Asking Gemini {'Flash' if use_flash else 'Pro '}: {label} {group_name}")
|
tprint(f"Asking Gemini {'Flash' if use_flash else 'Pro '}: {label} {group_name}")
|
||||||
full_response_text = call_gemini_with_retries(model_to_use, contents, config)
|
full_response_text = call_gemini_with_retries(model_to_use, contents, config)
|
||||||
|
|
||||||
@@ -461,8 +540,28 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
if res["error"] != "":
|
if res["error"] != "":
|
||||||
tprint("\tError :", res["error"], "for Copie", pid, group_name)
|
tprint("\tError :", res["error"], "for Copie", pid, group_name)
|
||||||
|
|
||||||
if can_spawn_tasks and res.get("error") in ["wrong-label", "additional-answer"]:
|
if can_spawn_tasks and res.get("error") in [
|
||||||
new_tasks.extend(handle_label_errors(pid, label, res, pdf_path))
|
"wrong-label",
|
||||||
|
"additional-answer",
|
||||||
|
]:
|
||||||
|
if stop_requested.is_set():
|
||||||
|
with io_lock:
|
||||||
|
pending_responses[file_path] = json.dumps(json_data)
|
||||||
|
atomic_write_json(
|
||||||
|
pending_responses_path, pending_responses
|
||||||
|
)
|
||||||
|
raise CorrectionStopRequested
|
||||||
|
try:
|
||||||
|
new_tasks.extend(
|
||||||
|
handle_label_errors(pid, label, res, pdf_path)
|
||||||
|
)
|
||||||
|
except CorrectionStopRequested:
|
||||||
|
with io_lock:
|
||||||
|
pending_responses[file_path] = json.dumps(json_data)
|
||||||
|
atomic_write_json(
|
||||||
|
pending_responses_path, pending_responses
|
||||||
|
)
|
||||||
|
raise
|
||||||
# Si "wrong-label" a déplacé le fichier courant vers _old
|
# Si "wrong-label" a déplacé le fichier courant vers _old
|
||||||
if res.get("error", "").startswith("wrg-lbl-moved-to:"):
|
if res.get("error", "").startswith("wrg-lbl-moved-to:"):
|
||||||
current_suffix = "_old"
|
current_suffix = "_old"
|
||||||
@@ -475,6 +574,9 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
for (i,f) in enumerate(res["feedback"]):
|
for (i,f) in enumerate(res["feedback"]):
|
||||||
b = f.get("box_2d")
|
b = f.get("box_2d")
|
||||||
if b:
|
if b:
|
||||||
|
if not valid_feedback_box(b):
|
||||||
|
needs_correction.append(i)
|
||||||
|
continue
|
||||||
ymin, _xmin, ymax, xmax = b
|
ymin, _xmin, ymax, xmax = b
|
||||||
ymin = ymin * total_height // 1000
|
ymin = ymin * total_height // 1000
|
||||||
ymax = ymax * total_height // 1000
|
ymax = ymax * total_height // 1000
|
||||||
@@ -484,9 +586,11 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
pid, label, group_name)
|
pid, label, group_name)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if (ymin < yming - 50 or ymax > ymaxg + 50 or xmax / 1000 > width_r):
|
if (ymin < yming - 50 or ymax > ymaxg + 50
|
||||||
|
or ymin > ymaxg + 50 or ymax < yming - 50
|
||||||
|
or _xmin < 0 or xmax / 1000 > width_r):
|
||||||
needs_correction.append(i)
|
needs_correction.append(i)
|
||||||
break
|
continue
|
||||||
if ymin < yming - 5:
|
if ymin < yming - 5:
|
||||||
ymin = yming - 5
|
ymin = yming - 5
|
||||||
b[0] = ymin * 1000 // total_height
|
b[0] = ymin * 1000 // total_height
|
||||||
@@ -498,6 +602,8 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
if needs_correction:
|
if needs_correction:
|
||||||
tprint(f"\tBox anomalies detected for Copie {pid} {group_name}. \n\tRequesting isolated correction from Gemini Flash...")
|
tprint(f"\tBox anomalies detected for Copie {pid} {group_name}. \n\tRequesting isolated correction from Gemini Flash...")
|
||||||
try:
|
try:
|
||||||
|
if stop_requested.is_set():
|
||||||
|
raise CorrectionStopRequested
|
||||||
# Pensez à passer pdf_path à la fonction modifiée !
|
# Pensez à passer pdf_path à la fonction modifiée !
|
||||||
res["feedback"] = correct_boxes_with_gemini(
|
res["feedback"] = correct_boxes_with_gemini(
|
||||||
pid, label, pdf_path, res["feedback"],
|
pid, label, pdf_path, res["feedback"],
|
||||||
@@ -520,7 +626,12 @@ def process_single_task(task_tuple, precomputed_response=None):
|
|||||||
# To track progress
|
# To track progress
|
||||||
completed_tasks.append((file_path, label))
|
completed_tasks.append((file_path, label))
|
||||||
atomic_write_json(progress_path, completed_tasks)
|
atomic_write_json(progress_path, completed_tasks)
|
||||||
|
if file_path in pending_responses:
|
||||||
|
del pending_responses[file_path]
|
||||||
|
atomic_write_json(pending_responses_path, pending_responses)
|
||||||
|
|
||||||
|
except CorrectionStopRequested:
|
||||||
|
raise
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
tprint(f"Error decoding JSON for {file_path}", file=sys.stderr)
|
tprint(f"Error decoding JSON for {file_path}", file=sys.stderr)
|
||||||
with io_lock:
|
with io_lock:
|
||||||
@@ -575,7 +686,7 @@ def resolve_delayed_moves():
|
|||||||
if pdf_path != old_pdf_path:
|
if pdf_path != old_pdf_path:
|
||||||
shutil.move(str(pdf_path), str(old_pdf_path))
|
shutil.move(str(pdf_path), str(old_pdf_path))
|
||||||
|
|
||||||
idx = get_next_group_idx(target_label)
|
idx = reserve_next_group_idx(target_label)
|
||||||
height = grouping.get_pdf_height(str(new_pdf_path))
|
height = grouping.get_pdf_height(str(new_pdf_path))
|
||||||
grouping.create_jpg(target_label, idx, [(pid, str(new_pdf_path), height)], GROUPS_DIR)
|
grouping.create_jpg(target_label, idx, [(pid, str(new_pdf_path), height)], GROUPS_DIR)
|
||||||
new_tasks.append((str(GROUPS_DIR / target_label / f"Group_{idx+1}.jpg"), target_label, False))
|
new_tasks.append((str(GROUPS_DIR / target_label / f"Group_{idx+1}.jpg"), target_label, False))
|
||||||
@@ -593,7 +704,7 @@ def resolve_delayed_moves():
|
|||||||
resolved_any = True
|
resolved_any = True
|
||||||
|
|
||||||
shutil.copy(str(pdf_path), str(add_pdf_path))
|
shutil.copy(str(pdf_path), str(add_pdf_path))
|
||||||
idx = get_next_group_idx(target_label)
|
idx = reserve_next_group_idx(target_label)
|
||||||
height = grouping.get_pdf_height(str(add_pdf_path))
|
height = grouping.get_pdf_height(str(add_pdf_path))
|
||||||
grouping.create_jpg(target_label, idx, [(pid, str(add_pdf_path), height)], GROUPS_DIR)
|
grouping.create_jpg(target_label, idx, [(pid, str(add_pdf_path), height)], GROUPS_DIR)
|
||||||
new_tasks.append((str(GROUPS_DIR / target_label / f"Group_{idx+1}.jpg"), target_label, False))
|
new_tasks.append((str(GROUPS_DIR / target_label / f"Group_{idx+1}.jpg"), target_label, False))
|
||||||
@@ -679,7 +790,7 @@ def run_configured(args: argparse.Namespace) -> ExitCode:
|
|||||||
# pdf_path = copie_dir / f"{label}_old.pdf"
|
# pdf_path = copie_dir / f"{label}_old.pdf"
|
||||||
|
|
||||||
if pdf_path.exists():
|
if pdf_path.exists():
|
||||||
idx = get_next_group_idx(label)
|
idx = reserve_next_group_idx(label)
|
||||||
height = grouping.get_pdf_height(str(pdf_path))
|
height = grouping.get_pdf_height(str(pdf_path))
|
||||||
grouping.create_jpg(label, idx, [(pid, str(pdf_path), height)], GROUPS_DIR)
|
grouping.create_jpg(label, idx, [(pid, str(pdf_path), height)], GROUPS_DIR)
|
||||||
new_group_path = str(GROUPS_DIR / label / f"Group_{idx+1}.jpg")
|
new_group_path = str(GROUPS_DIR / label / f"Group_{idx+1}.jpg")
|
||||||
@@ -814,36 +925,82 @@ def run_configured(args: argparse.Namespace) -> ExitCode:
|
|||||||
else:
|
else:
|
||||||
print(f"Warning: Batch results file {batch_results_path} not found.", file=sys.stderr)
|
print(f"Warning: Batch results file {batch_results_path} not found.", file=sys.stderr)
|
||||||
|
|
||||||
|
report_live_progress = not any(
|
||||||
|
(args.batch, args.batch_from, args.deal_with_batched, args.refaire)
|
||||||
|
)
|
||||||
|
progress_total = len(tasks)
|
||||||
|
progress_completed = max(0, progress_total - len(tasks_to_process))
|
||||||
|
if report_live_progress:
|
||||||
|
report_group_progress(progress_completed, progress_total)
|
||||||
|
|
||||||
made_progress = True
|
made_progress = True
|
||||||
while tasks_to_process or made_progress:
|
while tasks_to_process or made_progress:
|
||||||
if tasks_to_process:
|
if tasks_to_process:
|
||||||
print(f"Starting processing on {len(tasks_to_process)} tasks with {NB_THREADS} threads...")
|
print(f"Starting processing on {len(tasks_to_process)} tasks with {NB_THREADS} threads...")
|
||||||
with concurrent.futures.ThreadPoolExecutor(max_workers=NB_THREADS) as executor:
|
with concurrent.futures.ThreadPoolExecutor(max_workers=NB_THREADS) as executor:
|
||||||
|
waiting_tasks = list(tasks_to_process)
|
||||||
futures = {}
|
futures = {}
|
||||||
for task in tasks_to_process:
|
|
||||||
file_path = task[0]
|
|
||||||
precomp = batched_responses.get(file_path)
|
|
||||||
futures[executor.submit(process_single_task, task, precomp)] = task
|
|
||||||
|
|
||||||
for future in concurrent.futures.as_completed(futures):
|
def submit_available_tasks() -> None:
|
||||||
try:
|
while (
|
||||||
new_generated_tasks = future.result()
|
waiting_tasks
|
||||||
if new_generated_tasks:
|
and len(futures) < NB_THREADS
|
||||||
for new_task in new_generated_tasks:
|
and not stop_requested.is_set()
|
||||||
futures[executor.submit(process_single_task, new_task)] = new_task
|
):
|
||||||
except Exception as e: # noqa: BLE001 - future boundary
|
task = waiting_tasks.pop(0)
|
||||||
print(f"Exception during task execution: {e}", file=sys.stderr)
|
file_path = task[0]
|
||||||
failed_task = futures[future]
|
precomp = pending_responses.get(
|
||||||
with io_lock:
|
file_path, batched_responses.get(file_path)
|
||||||
errors_summary.append((str(e), failed_task[0]))
|
)
|
||||||
|
futures[
|
||||||
|
executor.submit(process_single_task, task, precomp)
|
||||||
|
] = task
|
||||||
|
|
||||||
|
submit_available_tasks()
|
||||||
|
while futures:
|
||||||
|
completed_futures, _pending = concurrent.futures.wait(
|
||||||
|
tuple(futures),
|
||||||
|
return_when=concurrent.futures.FIRST_COMPLETED,
|
||||||
|
)
|
||||||
|
for future in completed_futures:
|
||||||
|
failed_task = futures.pop(future)
|
||||||
|
task_completed = False
|
||||||
|
try:
|
||||||
|
new_generated_tasks = future.result()
|
||||||
|
task_completed = True
|
||||||
|
if new_generated_tasks and not stop_requested.is_set():
|
||||||
|
if report_live_progress:
|
||||||
|
progress_total += len(new_generated_tasks)
|
||||||
|
waiting_tasks.extend(new_generated_tasks)
|
||||||
|
except CorrectionStopRequested:
|
||||||
|
pass
|
||||||
|
except Exception as e: # noqa: BLE001 - future boundary
|
||||||
|
print(
|
||||||
|
f"Exception during task execution: {e}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
with io_lock:
|
||||||
|
errors_summary.append((str(e), failed_task[0]))
|
||||||
|
if report_live_progress and task_completed:
|
||||||
|
progress_completed += 1
|
||||||
|
report_group_progress(
|
||||||
|
progress_completed, progress_total
|
||||||
|
)
|
||||||
|
submit_available_tasks()
|
||||||
|
|
||||||
tasks_to_process = [] # Vider la liste une fois traitée
|
tasks_to_process = [] # Vider la liste une fois traitée
|
||||||
|
|
||||||
# Après avoir traité toutes les tâches actuelles (live ou batched),
|
# Après avoir traité toutes les tâches actuelles (live ou batched),
|
||||||
# on tente de débloquer les mouvements qui étaient en attente
|
# on tente de débloquer les mouvements qui étaient en attente
|
||||||
|
if stop_requested.is_set():
|
||||||
|
break
|
||||||
|
|
||||||
delayed_tasks = resolve_delayed_moves()
|
delayed_tasks = resolve_delayed_moves()
|
||||||
if delayed_tasks:
|
if delayed_tasks:
|
||||||
print(f"Resolved {len(delayed_tasks)} delayed moves! Running executor for new tasks...")
|
print(f"Resolved {len(delayed_tasks)} delayed moves! Running executor for new tasks...")
|
||||||
|
if report_live_progress:
|
||||||
|
progress_total += len(delayed_tasks)
|
||||||
|
report_group_progress(progress_completed, progress_total)
|
||||||
tasks_to_process.extend(delayed_tasks)
|
tasks_to_process.extend(delayed_tasks)
|
||||||
made_progress = True
|
made_progress = True
|
||||||
else:
|
else:
|
||||||
@@ -852,7 +1009,7 @@ def run_configured(args: argparse.Namespace) -> ExitCode:
|
|||||||
# Check for remaining unresolved delayed tasks
|
# Check for remaining unresolved delayed tasks
|
||||||
unresolved_delayed = []
|
unresolved_delayed = []
|
||||||
with io_lock:
|
with io_lock:
|
||||||
for label, batches in results.items():
|
for label, batches in sorted(results.items()):
|
||||||
for batch in batches:
|
for batch in batches:
|
||||||
for p in batch:
|
for p in batch:
|
||||||
res = p.get("result", {})
|
res = p.get("result", {})
|
||||||
@@ -869,7 +1026,7 @@ def run_configured(args: argparse.Namespace) -> ExitCode:
|
|||||||
manual_path = INPUT_DIR / "manual_resolutions.txt"
|
manual_path = INPUT_DIR / "manual_resolutions.txt"
|
||||||
atomic_write_text(
|
atomic_write_text(
|
||||||
manual_path,
|
manual_path,
|
||||||
"### Use -> x>, -x, ss, sx, xx, xs\n"
|
"### Use -> x>, -x, ss, sx, xx, xs, c{43}1>, c{43}2x\n"
|
||||||
+ "\n".join(unresolved_delayed)
|
+ "\n".join(unresolved_delayed)
|
||||||
+ "\n",
|
+ "\n",
|
||||||
)
|
)
|
||||||
@@ -885,6 +1042,12 @@ def run_configured(args: argparse.Namespace) -> ExitCode:
|
|||||||
print(err, file=sys.stderr)
|
print(err, file=sys.stderr)
|
||||||
escaped_path = shlex.quote(str(file))
|
escaped_path = shlex.quote(str(file))
|
||||||
print(f"Run : python -m copienator correct {escaped_path}")
|
print(f"Run : python -m copienator correct {escaped_path}")
|
||||||
|
if stop_requested.is_set():
|
||||||
|
print(
|
||||||
|
"[Interruption] Appels en cours terminés ; résultats disponibles sauvegardés.",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
return ExitCode.INTERRUPTED
|
||||||
return ExitCode.PARTIAL if errors_summary else ExitCode.SUCCESS
|
return ExitCode.PARTIAL if errors_summary else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
@@ -909,9 +1072,17 @@ def run(
|
|||||||
configure_runtime(workspace, discovered, args, api_client=api_client)
|
configure_runtime(workspace, discovered, args, api_client=api_client)
|
||||||
if not discovered and not args.refaire:
|
if not discovered and not args.refaire:
|
||||||
return ExitCode.PARTIAL
|
return ExitCode.PARTIAL
|
||||||
|
previous_handlers = {}
|
||||||
|
for signal_name in ("SIGINT", "SIGBREAK"):
|
||||||
|
interrupt_signal = getattr(signal, signal_name, None)
|
||||||
|
if interrupt_signal is not None:
|
||||||
|
previous_handlers[interrupt_signal] = signal.getsignal(interrupt_signal)
|
||||||
|
signal.signal(interrupt_signal, request_graceful_stop)
|
||||||
try:
|
try:
|
||||||
status = run_configured(args)
|
status = run_configured(args)
|
||||||
finally:
|
finally:
|
||||||
|
for interrupt_signal, previous_handler in previous_handlers.items():
|
||||||
|
signal.signal(interrupt_signal, previous_handler)
|
||||||
for thread_id in list(thread_logs):
|
for thread_id in list(thread_logs):
|
||||||
flush_thread_log(thread_id)
|
flush_thread_log(thread_id)
|
||||||
if warnings and status == ExitCode.SUCCESS:
|
if warnings and status == ExitCode.SUCCESS:
|
||||||
@@ -961,6 +1132,12 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
|
|
||||||
def handle(args: argparse.Namespace) -> ExitCode:
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
|
if args.refaire and args.overwrite:
|
||||||
|
raise CliError(
|
||||||
|
"--overwrite cannot be used with --refaire; --refaire already "
|
||||||
|
"replaces the corrections selected in refaire.json",
|
||||||
|
ExitCode.INVALID_ARGUMENTS,
|
||||||
|
)
|
||||||
workspace, target = workspace_from_target(args)
|
workspace, target = workspace_from_target(args)
|
||||||
targets = [target]
|
targets = [target]
|
||||||
for additional in args.additional_targets:
|
for additional in args.additional_targets:
|
||||||
|
|||||||
@@ -0,0 +1,226 @@
|
|||||||
|
"""Optionally replace split-answer PDFs with large bottom crops."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import multiprocessing
|
||||||
|
import shutil
|
||||||
|
import signal
|
||||||
|
import tempfile
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
|
||||||
|
from copienator.cli import CliError, ExitCode, execute, target_parser, workspace_from_target
|
||||||
|
from copienator.crop_exercise_bottoms import _full_page_height, process_exercise_pdf
|
||||||
|
from copienator.workspace import EvaluationWorkspace
|
||||||
|
|
||||||
|
|
||||||
|
def selected_files(workspace: EvaluationWorkspace, target: Path) -> list[Path]:
|
||||||
|
workspace.require_directories("Copies")
|
||||||
|
resolved = target.resolve()
|
||||||
|
if resolved not in {workspace.root.resolve(), workspace.copies_dir.resolve()}:
|
||||||
|
raise CliError(
|
||||||
|
"La cible doit être l’évaluation ou son dossier Copies.",
|
||||||
|
ExitCode.INVALID_ARGUMENTS,
|
||||||
|
)
|
||||||
|
files = sorted(
|
||||||
|
workspace.copies_dir.glob("Copie*/*.pdf"),
|
||||||
|
key=lambda path: (path.parent.name.casefold(), path.name.casefold()),
|
||||||
|
)
|
||||||
|
for source in files:
|
||||||
|
if source.is_symlink():
|
||||||
|
raise CliError(f"Lien symbolique non pris en charge : {source}")
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
def _initialize_worker() -> None:
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
||||||
|
|
||||||
|
|
||||||
|
def _process_file(job: tuple[Path, Path, float]) -> list[dict]:
|
||||||
|
source, destination, full_height = job
|
||||||
|
records = process_exercise_pdf(
|
||||||
|
source, destination, None, full_height, dpi=200, padding_mm=6
|
||||||
|
)
|
||||||
|
changed = sum(record["status"] == "cropped" for record in records)
|
||||||
|
if changed:
|
||||||
|
print(
|
||||||
|
f"{source.parent.name}/{source.name} : {changed}/{len(records)} page(s) rognée(s)",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def process_files(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
files: list[Path],
|
||||||
|
staging: Path,
|
||||||
|
workers: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
heights: dict[str, float] = {}
|
||||||
|
for copy_name in sorted({source.parent.name for source in files}):
|
||||||
|
copy_pdf = workspace.copies_dir / f"{copy_name}.pdf"
|
||||||
|
if not copy_pdf.is_file():
|
||||||
|
raise CliError(f"PDF source introuvable : {copy_pdf}")
|
||||||
|
heights[copy_name] = _full_page_height(copy_pdf)
|
||||||
|
jobs = [
|
||||||
|
(
|
||||||
|
source,
|
||||||
|
staging / source.relative_to(workspace.copies_dir),
|
||||||
|
heights[source.parent.name],
|
||||||
|
)
|
||||||
|
for source in files
|
||||||
|
]
|
||||||
|
count = min(workers, len(jobs))
|
||||||
|
print(
|
||||||
|
f"Analyse de {len(files)} PDF de réponses avec {count} traitement(s) en parallèle.",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
if count == 1:
|
||||||
|
previous_threads = cv2.getNumThreads()
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
try:
|
||||||
|
batches = [_process_file(job) for job in jobs]
|
||||||
|
finally:
|
||||||
|
cv2.setNumThreads(previous_threads)
|
||||||
|
else:
|
||||||
|
with multiprocessing.get_context("spawn").Pool(
|
||||||
|
count, _initialize_worker
|
||||||
|
) as pool:
|
||||||
|
batches = list(pool.imap_unordered(_process_file, jobs))
|
||||||
|
order = {source.as_posix(): index for index, source in enumerate(files)}
|
||||||
|
return sorted(
|
||||||
|
(record for batch in batches for record in batch),
|
||||||
|
key=lambda record: (order[record["file"]], record["page"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _publish(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
changed_files: list[Path],
|
||||||
|
staging: Path,
|
||||||
|
records: list[dict],
|
||||||
|
) -> Path:
|
||||||
|
workspace.runs_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
run_dir = Path(
|
||||||
|
tempfile.mkdtemp(prefix="crop-exercise-bottoms-", dir=workspace.runs_dir)
|
||||||
|
)
|
||||||
|
backup_root = run_dir / "Copies"
|
||||||
|
replaced: list[Path] = []
|
||||||
|
try:
|
||||||
|
for source in changed_files:
|
||||||
|
backup = backup_root / source.relative_to(workspace.copies_dir)
|
||||||
|
backup.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
shutil.copy2(source, backup)
|
||||||
|
(run_dir / "report.json").write_text(
|
||||||
|
json.dumps(records, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
for source in changed_files:
|
||||||
|
prepared = staging / source.relative_to(workspace.copies_dir)
|
||||||
|
prepared.replace(source)
|
||||||
|
replaced.append(source)
|
||||||
|
except BaseException:
|
||||||
|
for source in replaced:
|
||||||
|
backup = backup_root / source.relative_to(workspace.copies_dir)
|
||||||
|
if backup.is_file():
|
||||||
|
shutil.copy2(backup, source)
|
||||||
|
shutil.rmtree(run_dir, ignore_errors=True)
|
||||||
|
raise
|
||||||
|
return backup_root
|
||||||
|
|
||||||
|
|
||||||
|
def crop_statistics(records: list[dict]) -> tuple[int, float]:
|
||||||
|
"""Return cropped exercise count and mean removed percentage per exercise."""
|
||||||
|
totals: dict[str, list[float]] = {}
|
||||||
|
cropped_files: set[str] = set()
|
||||||
|
for record in records:
|
||||||
|
total_height, removed_height = totals.setdefault(record["file"], [0.0, 0.0])
|
||||||
|
totals[record["file"]] = [
|
||||||
|
total_height + float(record["height_lines"]),
|
||||||
|
removed_height + float(record["bottom_removed_lines"]),
|
||||||
|
]
|
||||||
|
if record["status"] == "cropped":
|
||||||
|
cropped_files.add(record["file"])
|
||||||
|
percentages = [
|
||||||
|
min(100.0, totals[file_name][1] / totals[file_name][0] * 100)
|
||||||
|
for file_name in cropped_files
|
||||||
|
if totals[file_name][0] > 0
|
||||||
|
]
|
||||||
|
mean_percentage = sum(percentages) / len(percentages) if percentages else 0.0
|
||||||
|
return len(cropped_files), mean_percentage
|
||||||
|
|
||||||
|
|
||||||
|
def run(workspace: EvaluationWorkspace, target: Path, *, workers: int = 5) -> ExitCode:
|
||||||
|
if workers < 1:
|
||||||
|
raise CliError(
|
||||||
|
"Le nombre de traitements parallèles doit être positif.",
|
||||||
|
ExitCode.INVALID_ARGUMENTS,
|
||||||
|
)
|
||||||
|
files = selected_files(workspace, target)
|
||||||
|
if not files:
|
||||||
|
raise CliError(
|
||||||
|
"Aucun PDF de réponse trouvé dans Copies/CopieXX/.",
|
||||||
|
ExitCode.INVALID_ARGUMENTS,
|
||||||
|
)
|
||||||
|
with tempfile.TemporaryDirectory(
|
||||||
|
prefix=".crop-exercise-bottoms-", dir=workspace.root
|
||||||
|
) as directory:
|
||||||
|
staging = Path(directory)
|
||||||
|
records = process_files(workspace, files, staging, workers)
|
||||||
|
changed_names = {
|
||||||
|
record["file"] for record in records if record["status"] == "cropped"
|
||||||
|
}
|
||||||
|
changed_files = [source for source in files if source.as_posix() in changed_names]
|
||||||
|
digests = {record["file"]: record["source_sha256"] for record in records}
|
||||||
|
for source in files:
|
||||||
|
if hashlib.sha256(source.read_bytes()).hexdigest() != digests[source.as_posix()]:
|
||||||
|
raise CliError(
|
||||||
|
f"{source} a changé pendant l’analyse. Aucun PDF remplacé."
|
||||||
|
)
|
||||||
|
cropped_exercises, mean_percentage = crop_statistics(records)
|
||||||
|
if not changed_files:
|
||||||
|
print("Terminé : aucun exercice ne remplit les critères de rognage.", flush=True)
|
||||||
|
print(
|
||||||
|
"Rognage moyen des exercices modifiés : 0.0 %.", flush=True
|
||||||
|
)
|
||||||
|
return ExitCode.SUCCESS
|
||||||
|
backup = _publish(workspace, changed_files, staging, records)
|
||||||
|
cropped_pages = sum(record["status"] == "cropped" for record in records)
|
||||||
|
print(f"Sauvegarde des PDF non rognés : {backup}", flush=True)
|
||||||
|
print(
|
||||||
|
f"Terminé : {cropped_exercises} exercice(s) rogné(s), soit "
|
||||||
|
f"{cropped_pages} page(s) dans {len(changed_files)} PDF remplacé(s).",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
f"Rognage moyen des exercices modifiés : {mean_percentage:.1f} %.",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
return ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
|
parser = target_parser(
|
||||||
|
"Rogner les grands espaces vides au bas des réponses déjà découpées"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--workers",
|
||||||
|
type=int,
|
||||||
|
default=5,
|
||||||
|
help="Nombre de PDF traités en parallèle (défaut : 5)",
|
||||||
|
)
|
||||||
|
|
||||||
|
def handle(arguments: argparse.Namespace) -> ExitCode:
|
||||||
|
workspace, target = workspace_from_target(arguments)
|
||||||
|
return run(workspace, target, workers=arguments.workers)
|
||||||
|
|
||||||
|
return execute(parser, argv, handle)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
"""Optional preprocessing: replace split copies with ink-guided crops."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import multiprocessing
|
||||||
|
import signal
|
||||||
|
import shutil
|
||||||
|
import tempfile
|
||||||
|
import cv2
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from copienator.cli import CliError, ExitCode, execute, target_parser, workspace_from_target
|
||||||
|
from copienator.crop_blank_margins import process_pdf
|
||||||
|
from copienator.filesystem import staged_files
|
||||||
|
from copienator.workspace import EvaluationWorkspace
|
||||||
|
|
||||||
|
|
||||||
|
def selected_files(workspace: EvaluationWorkspace, target: Path) -> list[Path]:
|
||||||
|
workspace.require_directories("Copies")
|
||||||
|
copies = workspace.copies_dir.resolve()
|
||||||
|
target = target.resolve()
|
||||||
|
if target.is_file():
|
||||||
|
if target.parent != copies or target.suffix.lower() != ".pdf":
|
||||||
|
raise CliError("La cible doit être un PDF du dossier Copies.", ExitCode.INVALID_ARGUMENTS)
|
||||||
|
files = [target]
|
||||||
|
elif target in (workspace.root, copies):
|
||||||
|
files = sorted(copies.glob("*.pdf"), key=lambda path: path.name.casefold())
|
||||||
|
else:
|
||||||
|
raise CliError("Cible attendue : évaluation, dossier Copies ou PDF dans Copies.",
|
||||||
|
ExitCode.INVALID_ARGUMENTS)
|
||||||
|
for source in files:
|
||||||
|
if source.is_symlink():
|
||||||
|
raise CliError(f"Lien symbolique non pris en charge : {source}")
|
||||||
|
if source.with_suffix(".json").exists():
|
||||||
|
raise CliError(
|
||||||
|
f"{source.name} possède déjà des coordonnées de labels. "
|
||||||
|
"Le rognage doit précéder leur détection. Pour reprendre le prétraitement, "
|
||||||
|
"mettez de côté le JSON associé, puis régénérez la découpe des marges et les labels. "
|
||||||
|
"Aucun PDF n’a été remplacé.")
|
||||||
|
return files
|
||||||
|
|
||||||
|
|
||||||
|
def _initialize_worker() -> None:
|
||||||
|
# Five copies should use five cores, not five OpenCV thread pools. MuPDF
|
||||||
|
# must also stay isolated in separate processes rather than Python threads.
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
||||||
|
|
||||||
|
|
||||||
|
def _process_copy(job: tuple[Path, Path]) -> list[dict]:
|
||||||
|
source, destination = job
|
||||||
|
|
||||||
|
def progress(page, total, row):
|
||||||
|
removed = row["top_removed_mm"]+row["bottom_removed_mm"]
|
||||||
|
print(f"{source.name} — Page {page}/{total} : {removed:.1f} mm retirés", flush=True)
|
||||||
|
|
||||||
|
return process_pdf(source, destination, None, 200, 6, 5, progress=progress)
|
||||||
|
|
||||||
|
|
||||||
|
def process_copies(files: list[Path], staging: Path, workers: int) -> list[dict]:
|
||||||
|
jobs = [(source, staging/source.name) for source in files]
|
||||||
|
count = min(workers, len(jobs))
|
||||||
|
print(f"Rognage de {len(files)} copies avec {count} traitement(s) en parallèle.", flush=True)
|
||||||
|
if count == 1:
|
||||||
|
previous_threads = cv2.getNumThreads()
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
try:
|
||||||
|
batches = [_process_copy(job) for job in jobs]
|
||||||
|
finally:
|
||||||
|
cv2.setNumThreads(previous_threads)
|
||||||
|
else:
|
||||||
|
# spawn works on Windows and avoids inheriting GUI/native-library state.
|
||||||
|
# Pool's context terminates and joins workers on errors or cancellation
|
||||||
|
# before staged_files removes the unpublished PDFs.
|
||||||
|
with multiprocessing.get_context("spawn").Pool(count, _initialize_worker) as pool:
|
||||||
|
batches = list(pool.imap_unordered(_process_copy, jobs))
|
||||||
|
order = {source.name: i for i, source in enumerate(files)}
|
||||||
|
return sorted((row for batch in batches for row in batch),
|
||||||
|
key=lambda row: (order[row["file"]], row["page"]))
|
||||||
|
|
||||||
|
|
||||||
|
def crop_statistics(records: list[dict]) -> tuple[int, float, int]:
|
||||||
|
"""Return cropped page count, their mean removed percentage, and >30% count."""
|
||||||
|
percentages: list[float] = []
|
||||||
|
for record in records:
|
||||||
|
removed_mm = record["top_removed_mm"] + record["bottom_removed_mm"]
|
||||||
|
if removed_mm <= 0:
|
||||||
|
continue
|
||||||
|
x0, y0, x1, y1 = record["original_cropbox"]
|
||||||
|
original_height_points = (
|
||||||
|
x1 - x0 if record.get("rotation", 0) % 180 else y1 - y0
|
||||||
|
)
|
||||||
|
if original_height_points <= 0:
|
||||||
|
continue
|
||||||
|
original_height_mm = original_height_points * 25.4 / 72
|
||||||
|
percentages.append(min(100.0, removed_mm / original_height_mm * 100))
|
||||||
|
mean_percentage = sum(percentages) / len(percentages) if percentages else 0.0
|
||||||
|
over_thirty = sum(percentage > 30 for percentage in percentages)
|
||||||
|
return len(percentages), mean_percentage, over_thirty
|
||||||
|
|
||||||
|
|
||||||
|
def run(workspace: EvaluationWorkspace, target: Path, *, workers: int = 5) -> ExitCode:
|
||||||
|
if workers < 1:
|
||||||
|
raise CliError("Le nombre de traitements parallèles doit être positif.",
|
||||||
|
ExitCode.INVALID_ARGUMENTS)
|
||||||
|
files = selected_files(workspace, target)
|
||||||
|
if not files:
|
||||||
|
raise CliError("Aucun PDF trouvé dans Copies.", ExitCode.INVALID_ARGUMENTS)
|
||||||
|
# Prepare the whole batch before replacing any copy. A detection failure or
|
||||||
|
# interruption leaves the working PDFs intact; commit errors roll back.
|
||||||
|
with staged_files(workspace.copies_dir) as staging:
|
||||||
|
records = process_copies(files, staging, workers)
|
||||||
|
# Detect edits made while the batch was being analysed, before saving
|
||||||
|
# backups or publishing results derived from an obsolete source.
|
||||||
|
digests = {row["file"]: row["source_sha256"] for row in records}
|
||||||
|
for source in files:
|
||||||
|
if hashlib.sha256(source.read_bytes()).hexdigest() != digests[source.name]:
|
||||||
|
raise CliError(f"{source.name} a changé pendant l’analyse. Aucun PDF remplacé.")
|
||||||
|
workspace.runs_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
backup = Path(tempfile.mkdtemp(prefix="crop-margins-", dir=workspace.runs_dir))
|
||||||
|
originals = backup/"Copies"
|
||||||
|
originals.mkdir()
|
||||||
|
for source in files:
|
||||||
|
shutil.copy2(source, originals/source.name)
|
||||||
|
(backup/"report.json").write_text(json.dumps(records, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8")
|
||||||
|
print(f"Sauvegarde des PDF non rognés : {originals}", flush=True)
|
||||||
|
cropped, mean_percentage, over_thirty = crop_statistics(records)
|
||||||
|
print(f"Terminé : {cropped}/{len(records)} pages rognées ; "
|
||||||
|
f"{len(files)} PDF remplacés dans Copies.", flush=True)
|
||||||
|
print(
|
||||||
|
f"Rognage moyen des pages modifiées : {mean_percentage:.1f} % ; "
|
||||||
|
f"{over_thirty} page(s) rognée(s) de plus de 30 %.",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
return ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
|
parser = target_parser("Rogner les zones vides des PDF dans Copies, avant les labels")
|
||||||
|
parser.add_argument("--workers", type=int, default=5,
|
||||||
|
help="Nombre de copies traitées en parallèle (défaut : 5)")
|
||||||
|
|
||||||
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
|
workspace, target = workspace_from_target(args)
|
||||||
|
return run(workspace, target, workers=args.workers)
|
||||||
|
|
||||||
|
return execute(parser, argv, handle)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -2,8 +2,8 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import threading
|
import threading
|
||||||
import time
|
|
||||||
import tkinter as tk
|
import tkinter as tk
|
||||||
|
from tkinter import messagebox
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -23,10 +23,13 @@ from copienator import (
|
|||||||
workspace_from_target,
|
workspace_from_target,
|
||||||
)
|
)
|
||||||
from copienator.filesystem import staged_files
|
from copienator.filesystem import staged_files
|
||||||
|
from copienator.copy_errors import clear_copy_error, mark_copy_error, marked_copy_paths
|
||||||
|
|
||||||
DELIMITER_WIDTH = 5
|
DELIMITER_WIDTH = 5
|
||||||
DELIMITER_COLOR = (0, 0, 0)
|
DELIMITER_COLOR = (0, 0, 0)
|
||||||
OUTPUT_SIZE = (1800, 1000)
|
OUTPUT_SIZE = (1800, 1000)
|
||||||
|
CROP_SHIFT_STEP = 50
|
||||||
|
CROP_WIDTH_STEP = 50
|
||||||
pdf_cache_lock = threading.Lock()
|
pdf_cache_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
@@ -64,7 +67,9 @@ def stitch_images(image_list: list[Image.Image]) -> Image.Image | None:
|
|||||||
|
|
||||||
@lru_cache(maxsize=3)
|
@lru_cache(maxsize=3)
|
||||||
def _get_pdf_pages_cached(pdf_path: Path) -> list[Image.Image]:
|
def _get_pdf_pages_cached(pdf_path: Path) -> list[Image.Image]:
|
||||||
return convert_from_path(pdf_path)
|
# Label coordinates use the full, displayed MediaBox, including on PDFs
|
||||||
|
# previously cropped by crop-margins. Keep this in sync with split-answers.
|
||||||
|
return convert_from_path(pdf_path, use_cropbox=False)
|
||||||
|
|
||||||
|
|
||||||
def get_pdf_pages(pdf_path: Path) -> list[Image.Image]:
|
def get_pdf_pages(pdf_path: Path) -> list[Image.Image]:
|
||||||
@@ -77,6 +82,7 @@ def process_single_pdf(
|
|||||||
pdf_path: Path,
|
pdf_path: Path,
|
||||||
shift_offset: int = 0,
|
shift_offset: int = 0,
|
||||||
max_per_file: int = 5,
|
max_per_file: int = 5,
|
||||||
|
width_offset: int = 0,
|
||||||
) -> tuple[Image.Image, list[Image.Image], dict[str, object]] | None:
|
) -> tuple[Image.Image, list[Image.Image], dict[str, object]] | None:
|
||||||
"""Convert one PDF into a preview, full-resolution splits and metadata."""
|
"""Convert one PDF into a preview, full-resolution splits and metadata."""
|
||||||
try:
|
try:
|
||||||
@@ -87,7 +93,10 @@ def process_single_pdf(
|
|||||||
left, right = 0, width
|
left, right = 0, width
|
||||||
else:
|
else:
|
||||||
left = max(0, 100 + shift_offset)
|
left = max(0, 100 + shift_offset)
|
||||||
right = min(width, width // 3 + 100 + shift_offset)
|
right = min(
|
||||||
|
width,
|
||||||
|
width // 3 + 100 + shift_offset + max(0, width_offset),
|
||||||
|
)
|
||||||
if right > left:
|
if right > left:
|
||||||
cropped_images.append(image.crop((left, 0, right, height)))
|
cropped_images.append(image.crop((left, 0, right, height)))
|
||||||
if not cropped_images:
|
if not cropped_images:
|
||||||
@@ -156,8 +165,14 @@ class ImageReviewer:
|
|||||||
) -> None:
|
) -> None:
|
||||||
self.files = files
|
self.files = files
|
||||||
self.output_dir = output_dir
|
self.output_dir = output_dir
|
||||||
|
self.workspace = EvaluationWorkspace(output_dir.parent)
|
||||||
|
self.completed = False
|
||||||
|
self.had_errors = False
|
||||||
|
self.stop_prefetch = threading.Event()
|
||||||
|
self.current_result = None
|
||||||
self.index = 0
|
self.index = 0
|
||||||
self.current_shift = 0
|
self.current_shift = 0
|
||||||
|
self.current_width_offset = 0
|
||||||
self.default_max_per_file = default_max_per_file
|
self.default_max_per_file = default_max_per_file
|
||||||
self.current_max_per_file = default_max_per_file
|
self.current_max_per_file = default_max_per_file
|
||||||
self.current_preview: Image.Image | None = None
|
self.current_preview: Image.Image | None = None
|
||||||
@@ -174,9 +189,12 @@ class ImageReviewer:
|
|||||||
self.label_info = tk.Label(self.root, text="", font=("Arial", 12, "bold"))
|
self.label_info = tk.Label(self.root, text="", font=("Arial", 12, "bold"))
|
||||||
self.label_info.pack(pady=5)
|
self.label_info.pack(pady=5)
|
||||||
self.root.bind("<Return>", self.on_next)
|
self.root.bind("<Return>", self.on_next)
|
||||||
self.root.bind("n", lambda _event: self.on_shift(50))
|
self.root.bind("s", self.on_skip)
|
||||||
self.root.bind("N", lambda _event: self.on_shift(100))
|
self.root.protocol("WM_DELETE_WINDOW", self.on_close)
|
||||||
self.root.bind("t", lambda _event: self.on_shift(-50))
|
self.root.bind("n", lambda _event: self.on_shift(CROP_SHIFT_STEP))
|
||||||
|
self.root.bind("N", lambda _event: self.on_shift(2 * CROP_SHIFT_STEP))
|
||||||
|
self.root.bind("t", lambda _event: self.on_shift(-CROP_SHIFT_STEP))
|
||||||
|
self.root.bind("l", lambda _event: self.on_enlarge(CROP_WIDTH_STEP))
|
||||||
self.root.bind("1", lambda _event: self.on_set_max_pages(1))
|
self.root.bind("1", lambda _event: self.on_set_max_pages(1))
|
||||||
|
|
||||||
Thread(target=self.prefetch_worker, daemon=True).start()
|
Thread(target=self.prefetch_worker, daemon=True).start()
|
||||||
@@ -194,20 +212,26 @@ class ImageReviewer:
|
|||||||
|
|
||||||
def prefetch_worker(self) -> None:
|
def prefetch_worker(self) -> None:
|
||||||
processed_index = -1
|
processed_index = -1
|
||||||
while True:
|
while not self.stop_prefetch.is_set():
|
||||||
target = self.index + 1
|
target = self.index + 1
|
||||||
if target < len(self.files) and target != processed_index:
|
if target < len(self.files) and target != processed_index:
|
||||||
get_pdf_pages(self.files[target])
|
try:
|
||||||
|
get_pdf_pages(self.files[target])
|
||||||
|
except Exception:
|
||||||
|
pass # The foreground review reports and flags conversion errors.
|
||||||
processed_index = target
|
processed_index = target
|
||||||
time.sleep(0.05)
|
self.stop_prefetch.wait(0.05)
|
||||||
|
|
||||||
def load_current_image(self) -> None:
|
def load_current_image(self) -> None:
|
||||||
if self.index >= len(self.files):
|
if self.index >= len(self.files):
|
||||||
print("All files processed.")
|
print("All files processed.")
|
||||||
self.root.destroy()
|
self.completed = True
|
||||||
|
self.on_close()
|
||||||
return
|
return
|
||||||
self.is_processing = False
|
self.is_processing = False
|
||||||
self.current_shift = 0
|
self.current_shift = 0
|
||||||
|
self.current_width_offset = 0
|
||||||
|
self.current_result = None
|
||||||
self.trigger_processing(self.files[self.index], self.current_shift)
|
self.trigger_processing(self.files[self.index], self.current_shift)
|
||||||
|
|
||||||
def trigger_processing(self, pdf_path: Path, shift: int) -> None:
|
def trigger_processing(self, pdf_path: Path, shift: int) -> None:
|
||||||
@@ -219,7 +243,12 @@ class ImageReviewer:
|
|||||||
|
|
||||||
def worker() -> None:
|
def worker() -> None:
|
||||||
self.manual_queue.put(
|
self.manual_queue.put(
|
||||||
process_single_pdf(pdf_path, shift, self.current_max_per_file)
|
process_single_pdf(
|
||||||
|
pdf_path,
|
||||||
|
shift,
|
||||||
|
self.current_max_per_file,
|
||||||
|
self.current_width_offset,
|
||||||
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
Thread(target=worker, daemon=True).start()
|
Thread(target=worker, daemon=True).start()
|
||||||
@@ -228,13 +257,13 @@ class ImageReviewer:
|
|||||||
def check_manual_queue(self, pdf_path: Path) -> None:
|
def check_manual_queue(self, pdf_path: Path) -> None:
|
||||||
try:
|
try:
|
||||||
result = self.manual_queue.get_nowait()
|
result = self.manual_queue.get_nowait()
|
||||||
|
self.is_processing = False
|
||||||
if result is None:
|
if result is None:
|
||||||
print(f"Failed to process {pdf_path.name}, skipping.")
|
print(f"Failed to process {pdf_path.name}, skipping.")
|
||||||
self.index += 1
|
self._mark_error(pdf_path, "Échec de la conversion pour la découpe des marges")
|
||||||
self.load_current_image()
|
self._advance()
|
||||||
else:
|
else:
|
||||||
self.handle_processing_result(result, pdf_path)
|
self.handle_processing_result(result, pdf_path)
|
||||||
self.is_processing = False
|
|
||||||
except Empty:
|
except Empty:
|
||||||
self.root.after(100, lambda: self.check_manual_queue(pdf_path))
|
self.root.after(100, lambda: self.check_manual_queue(pdf_path))
|
||||||
|
|
||||||
@@ -244,7 +273,7 @@ class ImageReviewer:
|
|||||||
pdf_path: Path,
|
pdf_path: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.current_preview = result[0]
|
self.current_preview = result[0]
|
||||||
save_results(result, pdf_path, self.output_dir)
|
self.current_result = result
|
||||||
self.update_display(pdf_path.name, result[2])
|
self.update_display(pdf_path.name, result[2])
|
||||||
|
|
||||||
def update_display(self, filename: str, schema: dict[str, object]) -> None:
|
def update_display(self, filename: str, schema: dict[str, object]) -> None:
|
||||||
@@ -256,10 +285,12 @@ class ImageReviewer:
|
|||||||
self.label_info.configure(
|
self.label_info.configure(
|
||||||
text=(
|
text=(
|
||||||
f"[{self.index + 1}/{len(self.files)}] {filename} | "
|
f"[{self.index + 1}/{len(self.files)}] {filename} | "
|
||||||
f"Shift: {self.current_shift}px\nFiles: {schema['number_of_files']} | "
|
f"Shift: {self.current_shift}px | "
|
||||||
|
f"Extra width: {self.current_width_offset}px\n"
|
||||||
|
f"Files: {schema['number_of_files']} | "
|
||||||
f"Cols: {schema['columns_per_file']}\n"
|
f"Cols: {schema['columns_per_file']}\n"
|
||||||
"Enter: Next | n: +50 | N: +100 | t: -50 | "
|
"Enter: Save and next | s: flag error and skip | n: +50 | N: +100 | t: -50 | "
|
||||||
"1: use single column"
|
"l: widen by 50 | 1: use full pages"
|
||||||
),
|
),
|
||||||
fg="black",
|
fg="black",
|
||||||
)
|
)
|
||||||
@@ -271,11 +302,45 @@ class ImageReviewer:
|
|||||||
print(f"Applying shift: {self.current_shift}")
|
print(f"Applying shift: {self.current_shift}")
|
||||||
self.trigger_processing(self.files[self.index], self.current_shift)
|
self.trigger_processing(self.files[self.index], self.current_shift)
|
||||||
|
|
||||||
def on_next(self, _event: object) -> None:
|
def on_enlarge(self, amount: int) -> None:
|
||||||
if self.is_processing:
|
if self.is_processing:
|
||||||
return
|
return
|
||||||
|
self.current_width_offset += amount
|
||||||
|
print(f"Applying extra width: {self.current_width_offset}")
|
||||||
|
self.trigger_processing(self.files[self.index], self.current_shift)
|
||||||
|
|
||||||
|
def on_next(self, _event: object) -> None:
|
||||||
|
if self.is_processing or self.current_result is None:
|
||||||
|
return
|
||||||
|
pdf_path = self.files[self.index]
|
||||||
|
try:
|
||||||
|
save_results(self.current_result, pdf_path, self.output_dir)
|
||||||
|
clear_copy_error(self.workspace, pdf_path)
|
||||||
|
except Exception as exc:
|
||||||
|
self._mark_error(pdf_path, f"Échec de l’enregistrement : {exc}")
|
||||||
|
messagebox.showerror("Enregistrement impossible", str(exc), parent=self.root)
|
||||||
|
return
|
||||||
|
self._advance()
|
||||||
|
|
||||||
|
def _mark_error(self, pdf_path: Path, reason: str) -> None:
|
||||||
|
mark_copy_error(self.workspace, pdf_path, reason)
|
||||||
|
self.had_errors = True
|
||||||
|
print(f"[Copie signalée] {pdf_path.name}: {reason}")
|
||||||
|
|
||||||
|
def on_skip(self, _event=None) -> None:
|
||||||
|
if self.is_processing or self.index >= len(self.files):
|
||||||
|
return
|
||||||
|
self._mark_error(self.files[self.index], "Problème repéré pendant la découpe des marges")
|
||||||
|
self._advance()
|
||||||
|
|
||||||
|
def on_close(self) -> None:
|
||||||
|
self.stop_prefetch.set()
|
||||||
|
self.root.destroy()
|
||||||
|
|
||||||
|
def _advance(self) -> None:
|
||||||
self.index += 1
|
self.index += 1
|
||||||
self.current_shift = 0
|
self.current_shift = 0
|
||||||
|
self.current_width_offset = 0
|
||||||
self.current_max_per_file = self.default_max_per_file
|
self.current_max_per_file = self.default_max_per_file
|
||||||
self.load_current_image()
|
self.load_current_image()
|
||||||
|
|
||||||
@@ -304,23 +369,27 @@ def run(
|
|||||||
target: Path,
|
target: Path,
|
||||||
*,
|
*,
|
||||||
fullpage: bool = False,
|
fullpage: bool = False,
|
||||||
|
marked: bool = False,
|
||||||
) -> ExitCode:
|
) -> ExitCode:
|
||||||
files = _selected_files(workspace, target)
|
files = marked_copy_paths(workspace) if marked else _selected_files(workspace, target)
|
||||||
if not files:
|
if not files:
|
||||||
print("No PDF files found.")
|
print("No PDF files found.")
|
||||||
return ExitCode.SUCCESS
|
return ExitCode.SUCCESS
|
||||||
workspace.cutleft_dir.mkdir(parents=True, exist_ok=True)
|
workspace.cutleft_dir.mkdir(parents=True, exist_ok=True)
|
||||||
_get_pdf_pages_cached.cache_clear()
|
_get_pdf_pages_cached.cache_clear()
|
||||||
ImageReviewer(
|
reviewer = ImageReviewer(
|
||||||
files,
|
files,
|
||||||
workspace.cutleft_dir,
|
workspace.cutleft_dir,
|
||||||
default_max_per_file=1 if fullpage else 5,
|
default_max_per_file=1 if fullpage else 5,
|
||||||
)
|
)
|
||||||
return ExitCode.SUCCESS
|
if not reviewer.completed:
|
||||||
|
return ExitCode.INTERRUPTED
|
||||||
|
return ExitCode.PARTIAL if reviewer.had_errors else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
parser = target_parser("Interactively crop the label margin from PDF copies")
|
parser = target_parser("Interactively crop the label margin from PDF copies")
|
||||||
|
parser.add_argument("--marked", action="store_true", help="Review flagged copies and clear each flag after saving with Enter")
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--fullpage",
|
"--fullpage",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
@@ -334,11 +403,10 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
|
|
||||||
def handle(args: argparse.Namespace) -> ExitCode:
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
workspace, target = workspace_from_target(args)
|
workspace, target = workspace_from_target(args)
|
||||||
return run(workspace, target, fullpage=args.fullpage)
|
return run(workspace, target, fullpage=args.fullpage, marked=args.marked)
|
||||||
|
|
||||||
return execute(parser, argv, handle)
|
return execute(parser, argv, handle)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
raise SystemExit(main())
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ from copienator import (
|
|||||||
CliError,
|
CliError,
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
|
atomic_write_text,
|
||||||
evaluation_parser,
|
evaluation_parser,
|
||||||
execute,
|
execute,
|
||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
@@ -142,10 +143,11 @@ def save_split_content(text, path, base_fname, problem):
|
|||||||
def process_directory(workspace: EvaluationWorkspace) -> ExitCode:
|
def process_directory(workspace: EvaluationWorkspace) -> ExitCode:
|
||||||
directory = str(workspace.root)
|
directory = str(workspace.root)
|
||||||
# Find the first .tex file in the directory
|
# Find the first .tex file in the directory
|
||||||
tex_files = glob.glob(os.path.join(directory, "*.tex"))
|
enonce = workspace.root / "enonce.tex"
|
||||||
|
tex_files = [str(enonce)] if enonce.is_file() else sorted(glob.glob(os.path.join(directory, "*.tex")))
|
||||||
if not tex_files:
|
if not tex_files:
|
||||||
print(f"No .tex file found in {directory}. Looking in /Staging/Interro/")
|
print(f"No .tex file found in {directory}. Looking in /Staging/Interro/")
|
||||||
int_name = directory.removesuffix("/")
|
int_name = workspace.root.name
|
||||||
tex_path = os.path.join(os.path.expanduser("~"), "Prépa/Staging/Interro", f"{int_name}.tex")
|
tex_path = os.path.join(os.path.expanduser("~"), "Prépa/Staging/Interro", f"{int_name}.tex")
|
||||||
if os.path.exists(tex_path):
|
if os.path.exists(tex_path):
|
||||||
tex_file = tex_path
|
tex_file = tex_path
|
||||||
@@ -172,6 +174,7 @@ def process_directory(workspace: EvaluationWorkspace) -> ExitCode:
|
|||||||
labels_staging = labels_file.with_name(f".{labels_file.name}.{uuid4().hex}.tmp")
|
labels_staging = labels_file.with_name(f".{labels_file.name}.{uuid4().hex}.tmp")
|
||||||
current_ex_num = 1
|
current_ex_num = 1
|
||||||
had_errors = False
|
had_errors = False
|
||||||
|
exercise_groups = []
|
||||||
|
|
||||||
# Read entirely to allow chunking
|
# Read entirely to allow chunking
|
||||||
with open(tex_file, 'r', encoding='utf-8') as f_in:
|
with open(tex_file, 'r', encoding='utf-8') as f_in:
|
||||||
@@ -267,6 +270,7 @@ def process_directory(workspace: EvaluationWorkspace) -> ExitCode:
|
|||||||
|
|
||||||
for label in block_labels:
|
for label in block_labels:
|
||||||
f_labels.write(f"{label}\n")
|
f_labels.write(f"{label}\n")
|
||||||
|
exercise_groups.append(block_labels)
|
||||||
current_ex_num += 1
|
current_ex_num += 1
|
||||||
|
|
||||||
except WindowsLabelError:
|
except WindowsLabelError:
|
||||||
@@ -280,6 +284,8 @@ def process_directory(workspace: EvaluationWorkspace) -> ExitCode:
|
|||||||
had_errors = True
|
had_errors = True
|
||||||
|
|
||||||
labels_staging.replace(labels_file)
|
labels_staging.replace(labels_file)
|
||||||
|
atomic_write_text(workspace.label_groups_file,
|
||||||
|
"".join(", ".join(group) + "\n" for group in exercise_groups))
|
||||||
return ExitCode.PARTIAL if had_errors else ExitCode.SUCCESS
|
return ExitCode.PARTIAL if had_errors else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
@@ -293,4 +299,3 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
raise SystemExit(main())
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ from copienator import (
|
|||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
from copienator.platform import validate_windows_labels
|
from copienator.platform import validate_windows_labels
|
||||||
|
from copienator.filesystem import staged_directory
|
||||||
from copienator.utils import compile_to_pdf
|
from copienator.utils import compile_to_pdf
|
||||||
|
|
||||||
|
|
||||||
@@ -73,6 +74,9 @@ class RubricItem(BaseModel):
|
|||||||
class GroupRubrics(BaseModel):
|
class GroupRubrics(BaseModel):
|
||||||
rubrics: list[RubricItem]
|
rubrics: list[RubricItem]
|
||||||
|
|
||||||
|
class LabelGroups(BaseModel):
|
||||||
|
groups: list[list[str]]
|
||||||
|
|
||||||
PROMPT_4 = """Je te fournis les questions, le contexte éventuel, et les corrections pour un groupe de questions d'un examen.
|
PROMPT_4 = """Je te fournis les questions, le contexte éventuel, et les corrections pour un groupe de questions d'un examen.
|
||||||
Ta tâche :
|
Ta tâche :
|
||||||
Établir un barème de correction détaillé pour CHAQUE question.
|
Établir un barème de correction détaillé pour CHAQUE question.
|
||||||
@@ -82,6 +86,11 @@ même si certains textes fournis sont dans une autre langue.
|
|||||||
Conserve les formules mathématiques, les labels exacts des questions et les
|
Conserve les formules mathématiques, les labels exacts des questions et les
|
||||||
clés JSON `rubrics`, `label` et `rubric_content` sans les traduire.
|
clés JSON `rubrics`, `label` et `rubric_content` sans les traduire.
|
||||||
Chaque question DOIT être notée sur exactement 4 points. Propose une répartition logique de ces points.
|
Chaque question DOIT être notée sur exactement 4 points. Propose une répartition logique de ces points.
|
||||||
|
Il est inutile d'indiquer dans `rubric_content` que le barème totalise 4 points :
|
||||||
|
ce total est toujours implicite.
|
||||||
|
N'utilise pas de caractères mathématiques Unicode dans `rubric_content`.
|
||||||
|
Écris les expressions mathématiques en LaTeX, par exemple
|
||||||
|
`$\\lfloor \\sqrt{k} \\rfloor$` plutôt qu'avec des symboles Unicode.
|
||||||
Par exemple :
|
Par exemple :
|
||||||
- Au moins 2 points si le résultat est correct.
|
- Au moins 2 points si le résultat est correct.
|
||||||
- Mettre la moitié des points si le raisonnement est correct mais pas le résultat.
|
- Mettre la moitié des points si le raisonnement est correct mais pas le résultat.
|
||||||
@@ -176,6 +185,106 @@ def find_file(folder: Path, base_name: str) -> Path | None:
|
|||||||
return path
|
return path
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def generate_rubrics(client, group_context_text: str) -> dict[str, str]:
|
||||||
|
"""Use the same rubric request for full and selective statement generation."""
|
||||||
|
response = client.models.generate_content(
|
||||||
|
model=MODEL_ID,
|
||||||
|
contents=[types.Content(role="user", parts=[
|
||||||
|
types.Part.from_text(text=PROMPT_4),
|
||||||
|
types.Part.from_text(text=f"--- CONTENU DU GROUPE ---\n{group_context_text}"),
|
||||||
|
])],
|
||||||
|
config=types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
|
system_instruction="Rédige tous les barèmes et consignes de notation en français. Conserve les clés JSON, les labels et les formules mathématiques.",
|
||||||
|
temperature=0.2,
|
||||||
|
response_mime_type="application/json",
|
||||||
|
response_json_schema=GroupRubrics.model_json_schema(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
rubrics = GroupRubrics.model_validate_json(response.text).rubrics
|
||||||
|
if len({item.label for item in rubrics}) != len(rubrics):
|
||||||
|
raise ValueError("Gemini returned duplicate rubric labels")
|
||||||
|
return {item.label: item.rubric_content for item in rubrics}
|
||||||
|
|
||||||
|
|
||||||
|
def validate_groups(groups: list[list[str]], labels: list[str]) -> None:
|
||||||
|
flattened = [label for group in groups for label in group]
|
||||||
|
if (not groups or any(not group for group in groups)
|
||||||
|
or len(flattened) != len(set(flattened)) or set(flattened) != set(labels)):
|
||||||
|
raise CliError("Groups must contain every existing label exactly once")
|
||||||
|
|
||||||
|
|
||||||
|
def refine_existing(workspace: EvaluationWorkspace, mode: str, *, api_client=None) -> ExitCode:
|
||||||
|
"""Regroup or replace rubrics without regenerating statements or solutions."""
|
||||||
|
labels = workspace.read_labels()
|
||||||
|
if not labels or len(labels) != len(set(labels)):
|
||||||
|
raise CliError("Generate unique question labels before refining the statement")
|
||||||
|
validate_windows_labels(labels)
|
||||||
|
questions = {}
|
||||||
|
for label in labels:
|
||||||
|
safe_label = label.replace("/", "_")
|
||||||
|
parts = []
|
||||||
|
for directory, title in (("Text2", "Question"), ("Sol2", "Correction")):
|
||||||
|
path = workspace.root / directory / f"{safe_label}.tex"
|
||||||
|
if not path.is_file():
|
||||||
|
raise CliError(f"Missing {path}; generate statements and solutions first")
|
||||||
|
parts.append(f"{title} [{label}]:\n{path.read_text(encoding='utf-8')}")
|
||||||
|
questions[label] = "\n".join(parts)
|
||||||
|
context = utils.enonce_total(workspace.root)
|
||||||
|
if api_client is None:
|
||||||
|
if not api_key:
|
||||||
|
raise CliError("GEMINI_API_KEY is not configured")
|
||||||
|
api_client = genai.Client(api_key=api_key)
|
||||||
|
|
||||||
|
if mode == "groups":
|
||||||
|
response = api_client.models.generate_content(
|
||||||
|
model=MODEL_ID,
|
||||||
|
contents=[types.Content(role="user", parts=[types.Part.from_text(text=(
|
||||||
|
"Regroupe ces questions d’examen en groupes cohérents pour la correction "
|
||||||
|
"et l’annotation, selon leurs dépendances et leur contexte commun. "
|
||||||
|
"Ne mélange pas des exercices différents. Conserve l’ordre des questions. "
|
||||||
|
"Chaque label doit apparaître exactement une fois, sans modification. "
|
||||||
|
"Renvoie uniquement un objet JSON groups contenant des listes de labels.\n\n"
|
||||||
|
+ context + "\n\n" + "\n\n".join(questions.values())
|
||||||
|
))])],
|
||||||
|
config=types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
|
temperature=0.1, response_mime_type="application/json",
|
||||||
|
response_json_schema=LabelGroups.model_json_schema(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
groups = LabelGroups.model_validate_json(response.text).groups
|
||||||
|
validate_groups(groups, labels)
|
||||||
|
atomic_write_text(workspace.label_groups_file,
|
||||||
|
"".join(", ".join(group) + "\n" for group in groups))
|
||||||
|
print(f"Updated label_groups: {len(groups)} Gemini groups.")
|
||||||
|
elif mode == "persp":
|
||||||
|
if not workspace.label_groups_file.is_file():
|
||||||
|
raise CliError("Generate label_groups before generating rubrics")
|
||||||
|
groups = [[label.strip() for label in line.split(",") if label.strip()]
|
||||||
|
for line in workspace.label_groups_file.read_text(encoding="utf-8").splitlines()
|
||||||
|
if line.strip()]
|
||||||
|
validate_groups(groups, labels)
|
||||||
|
with staged_directory(workspace.root / "Persp") as staging:
|
||||||
|
for group in groups:
|
||||||
|
print(f"Generating rubric (Persp) for group: {', '.join(group)}...")
|
||||||
|
group_content = (
|
||||||
|
"Contexte général de l’examen, fourni uniquement pour comprendre les questions :\n"
|
||||||
|
+ context + "\n\nProduis des barèmes UNIQUEMENT pour les labels suivants : "
|
||||||
|
+ ", ".join(group) + "\n\n" + "\n\n".join(questions[label] for label in group)
|
||||||
|
)
|
||||||
|
rubrics = generate_rubrics(api_client, group_content)
|
||||||
|
if set(rubrics) != set(group) or any(not value.strip() for value in rubrics.values()):
|
||||||
|
raise CliError("Incomplete or unexpected Gemini rubrics; previous Persp preserved")
|
||||||
|
for label, rubric in rubrics.items():
|
||||||
|
(staging / label.replace("/", "_")).write_text(
|
||||||
|
f"{label}\n{rubric}", encoding="utf-8")
|
||||||
|
print("Replaced Persp with Gemini rubrics.")
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown statement refinement: {mode}")
|
||||||
|
return ExitCode.SUCCESS
|
||||||
|
|
||||||
def process_exam(
|
def process_exam(
|
||||||
workspace: EvaluationWorkspace,
|
workspace: EvaluationWorkspace,
|
||||||
restart: bool = False,
|
restart: bool = False,
|
||||||
@@ -235,6 +344,7 @@ def process_exam(
|
|||||||
]
|
]
|
||||||
|
|
||||||
config_1 = types.GenerateContentConfig(
|
config_1 = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=0.1,
|
temperature=0.1,
|
||||||
response_mime_type="application/json",
|
response_mime_type="application/json",
|
||||||
response_json_schema=ExamQuestions.model_json_schema(),
|
response_json_schema=ExamQuestions.model_json_schema(),
|
||||||
@@ -273,6 +383,7 @@ def process_exam(
|
|||||||
]
|
]
|
||||||
|
|
||||||
config_2 = types.GenerateContentConfig(
|
config_2 = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=0.1,
|
temperature=0.1,
|
||||||
response_mime_type="application/json",
|
response_mime_type="application/json",
|
||||||
response_json_schema=ExamSolutions.model_json_schema(),
|
response_json_schema=ExamSolutions.model_json_schema(),
|
||||||
@@ -309,6 +420,7 @@ def process_exam(
|
|||||||
]
|
]
|
||||||
|
|
||||||
config_3 = types.GenerateContentConfig(
|
config_3 = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=0.1,
|
temperature=0.1,
|
||||||
response_mime_type="application/json",
|
response_mime_type="application/json",
|
||||||
response_json_schema=ExamContext.model_json_schema(),
|
response_json_schema=ExamContext.model_json_schema(),
|
||||||
@@ -701,32 +813,9 @@ def process_exam(
|
|||||||
|
|
||||||
group_context_text = "\n\n---\n\n".join(group_text_parts)
|
group_context_text = "\n\n---\n\n".join(group_text_parts)
|
||||||
|
|
||||||
contents_4 = [
|
|
||||||
types.Content(
|
|
||||||
role="user",
|
|
||||||
parts=[
|
|
||||||
types.Part.from_text(text=PROMPT_4),
|
|
||||||
types.Part.from_text(text=f"--- CONTENU DU GROUPE ---\n{group_context_text}"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
config_4 = types.GenerateContentConfig(
|
|
||||||
system_instruction="Rédige tous les barèmes et consignes de notation en français. Conserve les clés JSON, les labels et les formules mathématiques.",
|
|
||||||
temperature=0.2,
|
|
||||||
response_mime_type="application/json",
|
|
||||||
response_json_schema=GroupRubrics.model_json_schema(),
|
|
||||||
)
|
|
||||||
|
|
||||||
print(f"Generating rubric (Persp) for group: {', '.join(labels)}...")
|
print(f"Generating rubric (Persp) for group: {', '.join(labels)}...")
|
||||||
try:
|
try:
|
||||||
response_r = client.models.generate_content(
|
rubrics_map = generate_rubrics(client, group_context_text)
|
||||||
model=MODEL_ID,
|
|
||||||
contents=contents_4,
|
|
||||||
config=config_4
|
|
||||||
)
|
|
||||||
rubrics_data = GroupRubrics.model_validate_json(response_r.text)
|
|
||||||
rubrics_map = {r.label: r.rubric_content for r in rubrics_data.rubrics}
|
|
||||||
except Exception as e: # noqa: BLE001 - remote API boundary
|
except Exception as e: # noqa: BLE001 - remote API boundary
|
||||||
print(f"Error generating rubric for group {labels[0]}: {e}")
|
print(f"Error generating rubric for group {labels[0]}: {e}")
|
||||||
processing_errors.append(str(e))
|
processing_errors.append(str(e))
|
||||||
@@ -822,12 +911,23 @@ def process_exam(
|
|||||||
workspace.labels_file,
|
workspace.labels_file,
|
||||||
"".join(f"{label}\n" for label in labels_list),
|
"".join(f"{label}\n" for label in labels_list),
|
||||||
)
|
)
|
||||||
|
atomic_write_text(
|
||||||
|
workspace.label_groups_file,
|
||||||
|
"".join(", ".join(item.label for item in group if isinstance(item, QuestionItem)) + "\n"
|
||||||
|
for group in grouped_extraction.groups
|
||||||
|
if any(isinstance(item, QuestionItem) for item in group)),
|
||||||
|
)
|
||||||
|
|
||||||
return ExitCode.PARTIAL if processing_errors else ExitCode.SUCCESS
|
return ExitCode.PARTIAL if processing_errors else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
parser = evaluation_parser("Extract exam and solution code via Gemini")
|
parser = evaluation_parser("Extract exam and solution code via Gemini")
|
||||||
|
actions = parser.add_mutually_exclusive_group()
|
||||||
|
actions.add_argument("--groups-only", action="store_true",
|
||||||
|
help="Regroup existing questions with Gemini; update only label_groups")
|
||||||
|
actions.add_argument("--persp-only", action="store_true",
|
||||||
|
help="Replace only Persp with Gemini rubrics for existing groups")
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--restart",
|
"--restart",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
@@ -841,7 +941,9 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
return execute(
|
return execute(
|
||||||
parser,
|
parser,
|
||||||
argv,
|
argv,
|
||||||
lambda args: process_exam(
|
lambda args: refine_existing(workspace_from_args(args),
|
||||||
|
"groups" if args.groups_only else "persp")
|
||||||
|
if args.groups_only or args.persp_only else process_exam(
|
||||||
workspace_from_args(args),
|
workspace_from_args(args),
|
||||||
restart=args.restart,
|
restart=args.restart,
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -76,6 +76,8 @@ be missing.
|
|||||||
|
|
||||||
##wrong_labels##
|
##wrong_labels##
|
||||||
|
|
||||||
|
##wrong_label_text_context##
|
||||||
|
|
||||||
Here's a list of the names of the students, pick the one that matches
|
Here's a list of the names of the students, pick the one that matches
|
||||||
the best or `\"Unknown\"` if you cannot read the name
|
the best or `\"Unknown\"` if you cannot read the name
|
||||||
|
|
||||||
@@ -133,6 +135,8 @@ be missing.
|
|||||||
|
|
||||||
##wrong_labels##
|
##wrong_labels##
|
||||||
|
|
||||||
|
##wrong_label_text_context##
|
||||||
|
|
||||||
Since this copy isn't the first part of a sequence, simply set the
|
Since this copy isn't the first part of a sequence, simply set the
|
||||||
name to `\"Continued\"`."""
|
name to `\"Continued\"`."""
|
||||||
|
|
||||||
@@ -147,7 +151,66 @@ class AnnotationData(BaseModel):
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def generate_request(file, labels, names, context_labels, wrong_labels):
|
TEXT_CONTEXT_MAX_CHARS = 4000
|
||||||
|
|
||||||
|
|
||||||
|
def _label_filename(path: Path) -> str:
|
||||||
|
return path.stem if path.suffix.casefold() in {".tex", ".txt"} else path.name
|
||||||
|
|
||||||
|
|
||||||
|
def _common_prefix_length(left: str, right: str) -> int:
|
||||||
|
left_folded = left.casefold()
|
||||||
|
right_folded = right.casefold()
|
||||||
|
limit = min(len(left_folded), len(right_folded))
|
||||||
|
for index in range(limit):
|
||||||
|
if left_folded[index] != right_folded[index]:
|
||||||
|
return index
|
||||||
|
return limit
|
||||||
|
|
||||||
|
|
||||||
|
def closest_text_context(
|
||||||
|
workspace: EvaluationWorkspace, wrong_labels: list[str]
|
||||||
|
) -> tuple[Path | None, str]:
|
||||||
|
"""Return a bounded excerpt from the Text file closest to an invalid label."""
|
||||||
|
text_dir = workspace.root / "Text"
|
||||||
|
if not wrong_labels or not text_dir.is_dir():
|
||||||
|
return None, ""
|
||||||
|
|
||||||
|
ranked: list[tuple[int, str, Path]] = []
|
||||||
|
for path in text_dir.iterdir():
|
||||||
|
if not path.is_file() or path.suffix.casefold() == ".pdf":
|
||||||
|
continue
|
||||||
|
filename = _label_filename(path)
|
||||||
|
prefix_length = max(
|
||||||
|
_common_prefix_length(filename, wrong_label)
|
||||||
|
for wrong_label in wrong_labels
|
||||||
|
)
|
||||||
|
if prefix_length:
|
||||||
|
ranked.append((prefix_length, filename.casefold(), path))
|
||||||
|
|
||||||
|
for _prefix_length, _filename, path in sorted(
|
||||||
|
ranked, key=lambda item: (-item[0], item[1])
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
content = path.read_text(encoding="utf-8")
|
||||||
|
except (OSError, UnicodeError):
|
||||||
|
continue
|
||||||
|
if len(content) > TEXT_CONTEXT_MAX_CHARS:
|
||||||
|
content = content[:TEXT_CONTEXT_MAX_CHARS] + "\n[excerpt truncated]"
|
||||||
|
return path, content
|
||||||
|
return None, ""
|
||||||
|
|
||||||
|
|
||||||
|
def generate_request(
|
||||||
|
file,
|
||||||
|
labels,
|
||||||
|
names,
|
||||||
|
context_labels,
|
||||||
|
wrong_labels,
|
||||||
|
wrong_label_text_context="",
|
||||||
|
wrong_label_text_file: Path | None = None,
|
||||||
|
seed: int = 0,
|
||||||
|
):
|
||||||
"""Generates request for Gemini with context."""
|
"""Generates request for Gemini with context."""
|
||||||
|
|
||||||
image_path = Path(file)
|
image_path = Path(file)
|
||||||
@@ -162,9 +225,36 @@ def generate_request(file, labels, names, context_labels, wrong_labels):
|
|||||||
text = my_prompt2.replace("##labels##", labels)\
|
text = my_prompt2.replace("##labels##", labels)\
|
||||||
.replace("##prev_context##", context_str)
|
.replace("##prev_context##", context_str)
|
||||||
if wrong_labels:
|
if wrong_labels:
|
||||||
text= text.replace("##wrong_labels##\n\n", f"On a previous request, you answered with the following wrong labels : {wrong_labels}. These are wrong, since they do not exactly match any of the labels in the previous list.")
|
formatted_wrong_labels = "\n".join(f'- "{label}"' for label in wrong_labels)
|
||||||
|
text = text.replace(
|
||||||
|
"##wrong_labels##",
|
||||||
|
"On the previous request for this image, you answered with these "
|
||||||
|
"invalid labels:\n"
|
||||||
|
f"{formatted_wrong_labels}\n"
|
||||||
|
"They are wrong because they do not exactly match any label in the "
|
||||||
|
"valid list above.\n\n"
|
||||||
|
"CRITICAL RETRY CONSTRAINT: NEVER return any of the invalid labels "
|
||||||
|
"listed above again. Your answer must use only exact labels copied "
|
||||||
|
"verbatim from the valid list. If the handwriting resembles an "
|
||||||
|
"invalid label, choose the closest exact valid label instead.",
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
text = text.replace("##wrong_labels##\n\n", "")
|
text = text.replace("##wrong_labels##", "")
|
||||||
|
|
||||||
|
if wrong_label_text_context and wrong_label_text_file:
|
||||||
|
text = text.replace(
|
||||||
|
"##wrong_label_text_context##",
|
||||||
|
"Here is an excerpt from the exam text file whose name has the "
|
||||||
|
"longest prefix in common with the invalid label(s), "
|
||||||
|
f"`{wrong_label_text_file.name}`:\n\n"
|
||||||
|
"<exam_text_excerpt>\n"
|
||||||
|
f"{wrong_label_text_context}\n"
|
||||||
|
"</exam_text_excerpt>\n\n"
|
||||||
|
"Use this excerpt as extra context for identifying the handwritten "
|
||||||
|
"label, but return only an exact label from the valid list above.",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
text = text.replace("##wrong_label_text_context##", "")
|
||||||
|
|
||||||
|
|
||||||
contents = [
|
contents = [
|
||||||
@@ -181,9 +271,10 @@ def generate_request(file, labels, names, context_labels, wrong_labels):
|
|||||||
]
|
]
|
||||||
|
|
||||||
generate_content_config = types.GenerateContentConfig(
|
generate_content_config = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=1.0,
|
temperature=1.0,
|
||||||
top_p=0.95,
|
top_p=0.95,
|
||||||
seed=0,
|
seed=seed,
|
||||||
max_output_tokens=65535,
|
max_output_tokens=65535,
|
||||||
response_mime_type= "application/json",
|
response_mime_type= "application/json",
|
||||||
response_json_schema= AnnotationData.model_json_schema(),
|
response_json_schema= AnnotationData.model_json_schema(),
|
||||||
@@ -249,6 +340,34 @@ def group_images(image_files: list[Path]) -> dict[str, list[Path]]:
|
|||||||
return dict(groups)
|
return dict(groups)
|
||||||
|
|
||||||
|
|
||||||
|
def sort_boxes_for_image(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
image_file: Path,
|
||||||
|
boxes: list[BoxItem],
|
||||||
|
) -> list[BoxItem]:
|
||||||
|
"""Sort boxes in the image's page-column reading order when schema exists."""
|
||||||
|
match = re.match(r"(.+)_(\d+)$", image_file.stem)
|
||||||
|
if not match:
|
||||||
|
return boxes
|
||||||
|
schema_path = workspace.cutleft_dir / f"{match.group(1)}_schema.json"
|
||||||
|
try:
|
||||||
|
schema = read_json(schema_path)
|
||||||
|
columns_per_file = schema["columns_per_file"]
|
||||||
|
column_count = int(columns_per_file[int(match.group(2)) - 1])
|
||||||
|
if column_count < 1:
|
||||||
|
return boxes
|
||||||
|
except (OSError, KeyError, IndexError, TypeError, ValueError):
|
||||||
|
return boxes
|
||||||
|
|
||||||
|
def position(item: BoxItem) -> tuple[int, int, int]:
|
||||||
|
ymin, xmin, _ymax, xmax = item.box_2d
|
||||||
|
center_x = (xmin + xmax) // 2
|
||||||
|
column = min(column_count - 1, max(0, center_x * column_count // 1000))
|
||||||
|
return column, ymin, xmin
|
||||||
|
|
||||||
|
return sorted(boxes, key=position)
|
||||||
|
|
||||||
|
|
||||||
def _existing_context(output_json: Path) -> list[str]:
|
def _existing_context(output_json: Path) -> list[str]:
|
||||||
try:
|
try:
|
||||||
loaded = read_json(output_json)
|
loaded = read_json(output_json)
|
||||||
@@ -293,17 +412,30 @@ def process_copy_group(
|
|||||||
f"{len(accumulated_labels)} accumulated labels..."
|
f"{len(accumulated_labels)} accumulated labels..."
|
||||||
)
|
)
|
||||||
attempt = 0
|
attempt = 0
|
||||||
|
label_retry_count = 0
|
||||||
wrong_labels: list[str] = []
|
wrong_labels: list[str] = []
|
||||||
while True:
|
while True:
|
||||||
if attempt > 0:
|
if attempt > 0:
|
||||||
sleep(10 * attempt)
|
sleep(10 * attempt)
|
||||||
try:
|
try:
|
||||||
|
text_context_file, text_context = closest_text_context(
|
||||||
|
workspace, wrong_labels
|
||||||
|
)
|
||||||
|
if text_context_file:
|
||||||
|
print(
|
||||||
|
f"[{group_key}] Retry context for {image_file.name}: "
|
||||||
|
f"{text_context_file.relative_to(workspace.root)}"
|
||||||
|
)
|
||||||
|
request_seed = max(0, label_retry_count - 1)
|
||||||
contents, request_config = generate_request(
|
contents, request_config = generate_request(
|
||||||
image_file,
|
image_file,
|
||||||
labels_text,
|
labels_text,
|
||||||
names_text,
|
names_text,
|
||||||
accumulated_labels,
|
accumulated_labels,
|
||||||
wrong_labels,
|
wrong_labels,
|
||||||
|
text_context,
|
||||||
|
text_context_file,
|
||||||
|
seed=request_seed,
|
||||||
)
|
)
|
||||||
response = client.models.generate_content(
|
response = client.models.generate_content(
|
||||||
model=MODEL_ID,
|
model=MODEL_ID,
|
||||||
@@ -321,9 +453,23 @@ def process_copy_group(
|
|||||||
f"Error: {image_file.name} contained unknown labels: "
|
f"Error: {image_file.name} contained unknown labels: "
|
||||||
f"{unknown}"
|
f"{unknown}"
|
||||||
)
|
)
|
||||||
wrong_labels.extend(unknown)
|
unique_unknown = list(dict.fromkeys(unknown))
|
||||||
attempt += 1
|
if (
|
||||||
continue
|
label_retry_count >= 2
|
||||||
|
and set(unique_unknown) == set(wrong_labels)
|
||||||
|
):
|
||||||
|
for item in annotation.list:
|
||||||
|
if item.label in unique_unknown:
|
||||||
|
item.label = f"??{item.label}"
|
||||||
|
print(
|
||||||
|
f"Warning: {image_file.name} repeated the same unknown "
|
||||||
|
"label(s) on the third try; keeping them with a ?? prefix."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
wrong_labels = unique_unknown
|
||||||
|
label_retry_count += 1
|
||||||
|
attempt += 1
|
||||||
|
continue
|
||||||
if annotation.name not in valid_names:
|
if annotation.name not in valid_names:
|
||||||
print(
|
print(
|
||||||
f"Error: {image_file.name} returned unknown name: "
|
f"Error: {image_file.name} returned unknown name: "
|
||||||
@@ -334,6 +480,9 @@ def process_copy_group(
|
|||||||
continue
|
continue
|
||||||
annotation.name = "Unknown"
|
annotation.name = "Unknown"
|
||||||
|
|
||||||
|
annotation.list = sort_boxes_for_image(
|
||||||
|
workspace, image_file, annotation.list
|
||||||
|
)
|
||||||
atomic_write_json(output_json, annotation.model_dump())
|
atomic_write_json(output_json, annotation.model_dump())
|
||||||
accumulated_labels.extend(box.label for box in annotation.list)
|
accumulated_labels.extend(box.label for box in annotation.list)
|
||||||
generated += 1
|
generated += 1
|
||||||
|
|||||||
@@ -10,12 +10,14 @@ from pathlib import Path
|
|||||||
from copienator import (
|
from copienator import (
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
|
configuration,
|
||||||
evaluation_parser,
|
evaluation_parser,
|
||||||
execute,
|
execute,
|
||||||
read_json,
|
read_json,
|
||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
from copienator.platform import replace_with_link_or_copy, safe_filename
|
from copienator.platform import replace_with_link_or_copy, safe_filename
|
||||||
|
from copienator.return_answers import publish_answer_returns
|
||||||
|
|
||||||
ANNOTATION_CHOICES = ("BGnot", "Bnot", "Anot")
|
ANNOTATION_CHOICES = ("BGnot", "Bnot", "Anot")
|
||||||
|
|
||||||
@@ -27,9 +29,20 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
choices=ANNOTATION_CHOICES,
|
choices=ANNOTATION_CHOICES,
|
||||||
help="Annotation directory to use",
|
help="Annotation directory to use",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--update",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"Update only the individual images in existing A Rendre/answers "
|
||||||
|
"directories, matching folders by their trailing copy ID"
|
||||||
|
),
|
||||||
|
)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
RETURN_COPY_ID = re.compile(r"\((\d+)\)$")
|
||||||
|
|
||||||
|
|
||||||
def _read_expected_names(workspace: EvaluationWorkspace) -> set[str]:
|
def _read_expected_names(workspace: EvaluationWorkspace) -> set[str]:
|
||||||
names_path = workspace.names_file()
|
names_path = workspace.names_file()
|
||||||
if not names_path.exists():
|
if not names_path.exists():
|
||||||
@@ -45,6 +58,70 @@ def _read_expected_names(workspace: EvaluationWorkspace) -> set[str]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _annotation_source(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
annotation_dir_name: str,
|
||||||
|
copy_id: str,
|
||||||
|
) -> Path | None:
|
||||||
|
selected = workspace.root / annotation_dir_name / f"Copie{copy_id}"
|
||||||
|
fallback = workspace.annotation_dir("simple") / f"Copie{copy_id}"
|
||||||
|
for candidate in (selected, fallback):
|
||||||
|
if (candidate / "score.json").is_file() and (
|
||||||
|
(candidate / "Concat.jpg").is_file()
|
||||||
|
or (candidate / "info.json").is_file()
|
||||||
|
):
|
||||||
|
return candidate
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def update_named_return_answers(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
annotation_dir_name: str,
|
||||||
|
) -> ExitCode:
|
||||||
|
"""Refresh only answers/ in existing returns, preserving manual names."""
|
||||||
|
workspace.require_directories(annotation_dir_name, "A Rendre")
|
||||||
|
had_errors = False
|
||||||
|
found = False
|
||||||
|
for destination in sorted(workspace.return_dir.iterdir()):
|
||||||
|
if not destination.is_dir():
|
||||||
|
continue
|
||||||
|
match = RETURN_COPY_ID.search(destination.name)
|
||||||
|
if match is None:
|
||||||
|
print(
|
||||||
|
f"Warning: cannot identify a copy ID in {destination.name!r}; skipped",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
had_errors = True
|
||||||
|
continue
|
||||||
|
found = True
|
||||||
|
copy_id = match.group(1)
|
||||||
|
source_folder = _annotation_source(
|
||||||
|
workspace, annotation_dir_name, copy_id
|
||||||
|
)
|
||||||
|
if source_folder is None:
|
||||||
|
print(
|
||||||
|
f"Warning: no annotation source found for Copie{copy_id}; skipped",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
had_errors = True
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
publish_answer_returns(
|
||||||
|
workspace.root,
|
||||||
|
source_folder,
|
||||||
|
destination,
|
||||||
|
answers_only=True,
|
||||||
|
)
|
||||||
|
print(f"Updated answers for {destination.name} from Copie{copy_id}")
|
||||||
|
except (OSError, TypeError, ValueError) as exc:
|
||||||
|
print(f"Error updating answers for {destination.name}: {exc}", file=sys.stderr)
|
||||||
|
had_errors = True
|
||||||
|
if not found:
|
||||||
|
print("Warning: no identifiable student folders found in A Rendre", file=sys.stderr)
|
||||||
|
had_errors = True
|
||||||
|
return ExitCode.PARTIAL if had_errors else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
def prepare_named_returns(
|
def prepare_named_returns(
|
||||||
workspace: EvaluationWorkspace,
|
workspace: EvaluationWorkspace,
|
||||||
annotation_dir_name: str,
|
annotation_dir_name: str,
|
||||||
@@ -72,9 +149,6 @@ def prepare_named_returns(
|
|||||||
had_errors = True
|
had_errors = True
|
||||||
|
|
||||||
assigned_names: set[str] = set()
|
assigned_names: set[str] = set()
|
||||||
selected_annotations = workspace.root / annotation_dir_name
|
|
||||||
fallback_annotations = workspace.annotation_dir("simple")
|
|
||||||
|
|
||||||
for name, copy_ids in copies_map.items():
|
for name, copy_ids in copies_map.items():
|
||||||
if name == "Unknown":
|
if name == "Unknown":
|
||||||
print(
|
print(
|
||||||
@@ -90,34 +164,40 @@ def prepare_named_returns(
|
|||||||
|
|
||||||
safe_name = safe_filename(name)
|
safe_name = safe_filename(name)
|
||||||
for copy_id in copy_ids:
|
for copy_id in copy_ids:
|
||||||
selected = selected_annotations / f"Copie{copy_id}"
|
source_folder = _annotation_source(
|
||||||
fallback = fallback_annotations / f"Copie{copy_id}"
|
workspace, annotation_dir_name, copy_id
|
||||||
source_folder = None
|
)
|
||||||
for candidate in (selected, fallback):
|
|
||||||
if (candidate / "Concat.jpg").exists() and (
|
|
||||||
candidate / "score.json"
|
|
||||||
).exists():
|
|
||||||
source_folder = candidate
|
|
||||||
break
|
|
||||||
if source_folder is None:
|
if source_folder is None:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
assigned_names.add(name)
|
assigned_names.add(name)
|
||||||
destination = workspace.return_dir / f"{safe_name} ({copy_id})"
|
destination = workspace.return_dir / f"{safe_name} ({copy_id})"
|
||||||
destination.mkdir(parents=True, exist_ok=True)
|
destination.mkdir(parents=True, exist_ok=True)
|
||||||
|
try:
|
||||||
|
publish_answer_returns(workspace.root, source_folder, destination)
|
||||||
|
except (OSError, TypeError, ValueError) as exc:
|
||||||
|
print(f"Error preparing answers for {destination.name}: {exc}", file=sys.stderr)
|
||||||
|
had_errors = True
|
||||||
|
continue
|
||||||
links = (
|
links = (
|
||||||
("Concat.jpg", f"{safe_name}.jpg"),
|
("Concat.jpg", f"{safe_name}.jpg", configuration.RETURN_JPEG_ENABLED),
|
||||||
("Concat_F.pdf", f"{safe_name}.pdf"),
|
("Concat_F.pdf", f"{safe_name}.pdf", configuration.RETURN_PDF_ENABLED),
|
||||||
("score.json", "score.json"),
|
("score.json", "score.json", True),
|
||||||
)
|
)
|
||||||
for source_name, destination_name in links:
|
for source_name, destination_name, enabled in links:
|
||||||
source = source_folder / source_name
|
source = source_folder / source_name
|
||||||
if not source.exists():
|
target = destination / destination_name
|
||||||
continue
|
|
||||||
try:
|
try:
|
||||||
|
if not enabled:
|
||||||
|
# Remove only the named return entry, never its link target.
|
||||||
|
target.unlink(missing_ok=True)
|
||||||
|
continue
|
||||||
|
if not source.exists():
|
||||||
|
target.unlink(missing_ok=True)
|
||||||
|
continue
|
||||||
method = replace_with_link_or_copy(
|
method = replace_with_link_or_copy(
|
||||||
source,
|
source,
|
||||||
destination / destination_name,
|
target,
|
||||||
prefer="symlink",
|
prefer="symlink",
|
||||||
)
|
)
|
||||||
if method == "copy":
|
if method == "copy":
|
||||||
@@ -145,7 +225,10 @@ def run(
|
|||||||
workspace: EvaluationWorkspace,
|
workspace: EvaluationWorkspace,
|
||||||
*,
|
*,
|
||||||
annotation_dir: str,
|
annotation_dir: str,
|
||||||
|
update: bool = False,
|
||||||
) -> ExitCode:
|
) -> ExitCode:
|
||||||
|
if update:
|
||||||
|
return update_named_return_answers(workspace, annotation_dir)
|
||||||
return prepare_named_returns(workspace, annotation_dir)
|
return prepare_named_returns(workspace, annotation_dir)
|
||||||
|
|
||||||
|
|
||||||
@@ -157,10 +240,10 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
lambda args: run(
|
lambda args: run(
|
||||||
workspace_from_args(args, repository=Path.cwd()),
|
workspace_from_args(args, repository=Path.cwd()),
|
||||||
annotation_dir=args.annotation_dir,
|
annotation_dir=args.annotation_dir,
|
||||||
|
update=args.update,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
raise SystemExit(main())
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ from collections.abc import Sequence
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from tkinter import messagebox
|
from tkinter import messagebox
|
||||||
|
|
||||||
import fitz # PyMuPDF
|
import pymupdf # PyMuPDF
|
||||||
from PIL import Image, ImageDraw, ImageTk
|
from PIL import Image, ImageDraw, ImageTk
|
||||||
from pypdf import PdfReader, PdfWriter
|
from pypdf import PdfReader, PdfWriter
|
||||||
|
|
||||||
@@ -26,6 +26,10 @@ from copienator import (
|
|||||||
workspace_from_target,
|
workspace_from_target,
|
||||||
)
|
)
|
||||||
from copienator.platform import launch_pdf_arranger
|
from copienator.platform import launch_pdf_arranger
|
||||||
|
from copienator.copy_errors import marked_copy_paths
|
||||||
|
|
||||||
|
# Keep the new shortcut available with older personal configuration files.
|
||||||
|
PAGE_SPLITTER_KB = {"reverse_pages": "i", **PAGE_SPLITTER_KB}
|
||||||
|
|
||||||
# --- Constants ---
|
# --- Constants ---
|
||||||
# Conversion factor: 1 cm to points (1 inch = 2.54 cm, 72 points = 1 inch)
|
# Conversion factor: 1 cm to points (1 inch = 2.54 cm, 72 points = 1 inch)
|
||||||
@@ -112,7 +116,7 @@ class PDFPreviewer:
|
|||||||
self.page_settings = []
|
self.page_settings = []
|
||||||
self.processing = False # Flag to prevent multiple finish calls
|
self.processing = False # Flag to prevent multiple finish calls
|
||||||
try:
|
try:
|
||||||
self.doc = fitz.open(self.pdf_path)
|
self.doc = pymupdf.open(self.pdf_path)
|
||||||
except (OSError, RuntimeError, ValueError) as e:
|
except (OSError, RuntimeError, ValueError) as e:
|
||||||
self.failed = True
|
self.failed = True
|
||||||
self._temporary_directory.cleanup()
|
self._temporary_directory.cleanup()
|
||||||
@@ -166,6 +170,7 @@ class PDFPreviewer:
|
|||||||
f"'{fmt('rotate_page')}': Rotate page 180°, '{fmt('rotate_all_pages')}' : rotate all pages, '{fmt('rotate_all_files')}' : rotate all files\n"
|
f"'{fmt('rotate_page')}': Rotate page 180°, '{fmt('rotate_all_pages')}' : rotate all pages, '{fmt('rotate_all_files')}' : rotate all files\n"
|
||||||
f"{fmt('keep_left')} {fmt('next_page')} {fmt('discard_page')} {fmt('keep_right')} {fmt('keep_as_is')}: keep left, next page, keep none, keep right, keep as is\n"
|
f"{fmt('keep_left')} {fmt('next_page')} {fmt('discard_page')} {fmt('keep_right')} {fmt('keep_as_is')}: keep left, next page, keep none, keep right, keep as is\n"
|
||||||
f"{fmt('send_end')}: send page to end, '{fmt('arranger')}': pdf arranger, '{fmt('restart_file')}': restart file, '{fmt('prev_file')}': previous file\n"
|
f"{fmt('send_end')}: send page to end, '{fmt('arranger')}': pdf arranger, '{fmt('restart_file')}': restart file, '{fmt('prev_file')}': previous file\n"
|
||||||
|
f"'{fmt('reverse_pages')}': reverse page order and restart from the new first page\n"
|
||||||
)
|
)
|
||||||
|
|
||||||
self.info_label = tk.Label(master, text=instructions, justify=tk.LEFT)
|
self.info_label = tk.Label(master, text=instructions, justify=tk.LEFT)
|
||||||
@@ -192,6 +197,7 @@ class PDFPreviewer:
|
|||||||
"next_page": self.confirm_and_next_page,
|
"next_page": self.confirm_and_next_page,
|
||||||
"discard_page": self.discard_page,
|
"discard_page": self.discard_page,
|
||||||
"send_end": self.send_page_end,
|
"send_end": self.send_page_end,
|
||||||
|
"reverse_pages": self.reverse_pages,
|
||||||
"restart_file": self.restart_current_file,
|
"restart_file": self.restart_current_file,
|
||||||
"arranger": self.start_arranger,
|
"arranger": self.start_arranger,
|
||||||
"prev_file": self.go_to_previous_file,
|
"prev_file": self.go_to_previous_file,
|
||||||
@@ -271,7 +277,7 @@ class PDFPreviewer:
|
|||||||
self.current_zoom = min(zoom_x, zoom_y) * 0.98
|
self.current_zoom = min(zoom_x, zoom_y) * 0.98
|
||||||
|
|
||||||
# --- Render Page ---
|
# --- Render Page ---
|
||||||
mat = fitz.Matrix(self.current_zoom, self.current_zoom)
|
mat = pymupdf.Matrix(self.current_zoom, self.current_zoom)
|
||||||
pix = page.get_pixmap(matrix=mat, alpha=False)
|
pix = page.get_pixmap(matrix=mat, alpha=False)
|
||||||
img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
|
img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
|
||||||
|
|
||||||
@@ -302,7 +308,7 @@ class PDFPreviewer:
|
|||||||
|
|
||||||
# Re-open the file from disk to reset changes (like moved pages)
|
# Re-open the file from disk to reset changes (like moved pages)
|
||||||
try:
|
try:
|
||||||
self.doc = fitz.open(self.pdf_path)
|
self.doc = pymupdf.open(self.pdf_path)
|
||||||
except (OSError, RuntimeError, ValueError) as e:
|
except (OSError, RuntimeError, ValueError) as e:
|
||||||
messagebox.showerror("Error", f"Failed to reopen PDF file: {e}")
|
messagebox.showerror("Error", f"Failed to reopen PDF file: {e}")
|
||||||
self.master.destroy()
|
self.master.destroy()
|
||||||
@@ -319,6 +325,16 @@ class PDFPreviewer:
|
|||||||
self.load_page()
|
self.load_page()
|
||||||
|
|
||||||
|
|
||||||
|
def reverse_pages(self, event=None):
|
||||||
|
"""Reverse the current document and discard earlier page decisions."""
|
||||||
|
if self.processing or not len(self.doc):
|
||||||
|
return
|
||||||
|
self.doc.select(list(reversed(range(len(self.doc)))))
|
||||||
|
self.current_page_index = 0
|
||||||
|
self.page_settings = []
|
||||||
|
self._initialize_current_page_settings()
|
||||||
|
self.load_page()
|
||||||
|
|
||||||
def move_line_left(self, event=None):
|
def move_line_left(self, event=None):
|
||||||
"""Moves the split line to the left."""
|
"""Moves the split line to the left."""
|
||||||
self.current_line_x = max(0, self.current_line_x - CM_TO_POINTS / 2)
|
self.current_line_x = max(0, self.current_line_x - CM_TO_POINTS / 2)
|
||||||
@@ -465,7 +481,7 @@ class PDFPreviewer:
|
|||||||
self.file_rotation + self.global_rotation) % 360
|
self.file_rotation + self.global_rotation) % 360
|
||||||
|
|
||||||
if keep == "as_is":
|
if keep == "as_is":
|
||||||
doc_full = fitz.open()
|
doc_full = pymupdf.open()
|
||||||
page_full = doc_full.new_page(width=page.rect.width, height=page.rect.height)
|
page_full = doc_full.new_page(width=page.rect.width, height=page.rect.height)
|
||||||
page_full.show_pdf_page(page_full.rect, self.doc, i)
|
page_full.show_pdf_page(page_full.rect, self.doc, i)
|
||||||
page_full.set_rotation(rotation)
|
page_full.set_rotation(rotation)
|
||||||
@@ -477,13 +493,13 @@ class PDFPreviewer:
|
|||||||
|
|
||||||
# --- Create Left Part ---
|
# --- Create Left Part ---
|
||||||
if rotation == 0:
|
if rotation == 0:
|
||||||
rect_left = fitz.Rect(0, 0, line_x, page.rect.height)
|
rect_left = pymupdf.Rect(0, 0, line_x, page.rect.height)
|
||||||
else:
|
else:
|
||||||
rect_left = fitz.Rect(page.rect.width-line_x, 0, page.rect.width, page.rect.height)
|
rect_left = pymupdf.Rect(page.rect.width-line_x, 0, page.rect.width, page.rect.height)
|
||||||
|
|
||||||
if (keep == "both" or keep == "left") and line_x > 0:
|
if (keep == "both" or keep == "left") and line_x > 0:
|
||||||
|
|
||||||
doc_left = fitz.open()
|
doc_left = pymupdf.open()
|
||||||
page_left = doc_left.new_page(width=rect_left.width, height=rect_left.height)
|
page_left = doc_left.new_page(width=rect_left.width, height=rect_left.height)
|
||||||
page_left.show_pdf_page(page_left.rect, self.doc, i, clip=rect_left)
|
page_left.show_pdf_page(page_left.rect, self.doc, i, clip=rect_left)
|
||||||
page_left.set_rotation(rotation)
|
page_left.set_rotation(rotation)
|
||||||
@@ -494,11 +510,11 @@ class PDFPreviewer:
|
|||||||
|
|
||||||
# --- Create Right Part ---
|
# --- Create Right Part ---
|
||||||
if rotation == 0:
|
if rotation == 0:
|
||||||
rect_right = fitz.Rect(line_x, 0, page.rect.width, page.rect.height)
|
rect_right = pymupdf.Rect(line_x, 0, page.rect.width, page.rect.height)
|
||||||
else:
|
else:
|
||||||
rect_right = fitz.Rect(0, 0, page.rect.width-line_x, page.rect.height)
|
rect_right = pymupdf.Rect(0, 0, page.rect.width-line_x, page.rect.height)
|
||||||
if (keep == "both" or keep == "right") and line_x < page.rect.width:
|
if (keep == "both" or keep == "right") and line_x < page.rect.width:
|
||||||
doc_right = fitz.open()
|
doc_right = pymupdf.open()
|
||||||
page_right = doc_right.new_page(width=rect_right.width, height=rect_right.height)
|
page_right = doc_right.new_page(width=rect_right.width, height=rect_right.height)
|
||||||
page_right.show_pdf_page(page_right.rect, self.doc, i, clip=rect_right)
|
page_right.show_pdf_page(page_right.rect, self.doc, i, clip=rect_right)
|
||||||
page_right.set_rotation(rotation)
|
page_right.set_rotation(rotation)
|
||||||
@@ -632,8 +648,8 @@ def _selected_inputs(
|
|||||||
return list(reversed(candidates))
|
return list(reversed(candidates))
|
||||||
|
|
||||||
|
|
||||||
def run(workspace: EvaluationWorkspace, target: Path) -> ExitCode:
|
def run(workspace: EvaluationWorkspace, target: Path, *, marked: bool = False) -> ExitCode:
|
||||||
inputs = _selected_inputs(workspace, target)
|
inputs = list(reversed(marked_copy_paths(workspace, originals=True))) if marked else _selected_inputs(workspace, target)
|
||||||
if not inputs:
|
if not inputs:
|
||||||
print(f"No PDF files found in {target}")
|
print(f"No PDF files found in {target}")
|
||||||
return ExitCode.SUCCESS
|
return ExitCode.SUCCESS
|
||||||
@@ -644,7 +660,9 @@ def run(workspace: EvaluationWorkspace, target: Path) -> ExitCode:
|
|||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
return target_parser("Interactively split and reorder scanned PDF pages")
|
parser = target_parser("Interactively split and reorder scanned PDF pages")
|
||||||
|
parser.add_argument("--marked", action="store_true", help="Process only copies flagged during margin review")
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
def main(argv: Sequence[str] | None = None) -> int:
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
@@ -652,7 +670,7 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
|
|
||||||
def handle(args: argparse.Namespace) -> ExitCode:
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
workspace, target = workspace_from_target(args)
|
workspace, target = workspace_from_target(args)
|
||||||
return run(workspace, target)
|
return run(workspace, target, marked=args.marked)
|
||||||
|
|
||||||
return execute(parser, argv, handle)
|
return execute(parser, argv, handle)
|
||||||
|
|
||||||
|
|||||||
+115
-36
@@ -1,7 +1,9 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import copy
|
||||||
import queue
|
import queue
|
||||||
|
import re
|
||||||
import threading
|
import threading
|
||||||
import tkinter as tk
|
import tkinter as tk
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
@@ -24,6 +26,9 @@ from copienator.utils import natural_key, read_all_labels
|
|||||||
|
|
||||||
# --- Configuration & Globals ---
|
# --- Configuration & Globals ---
|
||||||
padding = 60
|
padding = 60
|
||||||
|
MISSING_LABEL_COLOR = "orange"
|
||||||
|
COMMON_MISSING_LABEL_COLOR = "#403a00"
|
||||||
|
COMMON_MISSING_THRESHOLD = 0.66
|
||||||
|
|
||||||
try:
|
try:
|
||||||
font = ImageFont.truetype("DejaVuSans.ttf", size=30)
|
font = ImageFont.truetype("DejaVuSans.ttf", size=30)
|
||||||
@@ -81,13 +86,81 @@ def normalized_labels(entries):
|
|||||||
if str(value["label"]) != "_"
|
if str(value["label"]) != "_"
|
||||||
]
|
]
|
||||||
|
|
||||||
def prepare_image(image_path: str, bounding_boxes, all_labels, nb_pages, last_label_index):
|
|
||||||
|
def frequently_missing_labels(
|
||||||
|
copies_dir: Path,
|
||||||
|
all_labels: list[str],
|
||||||
|
threshold: float = COMMON_MISSING_THRESHOLD,
|
||||||
|
) -> set[str]:
|
||||||
|
"""Return labels absent from at least ``threshold`` of detected copies."""
|
||||||
|
labels_by_copy: dict[str, set[str]] = {}
|
||||||
|
for json_path in copies_dir.glob("*.json"):
|
||||||
|
match = re.fullmatch(r"(.+)_\d+", json_path.stem)
|
||||||
|
if match is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
data = read_json(json_path)
|
||||||
|
entries = data["list"]
|
||||||
|
if not isinstance(entries, list):
|
||||||
|
continue
|
||||||
|
present = set(normalized_labels(entries))
|
||||||
|
except (OSError, KeyError, TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
labels_by_copy.setdefault(match.group(1), set()).update(present)
|
||||||
|
|
||||||
|
copy_count = len(labels_by_copy)
|
||||||
|
if copy_count == 0:
|
||||||
|
return set()
|
||||||
|
return {
|
||||||
|
label
|
||||||
|
for label in all_labels
|
||||||
|
if sum(label not in present for present in labels_by_copy.values())
|
||||||
|
>= threshold * copy_count
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def label_color(
|
||||||
|
label: str | None,
|
||||||
|
all_labels: list[str],
|
||||||
|
last_label_index: int,
|
||||||
|
common_missing: set[str],
|
||||||
|
) -> tuple[str, int]:
|
||||||
|
"""Choose a label color and return the updated chronological index."""
|
||||||
|
color = "black"
|
||||||
|
if not label or label not in all_labels:
|
||||||
|
return color, last_label_index
|
||||||
|
|
||||||
|
current_index = all_labels.index(label)
|
||||||
|
if current_index < last_label_index or (
|
||||||
|
last_label_index == -1 and current_index != 0
|
||||||
|
):
|
||||||
|
color = "red"
|
||||||
|
elif current_index > last_label_index + 1:
|
||||||
|
only_previous_is_missing = current_index == last_label_index + 2
|
||||||
|
previous_label = all_labels[current_index - 1]
|
||||||
|
color = (
|
||||||
|
COMMON_MISSING_LABEL_COLOR
|
||||||
|
if only_previous_is_missing and previous_label in common_missing
|
||||||
|
else MISSING_LABEL_COLOR
|
||||||
|
)
|
||||||
|
return color, current_index
|
||||||
|
|
||||||
|
|
||||||
|
def prepare_image(
|
||||||
|
image_path: str,
|
||||||
|
bounding_boxes,
|
||||||
|
all_labels,
|
||||||
|
nb_pages,
|
||||||
|
last_label_index,
|
||||||
|
common_missing: set[str] | None = None,
|
||||||
|
):
|
||||||
im = Image.open(image_path)
|
im = Image.open(image_path)
|
||||||
im.load()
|
im.load()
|
||||||
width, height = im.size
|
width, height = im.size
|
||||||
new_im = Image.new(im.mode, (width + padding, height), "white")
|
new_im = Image.new(im.mode, (width + padding, height), "white")
|
||||||
new_im.paste(im, (0, 0))
|
new_im.paste(im, (0, 0))
|
||||||
draw = ImageDraw.Draw(new_im)
|
draw = ImageDraw.Draw(new_im)
|
||||||
|
common_missing = common_missing or set()
|
||||||
|
|
||||||
for bbox in sort_bounding_boxes(bounding_boxes, nb_pages):
|
for bbox in sort_bounding_boxes(bounding_boxes, nb_pages):
|
||||||
raw_y_min = int(bbox["box_2d"][0] * height / 1000)
|
raw_y_min = int(bbox["box_2d"][0] * height / 1000)
|
||||||
@@ -99,15 +172,13 @@ def prepare_image(image_path: str, bounding_boxes, all_labels, nb_pages, last_la
|
|||||||
abs_y_max = min(height, raw_y_max + 10)
|
abs_y_max = min(height, raw_y_max + 10)
|
||||||
abs_x_max = min(width, raw_x_max + 10)
|
abs_x_max = min(width, raw_x_max + 10)
|
||||||
|
|
||||||
color = "black"
|
|
||||||
label = bbox.get("label")
|
label = bbox.get("label")
|
||||||
if label and label in all_labels:
|
color, last_label_index = label_color(
|
||||||
current_index = all_labels.index(label)
|
label,
|
||||||
if current_index < last_label_index or (last_label_index == -1 and current_index != 0):
|
all_labels,
|
||||||
color = "red"
|
last_label_index,
|
||||||
elif current_index > last_label_index + 1:
|
common_missing,
|
||||||
color = "orange"
|
)
|
||||||
last_label_index = current_index
|
|
||||||
|
|
||||||
draw.rectangle(((abs_x_min, abs_y_min), (abs_x_max, abs_y_max)), outline=color, width=4)
|
draw.rectangle(((abs_x_min, abs_y_min), (abs_x_max, abs_y_max)), outline=color, width=4)
|
||||||
if label:
|
if label:
|
||||||
@@ -126,6 +197,7 @@ def _worker_items(base_dir, files_to_process, all_labels, output_queue):
|
|||||||
"""
|
"""
|
||||||
previous_copie = None
|
previous_copie = None
|
||||||
last_label_index = None
|
last_label_index = None
|
||||||
|
common_missing = frequently_missing_labels(base_dir / "Copies", all_labels)
|
||||||
for img_path in files_to_process:
|
for img_path in files_to_process:
|
||||||
json_path = base_dir / "Copies" / f"{img_path.stem}.json"
|
json_path = base_dir / "Copies" / f"{img_path.stem}.json"
|
||||||
copie_part = int(img_path.stem[-2:])
|
copie_part = int(img_path.stem[-2:])
|
||||||
@@ -158,7 +230,14 @@ def _worker_items(base_dir, files_to_process, all_labels, output_queue):
|
|||||||
try:
|
try:
|
||||||
print(f"Buffering {img_path.name}...")
|
print(f"Buffering {img_path.name}...")
|
||||||
(pil_image, last_label_index) = \
|
(pil_image, last_label_index) = \
|
||||||
prepare_image(str(img_path), bb_list, all_labels, nb_pages, last_label_index)
|
prepare_image(
|
||||||
|
str(img_path),
|
||||||
|
bb_list,
|
||||||
|
all_labels,
|
||||||
|
nb_pages,
|
||||||
|
last_label_index,
|
||||||
|
common_missing,
|
||||||
|
)
|
||||||
error_msg = None
|
error_msg = None
|
||||||
|
|
||||||
except Exception as e: # noqa: BLE001 - keep the item editable in the GUI
|
except Exception as e: # noqa: BLE001 - keep the item editable in the GUI
|
||||||
@@ -271,7 +350,6 @@ class ImageViewer:
|
|||||||
# Start new batch
|
# Start new batch
|
||||||
self.active_copie_name = metadata["copie"]
|
self.active_copie_name = metadata["copie"]
|
||||||
self.accumulated_results = {"name": metadata["name"], "list": []}
|
self.accumulated_results = {"name": metadata["name"], "list": []}
|
||||||
self.history.clear()
|
|
||||||
|
|
||||||
self.display_image(pil_image, json_path, metadata)
|
self.display_image(pil_image, json_path, metadata)
|
||||||
except queue.Empty:
|
except queue.Empty:
|
||||||
@@ -293,16 +371,23 @@ class ImageViewer:
|
|||||||
def on_previous(self, event):
|
def on_previous(self, event):
|
||||||
if self.is_viewing and self.history:
|
if self.is_viewing and self.history:
|
||||||
print("Going back to previous image...")
|
print("Going back to previous image...")
|
||||||
prev_pil, prev_json, prev_meta, num_added = self.history.pop()
|
(
|
||||||
|
prev_pil,
|
||||||
# Undo the accumulation to prevent duplicates when we hit Enter again
|
prev_json,
|
||||||
if self.accumulated_results and num_added > 0:
|
prev_meta,
|
||||||
self.accumulated_results["list"] = self.accumulated_results["list"][:-num_added]
|
previous_copie_name,
|
||||||
|
previous_results,
|
||||||
|
) = self.history.pop()
|
||||||
|
|
||||||
# Push current image to the forward stack so we don't lose it
|
# Push current image to the forward stack so we don't lose it
|
||||||
self.forward_stack.append((self.current_pil_image,
|
self.forward_stack.append((self.current_pil_image,
|
||||||
self.current_json_path, self.current_meta))
|
self.current_json_path, self.current_meta))
|
||||||
|
|
||||||
|
# Restore the aggregation exactly as it was before validating the
|
||||||
|
# previous image. This also makes navigation across copies safe.
|
||||||
|
self.active_copie_name = previous_copie_name
|
||||||
|
self.accumulated_results = previous_results
|
||||||
|
|
||||||
# Display the previous image immediately
|
# Display the previous image immediately
|
||||||
self.display_image(prev_pil, prev_json, prev_meta)
|
self.display_image(prev_pil, prev_json, prev_meta)
|
||||||
def display_image(self, pil_image, json_path, metadata):
|
def display_image(self, pil_image, json_path, metadata):
|
||||||
@@ -329,32 +414,21 @@ class ImageViewer:
|
|||||||
def on_enter(self, event):
|
def on_enter(self, event):
|
||||||
if self.is_viewing:
|
if self.is_viewing:
|
||||||
print(f"Committing data for {self.current_json_path.name}...")
|
print(f"Committing data for {self.current_json_path.name}...")
|
||||||
num_added = 0 # ADD THIS LINE
|
previous_copie_name = self.active_copie_name
|
||||||
|
previous_results = copy.deepcopy(self.accumulated_results)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
current_data = read_json(self.current_json_path)
|
current_data = read_json(self.current_json_path)
|
||||||
nb_pages = self.current_meta["schema"]["columns_per_file"][
|
items = current_data["list"]
|
||||||
self.current_meta["part"] - 1
|
|
||||||
]
|
|
||||||
original_items = current_data["list"]
|
|
||||||
ordered_items = sort_bounding_boxes(original_items, nb_pages)
|
|
||||||
|
|
||||||
if ordered_items != original_items:
|
|
||||||
current_data["list"] = ordered_items
|
|
||||||
atomic_write_json(self.current_json_path, current_data)
|
|
||||||
print(
|
|
||||||
f"Reordered labels by column in "
|
|
||||||
f"{self.current_json_path.name}."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Perform the conversion now, post-edit
|
# Perform the conversion now, post-edit
|
||||||
converted_items = convert_list(
|
converted_items = convert_list(
|
||||||
ordered_items,
|
items,
|
||||||
self.current_meta["part"],
|
self.current_meta["part"],
|
||||||
self.current_meta["schema"]
|
self.current_meta["schema"]
|
||||||
)
|
)
|
||||||
|
|
||||||
labels = normalized_labels(ordered_items)
|
labels = normalized_labels(items)
|
||||||
false_labels = [
|
false_labels = [
|
||||||
label for label in labels if label not in self.valid_labels
|
label for label in labels if label not in self.valid_labels
|
||||||
]
|
]
|
||||||
@@ -364,8 +438,6 @@ class ImageViewer:
|
|||||||
print(msg)
|
print(msg)
|
||||||
messagebox.showerror("Label Error", msg)
|
messagebox.showerror("Label Error", msg)
|
||||||
return
|
return
|
||||||
num_added = len(converted_items)
|
|
||||||
|
|
||||||
# Add to accumulator
|
# Add to accumulator
|
||||||
if self.accumulated_results:
|
if self.accumulated_results:
|
||||||
self.accumulated_results["list"].extend(converted_items)
|
self.accumulated_results["list"].extend(converted_items)
|
||||||
@@ -380,8 +452,15 @@ class ImageViewer:
|
|||||||
messagebox.showerror("JSON Error", msg)
|
messagebox.showerror("JSON Error", msg)
|
||||||
return # Abort advancement
|
return # Abort advancement
|
||||||
|
|
||||||
self.history.append((self.current_pil_image, self.current_json_path,
|
self.history.append(
|
||||||
self.current_meta, num_added))
|
(
|
||||||
|
self.current_pil_image,
|
||||||
|
self.current_json_path,
|
||||||
|
self.current_meta,
|
||||||
|
previous_copie_name,
|
||||||
|
previous_results,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
# Advance UI
|
# Advance UI
|
||||||
self.is_viewing = False
|
self.is_viewing = False
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ from typing import Any
|
|||||||
from copienator import (
|
from copienator import (
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
|
atomic_write_bytes,
|
||||||
atomic_write_json,
|
atomic_write_json,
|
||||||
evaluation_parser,
|
evaluation_parser,
|
||||||
execute,
|
execute,
|
||||||
@@ -18,6 +19,10 @@ from copienator import (
|
|||||||
|
|
||||||
WORD_LIST_FILE = Path(__file__).resolve().parents[1] / "data" / "liste_francais.txt"
|
WORD_LIST_FILE = Path(__file__).resolve().parents[1] / "data" / "liste_francais.txt"
|
||||||
ACCENT_PATTERN = re.compile(r"[éèêëàâäîïôöùûüçœÉÈÊËÀÂÄÎÏÔÖÙÛÜÇŒ]")
|
ACCENT_PATTERN = re.compile(r"[éèêëàâäîïôöùûüçœÉÈÊËÀÂÄÎÏÔÖÙÛÜÇŒ]")
|
||||||
|
MATH_PATTERN = re.compile(
|
||||||
|
r"(\$\$.*?\$\$|\$.*?\$|\\\(.*?\\\)|\\\[.*?\\\])",
|
||||||
|
re.DOTALL,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
@@ -25,22 +30,37 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
|
|
||||||
|
|
||||||
def escape_latex_underscores(text: str) -> str:
|
def escape_latex_underscores(text: str) -> str:
|
||||||
r"""Escape underscores outside LaTeX math environments."""
|
r"""Escape underscores outside math without double-escaping existing ones."""
|
||||||
math_pattern = re.compile(
|
|
||||||
r"(\$\$.*?\$\$|\$.*?\$|\\\(.*?\\\)|\\\[.*?\\\])",
|
def escape_plain(value: str) -> str:
|
||||||
re.DOTALL,
|
# Collapse any existing escape run as well, making cleanup idempotent.
|
||||||
)
|
return re.sub(r"\\*_", lambda _match: r"\_", value)
|
||||||
|
|
||||||
parts: list[str] = []
|
parts: list[str] = []
|
||||||
last_end = 0
|
last_end = 0
|
||||||
for match in math_pattern.finditer(text):
|
for match in MATH_PATTERN.finditer(text):
|
||||||
start, end = match.span()
|
start, end = match.span()
|
||||||
parts.append(text[last_end:start].replace("_", r"\_"))
|
parts.append(escape_plain(text[last_end:start]))
|
||||||
parts.append(match.group(0))
|
parts.append(match.group(0))
|
||||||
last_end = end
|
last_end = end
|
||||||
parts.append(text[last_end:].replace("_", r"\_"))
|
parts.append(escape_plain(text[last_end:]))
|
||||||
return "".join(parts)
|
return "".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_overescaped_latex_commands(text: str) -> str:
|
||||||
|
r"""Collapse doubled command escapes inside LaTeX math environments.
|
||||||
|
|
||||||
|
Model responses occasionally contain ``\\mathbb`` after JSON decoding where
|
||||||
|
LaTeX requires ``\mathbb``. A doubled backslash followed by whitespace is a
|
||||||
|
legitimate row break (for example in ``cases``), so it must be preserved.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def normalize_math(match: re.Match[str]) -> str:
|
||||||
|
return re.sub(r"\\\\(?=[A-Za-z{}])", r"\\", match.group(0))
|
||||||
|
|
||||||
|
return MATH_PATTERN.sub(normalize_math, text)
|
||||||
|
|
||||||
|
|
||||||
def build_lookup_map(word_list_path: Path = WORD_LIST_FILE) -> dict[str, str]:
|
def build_lookup_map(word_list_path: Path = WORD_LIST_FILE) -> dict[str, str]:
|
||||||
words = word_list_path.read_text(encoding="utf-8").splitlines()
|
words = word_list_path.read_text(encoding="utf-8").splitlines()
|
||||||
lookup: dict[str, str] = {}
|
lookup: dict[str, str] = {}
|
||||||
@@ -67,8 +87,23 @@ def fix_hex_corruption_safe(text: str) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def some_other_replacements(text: str) -> str:
|
def repair_json_escape_corruption(text: str) -> str:
|
||||||
return text.replace("\neq", "\\neq").replace("\not", "\\not")
|
r"""Restore observed LaTeX commands consumed as JSON control escapes."""
|
||||||
|
replacements = (
|
||||||
|
("\x0crac", r"\frac"),
|
||||||
|
("\x0ceuille", r"\equiv"),
|
||||||
|
("\theta", r"\theta"),
|
||||||
|
("\times", r"\times"),
|
||||||
|
("\textbackslash ", "\\"),
|
||||||
|
("\negthinspace", r"\negthinspace"),
|
||||||
|
("\neq", r"\neq"),
|
||||||
|
("\not", r"\not"),
|
||||||
|
("∈", r"\ensuremath{\in}"),
|
||||||
|
("⊂", r"\ensuremath{\subset}"),
|
||||||
|
)
|
||||||
|
for broken, repaired in replacements:
|
||||||
|
text = text.replace(broken, repaired)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def clean_string(text: str, lookup: dict[str, str]) -> str:
|
def clean_string(text: str, lookup: dict[str, str]) -> str:
|
||||||
@@ -79,7 +114,9 @@ def clean_string(text: str, lookup: dict[str, str]) -> str:
|
|||||||
text = re.sub(r" \x00{1,2} ", " à ", text)
|
text = re.sub(r" \x00{1,2} ", " à ", text)
|
||||||
if "\x00" in text:
|
if "\x00" in text:
|
||||||
text = fast_fix(text, lookup).replace("\x00", "")
|
text = fast_fix(text, lookup).replace("\x00", "")
|
||||||
return escape_latex_underscores(some_other_replacements(text))
|
text = repair_json_escape_corruption(text)
|
||||||
|
text = normalize_overescaped_latex_commands(text)
|
||||||
|
return escape_latex_underscores(text)
|
||||||
|
|
||||||
|
|
||||||
def clean_obj(value: Any, lookup: dict[str, str]) -> Any:
|
def clean_obj(value: Any, lookup: dict[str, str]) -> Any:
|
||||||
@@ -104,6 +141,9 @@ def run(
|
|||||||
lookup = build_lookup_map(word_list_path)
|
lookup = build_lookup_map(word_list_path)
|
||||||
data = read_json(workspace.correction_file)
|
data = read_json(workspace.correction_file)
|
||||||
cleaned = clean_obj(data, lookup)
|
cleaned = clean_obj(data, lookup)
|
||||||
|
backup = workspace.root / "correction_precleanup.json"
|
||||||
|
atomic_write_bytes(backup, workspace.correction_file.read_bytes())
|
||||||
|
print(f"Original JSON backed up to {backup}")
|
||||||
atomic_write_json(workspace.correction_file, cleaned)
|
atomic_write_json(workspace.correction_file, cleaned)
|
||||||
print(f"Fixed JSON saved to {workspace.correction_file}")
|
print(f"Fixed JSON saved to {workspace.correction_file}")
|
||||||
return ExitCode.SUCCESS
|
return ExitCode.SUCCESS
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ from pdf2image import convert_from_path
|
|||||||
from PIL import Image, ImageChops, ImageDraw, ImageFilter
|
from PIL import Image, ImageChops, ImageDraw, ImageFilter
|
||||||
|
|
||||||
from copienator.commands import annotating
|
from copienator.commands import annotating
|
||||||
from copienator import utils
|
from copienator import configuration, utils
|
||||||
from copienator import (
|
from copienator import (
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
@@ -21,8 +21,10 @@ from copienator import (
|
|||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
from copienator.annotation_actions import apply_checkbox_actions, apply_score_overrides
|
from copienator.annotation_actions import apply_checkbox_actions, apply_score_overrides
|
||||||
|
from copienator.answer_info import build_answer_info
|
||||||
from copienator.annotation_data import AnnotationData, load_annotation_data
|
from copienator.annotation_data import AnnotationData, load_annotation_data
|
||||||
from copienator.filesystem import staged_files
|
from copienator.filesystem import staged_files
|
||||||
|
from copienator.return_answers import save_return_answer_options
|
||||||
|
|
||||||
Image.MAX_IMAGE_PIXELS = None
|
Image.MAX_IMAGE_PIXELS = None
|
||||||
|
|
||||||
@@ -68,10 +70,11 @@ def detect_checks_and_notes(
|
|||||||
print(f" Resizing annotated PDF from {user_image.size} to {reference.size}")
|
print(f" Resizing annotated PDF from {user_image.size} to {reference.size}")
|
||||||
user_image = user_image.resize(reference.size, Image.Resampling.LANCZOS)
|
user_image = user_image.resize(reference.size, Image.Resampling.LANCZOS)
|
||||||
|
|
||||||
difference = np.abs(
|
# Keep the full-size difference in uint8. Converting both tall group images
|
||||||
np.array(reference).astype(int) - np.array(user_image).astype(int)
|
# to the platform ``int`` dtype used several gigabytes per scan worker.
|
||||||
).astype(np.uint8)
|
difference = np.asarray(
|
||||||
difference_gray = np.mean(difference, axis=2)
|
ImageChops.difference(reference, user_image), dtype=np.uint8
|
||||||
|
)
|
||||||
keep_mask = Image.new("L", reference.size, 255)
|
keep_mask = Image.new("L", reference.size, 255)
|
||||||
mask_draw = ImageDraw.Draw(keep_mask)
|
mask_draw = ImageDraw.Draw(keep_mask)
|
||||||
actions: list[dict[str, Any]] = []
|
actions: list[dict[str, Any]] = []
|
||||||
@@ -82,10 +85,14 @@ def detect_checks_and_notes(
|
|||||||
x1, y1, x2, y2 = map(int, raw_box["global_box"])
|
x1, y1, x2, y2 = map(int, raw_box["global_box"])
|
||||||
x1, y1 = max(0, x1), max(0, y1)
|
x1, y1 = max(0, x1), max(0, y1)
|
||||||
x2, y2 = min(reference.width, x2), min(reference.height, y2)
|
x2, y2 = min(reference.width, x2), min(reference.height, y2)
|
||||||
region = difference_gray[y1 + 5 : y2 - 5, x1 + 5 : x2 - 5]
|
region = difference[y1 + 5 : y2 - 5, x1 + 5 : x2 - 5]
|
||||||
if region.size == 0:
|
if region.size == 0:
|
||||||
continue
|
continue
|
||||||
density = np.sum(region > 30) / region.size
|
# Preserve the previous mean-across-RGB threshold, but allocate its
|
||||||
|
# temporary float array only for the small checkbox region.
|
||||||
|
density = np.count_nonzero(np.mean(region, axis=2) > 30) / (
|
||||||
|
region.shape[0] * region.shape[1]
|
||||||
|
)
|
||||||
if density > 0.05:
|
if density > 0.05:
|
||||||
actions.append(raw_box)
|
actions.append(raw_box)
|
||||||
mask_draw.rectangle([x1 - 15, y1 - 15, x2 + 15, y2 + 15], fill=0)
|
mask_draw.rectangle([x1 - 15, y1 - 15, x2 + 15, y2 + 15], fill=0)
|
||||||
@@ -94,13 +101,17 @@ def detect_checks_and_notes(
|
|||||||
if raw_box.get("type") == "score" and raw_box.get("value") == 0.0:
|
if raw_box.get("type") == "score" and raw_box.get("value") == 0.0:
|
||||||
mask_draw.rectangle([0, y1 - 10, reference.width, y2 + 10], fill=0)
|
mask_draw.rectangle([0, y1 - 10, reference.width, y2 + 10], fill=0)
|
||||||
|
|
||||||
|
del difference
|
||||||
reference_blur = reference.filter(ImageFilter.GaussianBlur(2))
|
reference_blur = reference.filter(ImageFilter.GaussianBlur(2))
|
||||||
user_blur = user_image.filter(ImageFilter.GaussianBlur(2))
|
user_blur = user_image.filter(ImageFilter.GaussianBlur(2))
|
||||||
diff_image = ImageChops.difference(reference_blur, user_blur).convert("L")
|
diff_image = ImageChops.difference(reference_blur, user_blur).convert("L")
|
||||||
alpha = np.where(np.array(diff_image) > 50, 255, 0).astype(np.uint8)
|
del reference_blur, user_blur
|
||||||
final_alpha = np.minimum(alpha, np.array(keep_mask))
|
alpha = np.asarray(diff_image, dtype=np.uint8).copy()
|
||||||
|
np.greater(alpha, 50, out=alpha)
|
||||||
|
alpha *= np.uint8(255)
|
||||||
|
np.minimum(alpha, np.asarray(keep_mask, dtype=np.uint8), out=alpha)
|
||||||
notes = user_image.convert("RGBA")
|
notes = user_image.convert("RGBA")
|
||||||
notes.putalpha(Image.fromarray(final_alpha))
|
notes.putalpha(Image.fromarray(alpha))
|
||||||
return actions, notes
|
return actions, notes
|
||||||
|
|
||||||
|
|
||||||
@@ -148,11 +159,14 @@ def apply_actions_and_regenerate(
|
|||||||
raise TypeError(f"Expected a JSON object in {bnote_path}")
|
raise TypeError(f"Expected a JSON object in {bnote_path}")
|
||||||
|
|
||||||
labels_data = data[student_id]
|
labels_data = data[student_id]
|
||||||
dirty_labels = apply_checkbox_actions(labels_data, actions, print)
|
apply_checkbox_actions(labels_data, actions, print)
|
||||||
|
score_path = output_dir / "score.json"
|
||||||
|
preserve_score_file = update_score and score_path.is_file()
|
||||||
if update_score:
|
if update_score:
|
||||||
dirty_labels |= apply_score_overrides(labels_data, output_dir / "score.json", print)
|
apply_score_overrides(labels_data, score_path, print)
|
||||||
|
|
||||||
scores = dict.fromkeys(all_labels, "")
|
scores = dict.fromkeys(all_labels, "")
|
||||||
|
answer_labels: list[str] = []
|
||||||
dirty_images: dict[str, Image.Image] = {}
|
dirty_images: dict[str, Image.Image] = {}
|
||||||
concatenated: list[Image.Image] = []
|
concatenated: list[Image.Image] = []
|
||||||
filtered: list[Image.Image] = []
|
filtered: list[Image.Image] = []
|
||||||
@@ -169,6 +183,8 @@ def apply_actions_and_regenerate(
|
|||||||
content = labels_data[label]
|
content = labels_data[label]
|
||||||
result = content["result"]
|
result = content["result"]
|
||||||
scores[label] = str(result.get("score", 0))
|
scores[label] = str(result.get("score", 0))
|
||||||
|
if result.get("error") == "empty-answer":
|
||||||
|
continue
|
||||||
|
|
||||||
sub_note = None
|
sub_note = None
|
||||||
if notes_layer is not None:
|
if notes_layer is not None:
|
||||||
@@ -204,30 +220,63 @@ def apply_actions_and_regenerate(
|
|||||||
body = sub_note.crop((0, old_header_height, width, height))
|
body = sub_note.crop((0, old_header_height, width, height))
|
||||||
final_image.paste(body, (0, new_header_height), mask=body)
|
final_image.paste(body, (0, new_header_height), mask=body)
|
||||||
|
|
||||||
if label in dirty_labels or has_notes:
|
dirty_images[label] = final_image
|
||||||
dirty_images[label] = final_image
|
answer_labels.append(label)
|
||||||
concatenated.append(final_image)
|
concatenated.append(final_image)
|
||||||
if float(scores[label]) != 4.0 or result.get("feedback", []):
|
if float(scores[label]) != 4.0 or result.get("feedback", []):
|
||||||
filtered.append(final_image)
|
filtered.append(final_image)
|
||||||
|
|
||||||
concat_image = concatenate(concatenated)
|
concat_image = concatenate(concatenated)
|
||||||
filtered_image = concatenate(filtered)
|
filtered_image = concatenate(filtered)
|
||||||
with staged_files(output_dir) as staging:
|
with staged_files(output_dir, remove=("Concat.jpg", "Concat_F.jpg", "Concat_F.pdf",
|
||||||
|
"touched.json", "answer_labels.json")) as staging:
|
||||||
for label, image in dirty_images.items():
|
for label, image in dirty_images.items():
|
||||||
image.save(staging / f"{label}.jpg")
|
image.save(staging / f"{label}.jpg")
|
||||||
atomic_write_json(staging / "score.json", scores)
|
if not preserve_score_file:
|
||||||
|
atomic_write_json(staging / "score.json", scores)
|
||||||
|
atomic_write_json(staging / "info.json", build_answer_info(
|
||||||
|
scores, labels_data, answer_labels
|
||||||
|
))
|
||||||
if concat_image is not None:
|
if concat_image is not None:
|
||||||
concat_image.save(staging / "Concat.jpg")
|
concat_image.save(staging / "Concat.jpg")
|
||||||
if filtered_image is not None:
|
if filtered_image is not None:
|
||||||
filtered_image.save(staging / "Concat_F.jpg")
|
filtered_image.save(staging / "Concat_F.jpg")
|
||||||
|
|
||||||
|
if preserve_score_file:
|
||||||
|
print(f" Preserved existing score.json in {output_dir}")
|
||||||
print(f" Saved regenerated files in {output_dir}")
|
print(f" Saved regenerated files in {output_dir}")
|
||||||
return ExitCode.PARTIAL if incomplete else ExitCode.SUCCESS
|
return ExitCode.PARTIAL if incomplete else ExitCode.SUCCESS
|
||||||
|
|
||||||
|
|
||||||
def run(workspace: EvaluationWorkspace, *, update_score: bool = False) -> ExitCode:
|
def run(
|
||||||
|
workspace: EvaluationWorkspace,
|
||||||
|
*,
|
||||||
|
update_score: bool = False,
|
||||||
|
return_answers_context: bool | None = None,
|
||||||
|
return_answers_question: bool | None = None,
|
||||||
|
return_answers_solution: bool | None = None,
|
||||||
|
) -> ExitCode:
|
||||||
workspace.require_files("labels", "correction.json")
|
workspace.require_files("labels", "correction.json")
|
||||||
workspace.require_directories("Copies", "Par label", "Bnot")
|
workspace.require_directories("Copies", "Par label", "Bnot")
|
||||||
|
if configuration.RETURN_ANSWERS_ENABLED:
|
||||||
|
save_return_answer_options(
|
||||||
|
workspace.root,
|
||||||
|
context=(
|
||||||
|
configuration.RETURN_ANSWERS_CONTEXT
|
||||||
|
if return_answers_context is None
|
||||||
|
else return_answers_context
|
||||||
|
),
|
||||||
|
question=(
|
||||||
|
configuration.RETURN_ANSWERS_QUESTION
|
||||||
|
if return_answers_question is None
|
||||||
|
else return_answers_question
|
||||||
|
),
|
||||||
|
solution=(
|
||||||
|
configuration.RETURN_ANSWERS_SOLUTION
|
||||||
|
if return_answers_solution is None
|
||||||
|
else return_answers_solution
|
||||||
|
),
|
||||||
|
)
|
||||||
all_labels = utils.read_all_labels(workspace.root)
|
all_labels = utils.read_all_labels(workspace.root)
|
||||||
loaded = load_annotation_data(workspace)
|
loaded = load_annotation_data(workspace)
|
||||||
for warning in loaded.warnings:
|
for warning in loaded.warnings:
|
||||||
@@ -268,7 +317,28 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--update-score",
|
"--update-score",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
help="Override generated scores with values from existing score.json files",
|
help=(
|
||||||
|
"Regenerate images with current statement/solution PDFs while "
|
||||||
|
"preserving and applying existing score.json values"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-context",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_CONTEXT,
|
||||||
|
help="Include applicable context pages in individual answer exports",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-question",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_QUESTION,
|
||||||
|
help="Include the current question PDF in individual answer exports",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-solution",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_SOLUTION,
|
||||||
|
help="Include the current solution PDF in individual answer exports",
|
||||||
)
|
)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
@@ -277,11 +347,16 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
|
|
||||||
def handle(args: argparse.Namespace) -> ExitCode:
|
def handle(args: argparse.Namespace) -> ExitCode:
|
||||||
return run(workspace_from_args(args), update_score=args.update_score)
|
return run(
|
||||||
|
workspace_from_args(args),
|
||||||
|
update_score=args.update_score,
|
||||||
|
return_answers_context=args.return_answers_context,
|
||||||
|
return_answers_question=args.return_answers_question,
|
||||||
|
return_answers_solution=args.return_answers_solution,
|
||||||
|
)
|
||||||
|
|
||||||
return execute(parser, argv, handle)
|
return execute(parser, argv, handle)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
raise SystemExit(main())
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ from typing import Any
|
|||||||
|
|
||||||
from PIL import Image, ImageDraw
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
from copienator import configuration
|
||||||
from copienator import (
|
from copienator import (
|
||||||
EvaluationWorkspace,
|
EvaluationWorkspace,
|
||||||
ExitCode,
|
ExitCode,
|
||||||
@@ -20,6 +21,7 @@ from copienator import (
|
|||||||
workspace_from_args,
|
workspace_from_args,
|
||||||
)
|
)
|
||||||
from copienator.annotation_actions import apply_checkbox_actions, apply_score_overrides
|
from copienator.annotation_actions import apply_checkbox_actions, apply_score_overrides
|
||||||
|
from copienator.answer_info import build_answer_info
|
||||||
from copienator.annotation_data import AnnotationData, RefaireList, load_annotation_data
|
from copienator.annotation_data import AnnotationData, RefaireList, load_annotation_data
|
||||||
from copienator.commands import annotating
|
from copienator.commands import annotating
|
||||||
from copienator.commands.reading_annotations import (
|
from copienator.commands.reading_annotations import (
|
||||||
@@ -28,9 +30,11 @@ from copienator.commands.reading_annotations import (
|
|||||||
has_significant_notes,
|
has_significant_notes,
|
||||||
)
|
)
|
||||||
from copienator.filesystem import staged_files
|
from copienator.filesystem import staged_files
|
||||||
|
from copienator.return_answers import save_return_answer_options
|
||||||
|
|
||||||
LabelNotes = dict[str, dict[str, Any]]
|
LabelNotes = dict[str, dict[str, Any]]
|
||||||
ScanResult = tuple[dict[str, list[dict[str, Any]]], dict[str, LabelNotes]]
|
ScanResult = tuple[dict[str, list[dict[str, Any]]], dict[str, LabelNotes]]
|
||||||
|
SCAN_WORKERS = 2
|
||||||
|
|
||||||
|
|
||||||
def get_extra_pdfs_as_images(
|
def get_extra_pdfs_as_images(
|
||||||
@@ -189,14 +193,15 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
logs = [f"\nProcessing compilation for: Copie{student_id}"]
|
logs = [f"\nProcessing compilation for: Copie{student_id}"]
|
||||||
output_dir = workspace.root / annotation_dir / f"Copie{student_id}"
|
output_dir = workspace.root / annotation_dir / f"Copie{student_id}"
|
||||||
labels_data = data.get(student_id, {})
|
labels_data = data.get(student_id, {})
|
||||||
dirty_labels = apply_checkbox_actions(labels_data, actions, logs.append)
|
apply_checkbox_actions(labels_data, actions, logs.append)
|
||||||
|
score_path = output_dir / "score.json"
|
||||||
|
preserve_score_file = update_score and score_path.is_file()
|
||||||
if update_score:
|
if update_score:
|
||||||
dirty_labels |= apply_score_overrides(
|
apply_score_overrides(
|
||||||
labels_data, output_dir / "score.json", logs.append
|
labels_data, score_path, logs.append
|
||||||
)
|
)
|
||||||
|
|
||||||
selected_labels = selected_labels if selected_labels is not None else set()
|
selected_labels = selected_labels if selected_labels is not None else set()
|
||||||
dirty_labels |= selected_labels
|
|
||||||
simple_layout = None
|
simple_layout = None
|
||||||
simple_annotated = None
|
simple_annotated = None
|
||||||
if selected_labels and annotation_dir == "Anot":
|
if selected_labels and annotation_dir == "Anot":
|
||||||
@@ -239,6 +244,8 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
else {}
|
else {}
|
||||||
)
|
)
|
||||||
scores = dict.fromkeys(all_labels, "")
|
scores = dict.fromkeys(all_labels, "")
|
||||||
|
touched = dict.fromkeys(all_labels, False)
|
||||||
|
answer_labels: list[str] = []
|
||||||
dirty_images: dict[str, Image.Image] = {}
|
dirty_images: dict[str, Image.Image] = {}
|
||||||
concat_images: list[Image.Image] = []
|
concat_images: list[Image.Image] = []
|
||||||
filtered_groups: list[list[Image.Image]] = []
|
filtered_groups: list[list[Image.Image]] = []
|
||||||
@@ -255,6 +262,9 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
):
|
):
|
||||||
result["score"] = old_scores[label]
|
result["score"] = old_scores[label]
|
||||||
scores[label] = str(result.get("score", 0))
|
scores[label] = str(result.get("score", 0))
|
||||||
|
touched[label] = False
|
||||||
|
if result.get("error") == "empty-answer":
|
||||||
|
continue
|
||||||
saved_image = output_dir / f"{label}.jpg"
|
saved_image = output_dir / f"{label}.jpg"
|
||||||
if selected_labels and label not in selected_labels and saved_image.is_file():
|
if selected_labels and label not in selected_labels and saved_image.is_file():
|
||||||
with Image.open(saved_image) as saved:
|
with Image.open(saved_image) as saved:
|
||||||
@@ -271,12 +281,14 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
dirty_images[label] = final_image
|
dirty_images[label] = final_image
|
||||||
scores[label] = str(old_scores.get(label, scores[label]))
|
scores[label] = str(old_scores.get(label, scores[label]))
|
||||||
concat_images.append(final_image)
|
concat_images.append(final_image)
|
||||||
|
answer_labels.append(label)
|
||||||
# Keep previously reviewed content, including handwriting.
|
# Keep previously reviewed content, including handwriting.
|
||||||
if annotation_dir == "BGnot":
|
if annotation_dir == "BGnot":
|
||||||
extras = get_extra_pdfs_as_images(
|
extras = get_extra_pdfs_as_images(
|
||||||
workspace.root, label, annotating, all_labels
|
workspace.root, label, annotating, all_labels
|
||||||
)
|
)
|
||||||
filtered_groups.append([*extras, final_image])
|
filtered_groups.append([*extras, final_image])
|
||||||
|
touched[label] = True
|
||||||
else:
|
else:
|
||||||
filtered_groups.append([final_image])
|
filtered_groups.append([final_image])
|
||||||
continue
|
continue
|
||||||
@@ -313,9 +325,9 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
body = sub_note.crop((0, old_header_height, width, height))
|
body = sub_note.crop((0, old_header_height, width, height))
|
||||||
final_image.paste(body, (0, new_header_height), mask=body)
|
final_image.paste(body, (0, new_header_height), mask=body)
|
||||||
|
|
||||||
if label in dirty_labels or has_notes or selected_labels:
|
# Persist every final block, including unchanged answers, for returns.
|
||||||
dirty_images[label] = final_image
|
dirty_images[label] = final_image
|
||||||
logs.append(f" Saved dirty image: {label}.jpg")
|
answer_labels.append(label)
|
||||||
concat_images.append(final_image)
|
concat_images.append(final_image)
|
||||||
|
|
||||||
feedbacks = result.get("feedback", [])
|
feedbacks = result.get("feedback", [])
|
||||||
@@ -329,11 +341,13 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
else []
|
else []
|
||||||
)
|
)
|
||||||
filtered_groups.append([*extras, final_image])
|
filtered_groups.append([*extras, final_image])
|
||||||
|
touched[label] = annotation_dir == "BGnot"
|
||||||
|
|
||||||
concat_image = concatenate(concat_images)
|
concat_image = concatenate(concat_images)
|
||||||
if incomplete:
|
if incomplete:
|
||||||
return ExitCode.PARTIAL, "\n".join(logs)
|
return ExitCode.PARTIAL, "\n".join(logs)
|
||||||
with staged_files(output_dir, remove=("Concat_F.pdf", "Concat_F.jpg")) as staging:
|
with staged_files(output_dir, remove=("Concat.jpg", "Concat_F.pdf", "Concat_F.jpg",
|
||||||
|
"touched.json", "answer_labels.json")) as staging:
|
||||||
if simple_layout is not None:
|
if simple_layout is not None:
|
||||||
simple_layout["replaced"] = sorted(
|
simple_layout["replaced"] = sorted(
|
||||||
set(simple_layout["replaced"]) | selected_labels
|
set(simple_layout["replaced"]) | selected_labels
|
||||||
@@ -341,7 +355,11 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
atomic_write_json(staging / "refaire_simple_layout.json", simple_layout)
|
atomic_write_json(staging / "refaire_simple_layout.json", simple_layout)
|
||||||
for label, image in dirty_images.items():
|
for label, image in dirty_images.items():
|
||||||
image.save(staging / f"{label}.jpg")
|
image.save(staging / f"{label}.jpg")
|
||||||
atomic_write_json(staging / "score.json", scores)
|
if not preserve_score_file:
|
||||||
|
atomic_write_json(staging / "score.json", scores)
|
||||||
|
atomic_write_json(staging / "info.json", build_answer_info(
|
||||||
|
scores, labels_data, answer_labels, touched
|
||||||
|
))
|
||||||
if concat_image is not None:
|
if concat_image is not None:
|
||||||
concat_image.save(staging / "Concat.jpg")
|
concat_image.save(staging / "Concat.jpg")
|
||||||
if filtered_groups:
|
if filtered_groups:
|
||||||
@@ -352,6 +370,8 @@ def apply_actions_and_regenerate_grouped(
|
|||||||
[image for group in filtered_groups for image in group]
|
[image for group in filtered_groups for image in group]
|
||||||
)
|
)
|
||||||
filtered_image.save(staging / "Concat_F.jpg")
|
filtered_image.save(staging / "Concat_F.jpg")
|
||||||
|
if preserve_score_file:
|
||||||
|
logs.append(f" Preserved existing score.json in {output_dir}")
|
||||||
logs.append(f" Saved regenerated files in {output_dir}")
|
logs.append(f" Saved regenerated files in {output_dir}")
|
||||||
status = ExitCode.PARTIAL if incomplete else ExitCode.SUCCESS
|
status = ExitCode.PARTIAL if incomplete else ExitCode.SUCCESS
|
||||||
return status, "\n".join(logs)
|
return status, "\n".join(logs)
|
||||||
@@ -449,6 +469,9 @@ def run(
|
|||||||
refaire: bool = False,
|
refaire: bool = False,
|
||||||
update_score: bool = False,
|
update_score: bool = False,
|
||||||
annotation_dir: str = "BGnot",
|
annotation_dir: str = "BGnot",
|
||||||
|
return_answers_context: bool | None = None,
|
||||||
|
return_answers_question: bool | None = None,
|
||||||
|
return_answers_solution: bool | None = None,
|
||||||
) -> ExitCode:
|
) -> ExitCode:
|
||||||
workspace.require_files("labels", "correction.json")
|
workspace.require_files("labels", "correction.json")
|
||||||
workspace.require_directories("Copies", "Par label", annotation_dir)
|
workspace.require_directories("Copies", "Par label", annotation_dir)
|
||||||
@@ -458,6 +481,25 @@ def run(
|
|||||||
workspace.require_files("refaire.json")
|
workspace.require_files("refaire.json")
|
||||||
workspace.require_directories("BRnot")
|
workspace.require_directories("BRnot")
|
||||||
refaire_list, refaire_by_student = _read_refaire(workspace)
|
refaire_list, refaire_by_student = _read_refaire(workspace)
|
||||||
|
if configuration.RETURN_ANSWERS_ENABLED:
|
||||||
|
save_return_answer_options(
|
||||||
|
workspace.root,
|
||||||
|
context=(
|
||||||
|
configuration.RETURN_ANSWERS_CONTEXT
|
||||||
|
if return_answers_context is None
|
||||||
|
else return_answers_context
|
||||||
|
),
|
||||||
|
question=(
|
||||||
|
configuration.RETURN_ANSWERS_QUESTION
|
||||||
|
if return_answers_question is None
|
||||||
|
else return_answers_question
|
||||||
|
),
|
||||||
|
solution=(
|
||||||
|
configuration.RETURN_ANSWERS_SOLUTION
|
||||||
|
if return_answers_solution is None
|
||||||
|
else return_answers_solution
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
all_labels = utils.read_all_labels(workspace.root)
|
all_labels = utils.read_all_labels(workspace.root)
|
||||||
loaded = load_annotation_data(workspace)
|
loaded = load_annotation_data(workspace)
|
||||||
@@ -482,7 +524,10 @@ def run(
|
|||||||
and path.is_dir()
|
and path.is_dir()
|
||||||
and not path.name.startswith("Copie")
|
and not path.name.startswith("Copie")
|
||||||
]
|
]
|
||||||
with concurrent.futures.ThreadPoolExecutor(max_workers=6) as executor:
|
# Each worker decodes a full-height returned group and its reference image.
|
||||||
|
# Keep this stage deliberately narrow; answer regeneration below has its own
|
||||||
|
# parallel executor and a much smaller per-task memory footprint.
|
||||||
|
with concurrent.futures.ThreadPoolExecutor(max_workers=SCAN_WORKERS) as executor:
|
||||||
futures = [
|
futures = [
|
||||||
executor.submit(_scan_annotation_directory, path, only_ids)
|
executor.submit(_scan_annotation_directory, path, only_ids)
|
||||||
for path in group_dirs
|
for path in group_dirs
|
||||||
@@ -587,7 +632,28 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--update-score",
|
"--update-score",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
help="Override generated scores with values from existing score.json files",
|
help=(
|
||||||
|
"Regenerate images with current statement/solution PDFs while "
|
||||||
|
"preserving and applying existing score.json values"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-context",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_CONTEXT,
|
||||||
|
help="Include applicable context pages in individual answer exports",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-question",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_QUESTION,
|
||||||
|
help="Include the current question PDF in individual answer exports",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--return-answers-solution",
|
||||||
|
action=argparse.BooleanOptionalAction,
|
||||||
|
default=configuration.RETURN_ANSWERS_SOLUTION,
|
||||||
|
help="Include the current solution PDF in individual answer exports",
|
||||||
)
|
)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
@@ -603,6 +669,9 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
refaire=args.refaire,
|
refaire=args.refaire,
|
||||||
update_score=args.update_score,
|
update_score=args.update_score,
|
||||||
annotation_dir=args.annotation_dir,
|
annotation_dir=args.annotation_dir,
|
||||||
|
return_answers_context=args.return_answers_context,
|
||||||
|
return_answers_question=args.return_answers_question,
|
||||||
|
return_answers_solution=args.return_answers_solution,
|
||||||
)
|
)
|
||||||
|
|
||||||
return execute(parser, argv, handle)
|
return execute(parser, argv, handle)
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ from pathlib import Path
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from pypdf import PdfWriter
|
from pypdf import PdfWriter
|
||||||
|
from copienator.pdf_cut import split_pdf
|
||||||
|
|
||||||
from copienator import (
|
from copienator import (
|
||||||
CliError,
|
CliError,
|
||||||
@@ -22,7 +23,8 @@ from copienator import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
OPERATORS = ("-x", "->", "x>", "ss", "sx", "xx", "xs")
|
OPERATORS = ("-x", "->", "x>", "ss", "sx", "xx", "xs")
|
||||||
OPERATOR_PATTERN = re.compile(r"\s+(-x|->|x>|ss|sx|xx|xs)\s+")
|
CUT_PATTERN = re.compile(r"c\{(\d+(?:\.\d+)?)\}([12])([x>])")
|
||||||
|
OPERATOR_PATTERN = re.compile(r"\s+(-x|->|x>|ss|sx|xx|xs|c\{\d+(?:\.\d+)?\}[12][x>])\s+")
|
||||||
COPY_PATTERN = re.compile(r"Copie(\d+)\s+(.+)")
|
COPY_PATTERN = re.compile(r"Copie(\d+)\s+(.+)")
|
||||||
|
|
||||||
|
|
||||||
@@ -34,6 +36,11 @@ class ManualInstruction:
|
|||||||
new_label: str
|
new_label: str
|
||||||
pipe_first: bool
|
pipe_first: bool
|
||||||
|
|
||||||
|
@property
|
||||||
|
def cut(self) -> tuple[float, int] | None:
|
||||||
|
match = CUT_PATTERN.fullmatch(self.operator)
|
||||||
|
return (float(match[1]), int(match[2])) if match else None
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def should_merge(self) -> bool:
|
def should_merge(self) -> bool:
|
||||||
return self.operator.endswith(">")
|
return self.operator.endswith(">")
|
||||||
@@ -48,10 +55,14 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
|
|
||||||
|
|
||||||
def parse_instructions(path: Path) -> list[ManualInstruction]:
|
def parse_instructions(path: Path) -> list[ManualInstruction]:
|
||||||
|
return parse_instruction_text(path.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def parse_instruction_text(text: str) -> list[ManualInstruction]:
|
||||||
instructions: list[ManualInstruction] = []
|
instructions: list[ManualInstruction] = []
|
||||||
malformed: list[int] = []
|
malformed: list[int] = []
|
||||||
for line_number, raw_line in enumerate(
|
for line_number, raw_line in enumerate(
|
||||||
path.read_text(encoding="utf-8").splitlines(), start=1
|
text.splitlines(), start=1
|
||||||
):
|
):
|
||||||
line = raw_line.strip()
|
line = raw_line.strip()
|
||||||
if not line or line.startswith("###"):
|
if not line or line.startswith("###"):
|
||||||
@@ -64,7 +75,10 @@ def parse_instructions(path: Path) -> list[ManualInstruction]:
|
|||||||
right = line[operator_match.end() :].strip()
|
right = line[operator_match.end() :].strip()
|
||||||
copy_match = COPY_PATTERN.fullmatch(left)
|
copy_match = COPY_PATTERN.fullmatch(left)
|
||||||
new_label = right.strip("|").strip()
|
new_label = right.strip("|").strip()
|
||||||
if copy_match is None or not new_label:
|
cut_match = CUT_PATTERN.fullmatch(operator_match.group(1))
|
||||||
|
if (copy_match is None or not new_label
|
||||||
|
or (cut_match and (not 0 < float(cut_match[1]) < 100
|
||||||
|
or copy_match.group(2).strip() == new_label))):
|
||||||
malformed.append(line_number)
|
malformed.append(line_number)
|
||||||
continue
|
continue
|
||||||
instructions.append(
|
instructions.append(
|
||||||
@@ -97,15 +111,25 @@ def set_suffix_and_clean_error(
|
|||||||
item["result"]["suffix"] = suffix
|
item["result"]["suffix"] = suffix
|
||||||
error = item["result"].get("error", "")
|
error = item["result"].get("error", "")
|
||||||
if new_label_target:
|
if new_label_target:
|
||||||
if f"wrg-lbl:{new_label_target}?delayed" in error:
|
# This instruction acknowledges this source/target conflict,
|
||||||
item["result"]["error"] = (
|
# including decisions to keep/discard PDFs without merging.
|
||||||
f"wrg-lbl-moved-to:{new_label_target}"
|
# Leave other pending targets (and other copies) untouched.
|
||||||
)
|
result = item["result"]
|
||||||
if f"(delayed){new_label_target}" in error:
|
if "delayed" in result:
|
||||||
item["result"]["error"] = error.replace(
|
pending = [entry for entry in result["delayed"]
|
||||||
f"(delayed){new_label_target}",
|
if entry not in (["wrong-label", new_label_target],
|
||||||
f"(->){new_label_target}",
|
["add-label", new_label_target])]
|
||||||
)
|
if pending:
|
||||||
|
result["delayed"] = pending
|
||||||
|
else:
|
||||||
|
result.pop("delayed")
|
||||||
|
if error in {f"wrg-lbl:{new_label_target}?",
|
||||||
|
f"wrg-lbl:{new_label_target}?delayed",
|
||||||
|
f"wrg-lbl:{new_label_target}?exists"}:
|
||||||
|
error = f"wrg-lbl-moved-to:{new_label_target}"
|
||||||
|
error = error.replace(f"(delayed){new_label_target}", f"(->){new_label_target}")
|
||||||
|
error = error.replace(f"(->){new_label_target}?", f"(->){new_label_target}")
|
||||||
|
result["error"] = error
|
||||||
|
|
||||||
|
|
||||||
def get_actual_pdf(copies_dir: Path, copy_id: str, label: str) -> Path:
|
def get_actual_pdf(copies_dir: Path, copy_id: str, label: str) -> Path:
|
||||||
@@ -154,6 +178,9 @@ def resolve_manual(workspace: EvaluationWorkspace) -> ExitCode:
|
|||||||
raise CliError("correction.json must contain a JSON object")
|
raise CliError("correction.json must contain a JSON object")
|
||||||
results: dict[str, Any] = loaded
|
results: dict[str, Any] = loaded
|
||||||
instructions = parse_instructions(workspace.manual_resolutions_file)
|
instructions = parse_instructions(workspace.manual_resolutions_file)
|
||||||
|
cut_sources = [(item.copy_id, item.old_label) for item in instructions if item.cut]
|
||||||
|
if len(set(cut_sources)) != len(cut_sources):
|
||||||
|
raise CliError("Une seule coupe par label source est autorisée dans une résolution.")
|
||||||
|
|
||||||
initial_paths: dict[tuple[str, str], Path] = {}
|
initial_paths: dict[tuple[str, str], Path] = {}
|
||||||
current_paths: dict[tuple[str, str], Path] = {}
|
current_paths: dict[tuple[str, str], Path] = {}
|
||||||
@@ -174,6 +201,15 @@ def resolve_manual(workspace: EvaluationWorkspace) -> ExitCode:
|
|||||||
key_new = (instruction.copy_id, instruction.new_label)
|
key_new = (instruction.copy_id, instruction.new_label)
|
||||||
source = initial_paths[key_old]
|
source = initial_paths[key_old]
|
||||||
destination = current_paths[key_new]
|
destination = current_paths[key_new]
|
||||||
|
if instruction.cut:
|
||||||
|
percent, keep = instruction.cut
|
||||||
|
first = source.parent / f"temp_{len(temp_files)}.pdf"
|
||||||
|
second = source.parent / f"temp_{len(temp_files) + 1}.pdf"
|
||||||
|
temp_files.extend((first, second))
|
||||||
|
split_pdf(source, percent, first, second)
|
||||||
|
retained, source = (first, second) if keep == 1 else (second, first)
|
||||||
|
current_paths[key_old] = retained
|
||||||
|
files_to_old.add(initial_paths[key_old])
|
||||||
temp_output = (
|
temp_output = (
|
||||||
workspace.copies_dir
|
workspace.copies_dir
|
||||||
/ f"Copie{instruction.copy_id}"
|
/ f"Copie{instruction.copy_id}"
|
||||||
|
|||||||
@@ -1,13 +1,14 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import math
|
||||||
import shutil
|
import shutil
|
||||||
import tempfile
|
import tempfile
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import fitz
|
import pymupdf
|
||||||
from pypdf import PdfReader, PdfWriter
|
from pypdf import PdfReader, PdfWriter
|
||||||
|
|
||||||
from copienator import utils
|
from copienator import utils
|
||||||
@@ -23,6 +24,7 @@ from copienator import (
|
|||||||
from copienator.filesystem import staged_directory
|
from copienator.filesystem import staged_directory
|
||||||
|
|
||||||
SQUARE = 1000 // 38
|
SQUARE = 1000 // 38
|
||||||
|
ANSWER_TOP_PADDING_POINTS = 4 * 72 / 25.4
|
||||||
Coordinate = tuple[str, int, int, int, int, int]
|
Coordinate = tuple[str, int, int, int, int, int]
|
||||||
ParsedCoordinate = tuple[str, str, int, int, int, int, int]
|
ParsedCoordinate = tuple[str, str, int, int, int, int, int]
|
||||||
|
|
||||||
@@ -72,7 +74,7 @@ def _parse_coordinates(coords_list: list[Coordinate]) -> list[ParsedCoordinate]:
|
|||||||
|
|
||||||
|
|
||||||
def _save_cropped_page(
|
def _save_cropped_page(
|
||||||
document: fitz.Document,
|
document: pymupdf.Document,
|
||||||
page_number: int,
|
page_number: int,
|
||||||
x0: float,
|
x0: float,
|
||||||
y0: float,
|
y0: float,
|
||||||
@@ -80,38 +82,51 @@ def _save_cropped_page(
|
|||||||
y1: float,
|
y1: float,
|
||||||
output_path: Path,
|
output_path: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
page = document[page_number]
|
# The source has been normalized by _prepare_split_pages: no rotation or
|
||||||
rotated_rectangle = page.rect * page.transformation_matrix
|
# CropBox translation remains in the coordinates passed to show_pdf_page.
|
||||||
visual_crop = fitz.Rect(
|
visual_crop = pymupdf.Rect(x0, y0, x1, y1)
|
||||||
rotated_rectangle.x0 + x0,
|
cropped = pymupdf.open()
|
||||||
y0,
|
|
||||||
rotated_rectangle.x0 + x1,
|
|
||||||
y1,
|
|
||||||
)
|
|
||||||
unrotated_clip = visual_crop * page.derotation_matrix
|
|
||||||
cropped = fitz.open()
|
|
||||||
try:
|
try:
|
||||||
target_page = cropped.new_page(width=visual_crop.width, height=visual_crop.height)
|
target_page = cropped.new_page(width=visual_crop.width, height=visual_crop.height)
|
||||||
target_page.show_pdf_page(
|
target_page.show_pdf_page(
|
||||||
target_page.rect,
|
target_page.rect,
|
||||||
document,
|
document,
|
||||||
page_number,
|
page_number,
|
||||||
rotate=-page.rotation,
|
clip=visual_crop,
|
||||||
clip=unrotated_clip,
|
|
||||||
)
|
)
|
||||||
cropped.save(output_path)
|
cropped.save(output_path)
|
||||||
finally:
|
finally:
|
||||||
cropped.close()
|
cropped.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _prepare_split_pages(document: pymupdf.Document) -> list[pymupdf.Rect]:
|
||||||
|
"""Use the label preview's full-page coordinates; retain visible bounds.
|
||||||
|
|
||||||
|
Work only on the in-memory document. Baking rotation into the content after
|
||||||
|
restoring the MediaBox avoids show_pdf_page's rotated CropBox offsets.
|
||||||
|
Intersecting with the saved bounds later preserves prior margin cropping.
|
||||||
|
"""
|
||||||
|
visible_bounds = []
|
||||||
|
for page in document:
|
||||||
|
crop = page.cropbox
|
||||||
|
media = page.mediabox
|
||||||
|
full_crop = pymupdf.Rect(media.x0, 0, media.x1, media.height)
|
||||||
|
page.set_cropbox(full_crop)
|
||||||
|
crop -= (full_crop.x0, full_crop.y0, full_crop.x0, full_crop.y0)
|
||||||
|
visible_bounds.append(crop * page.rotation_matrix)
|
||||||
|
page.remove_rotation()
|
||||||
|
return visible_bounds
|
||||||
|
|
||||||
|
|
||||||
def _render_split_outputs(
|
def _render_split_outputs(
|
||||||
input_pdf: Path,
|
input_pdf: Path,
|
||||||
coords_list: list[Coordinate],
|
coords_list: list[Coordinate],
|
||||||
staging: Path,
|
staging: Path,
|
||||||
) -> set[str]:
|
) -> set[str]:
|
||||||
"""Render every current answer into an otherwise empty staging directory."""
|
"""Render every current answer into an otherwise empty staging directory."""
|
||||||
document = fitz.open(input_pdf)
|
document = pymupdf.open(input_pdf)
|
||||||
try:
|
try:
|
||||||
|
visible_bounds = _prepare_split_pages(document)
|
||||||
parsed = _parse_coordinates(coords_list)
|
parsed = _parse_coordinates(coords_list)
|
||||||
parts_by_label: defaultdict[str, list[Path]] = defaultdict(list)
|
parts_by_label: defaultdict[str, list[Path]] = defaultdict(list)
|
||||||
with tempfile.TemporaryDirectory(prefix="copienator-split-") as temp_directory:
|
with tempfile.TemporaryDirectory(prefix="copienator-split-") as temp_directory:
|
||||||
@@ -158,18 +173,30 @@ def _render_split_outputs(
|
|||||||
|
|
||||||
for page_number in range(start_page, end_page + 1):
|
for page_number in range(start_page, end_page + 1):
|
||||||
page = document[page_number]
|
page = document[page_number]
|
||||||
y0 = (y_start / 1000) * page.rect.height if page_number == start_page else 0
|
y0 = (
|
||||||
|
math.floor(max(
|
||||||
|
0,
|
||||||
|
(y_start / 1000) * page.rect.height
|
||||||
|
- ANSWER_TOP_PADDING_POINTS,
|
||||||
|
))
|
||||||
|
if page_number == start_page
|
||||||
|
else 0
|
||||||
|
)
|
||||||
y1 = (end_y / 1000) * page.rect.height if page_number == end_page else page.rect.height
|
y1 = (end_y / 1000) * page.rect.height if page_number == end_page else page.rect.height
|
||||||
if y1 <= y0 + 1:
|
clip = pymupdf.Rect(
|
||||||
|
fraction_x0 * page.rect.width, y0,
|
||||||
|
fraction_x1 * page.rect.width, y1,
|
||||||
|
) & visible_bounds[page_number] & page.rect
|
||||||
|
if clip.is_empty or clip.height <= 1 or clip.width <= 1:
|
||||||
continue
|
continue
|
||||||
part_path = temporary / f"part-{index}-{page_number}.pdf"
|
part_path = temporary / f"part-{index}-{page_number}.pdf"
|
||||||
_save_cropped_page(
|
_save_cropped_page(
|
||||||
document,
|
document,
|
||||||
page_number,
|
page_number,
|
||||||
fraction_x0 * page.rect.width,
|
clip.x0,
|
||||||
y0,
|
clip.y0,
|
||||||
fraction_x1 * page.rect.width,
|
clip.x1,
|
||||||
y1,
|
clip.y1,
|
||||||
part_path,
|
part_path,
|
||||||
)
|
)
|
||||||
parts_by_label[clean_label].append(part_path)
|
parts_by_label[clean_label].append(part_path)
|
||||||
@@ -264,4 +291,3 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
raise SystemExit(main())
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
|||||||
@@ -31,6 +31,15 @@ if CONFIG_PATH is None:
|
|||||||
else:
|
else:
|
||||||
_configuration = _load_user_config(CONFIG_PATH)
|
_configuration = _load_user_config(CONFIG_PATH)
|
||||||
|
|
||||||
|
# Keep new optional settings compatible with older personal configuration files.
|
||||||
|
ALWAYS_CROP = False
|
||||||
|
RETURN_JPEG_ENABLED = True
|
||||||
|
RETURN_PDF_ENABLED = True
|
||||||
|
RETURN_ANSWERS_ENABLED = False
|
||||||
|
RETURN_ANSWERS_CONTEXT = False
|
||||||
|
RETURN_ANSWERS_QUESTION = True
|
||||||
|
RETURN_ANSWERS_SOLUTION = False
|
||||||
|
FINAL_SCORE_HISTOGRAM_PATH = Path("histogramme.pdf")
|
||||||
for _name in dir(_configuration):
|
for _name in dir(_configuration):
|
||||||
if not _name.startswith("_"):
|
if not _name.startswith("_"):
|
||||||
globals()[_name] = getattr(_configuration, _name)
|
globals()[_name] = getattr(_configuration, _name)
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""Persistent copy flags shared by page splitting, margin review and the GUI."""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from copienator import CliError, EvaluationWorkspace, atomic_update_json, read_json
|
||||||
|
|
||||||
|
|
||||||
|
def _path(workspace: EvaluationWorkspace) -> Path:
|
||||||
|
return workspace.metadata_dir / "copy_errors.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _validate(value) -> dict[str, str]:
|
||||||
|
if not isinstance(value, dict) or any(
|
||||||
|
not isinstance(name, str) or not isinstance(reason, str)
|
||||||
|
or "/" in name or "\\" in name or Path(name).suffix.casefold() != ".pdf"
|
||||||
|
for name, reason in value.items()
|
||||||
|
):
|
||||||
|
raise CliError("Invalid copy_errors.json: expected PDF filenames and error descriptions")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def copy_errors(workspace: EvaluationWorkspace) -> dict[str, str]:
|
||||||
|
return _validate(read_json(_path(workspace), default={}))
|
||||||
|
|
||||||
|
|
||||||
|
def mark_copy_error(workspace: EvaluationWorkspace, pdf: Path, reason: str) -> None:
|
||||||
|
def update(errors):
|
||||||
|
_validate(errors)[pdf.name] = reason
|
||||||
|
atomic_update_json(_path(workspace), update, default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
def clear_copy_error(workspace: EvaluationWorkspace, pdf: Path) -> None:
|
||||||
|
if not _path(workspace).exists():
|
||||||
|
return
|
||||||
|
def update(errors):
|
||||||
|
_validate(errors).pop(pdf.name, None)
|
||||||
|
atomic_update_json(_path(workspace), update, default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
def marked_copy_paths(workspace: EvaluationWorkspace, *, originals: bool = False) -> list[Path]:
|
||||||
|
directories = ([workspace.original_copies_dir, workspace.copies_dir, workspace.root]
|
||||||
|
if originals else [workspace.copies_dir])
|
||||||
|
result = []
|
||||||
|
for name in sorted(copy_errors(workspace), key=str.casefold):
|
||||||
|
path = next((directory / name for directory in directories if (directory / name).is_file()), None)
|
||||||
|
if path is None:
|
||||||
|
raise CliError(f"Marked copy not found: {name}")
|
||||||
|
result.append(path)
|
||||||
|
return result
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
"""Propose conservative top/bottom crops for scanned, optionally ruled PDFs.
|
||||||
|
|
||||||
|
Run with ``python -m copienator.crop_blank_margins INPUT_DIR OUTPUT_DIR``.
|
||||||
|
Only PDFs directly in INPUT_DIR are processed. Originals are never modified.
|
||||||
|
Analysis is deskewed; output keeps the original scan and changes its CropBox.
|
||||||
|
This is a heuristic review utility, not a guarantee that a scan contains no ink.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import hashlib
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
import pymupdf
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
from copienator.ink_detection import detect_bounds
|
||||||
|
|
||||||
|
|
||||||
|
def apply_bounds(page: pymupdf.Page, top: float, bottom: float) -> None:
|
||||||
|
"""Apply fractional bounds in displayed orientation, respecting old CropBox."""
|
||||||
|
rect = page.rect
|
||||||
|
visible = pymupdf.Rect(0, top*rect.height, rect.width, bottom*rect.height)
|
||||||
|
box = visible * page.derotation_matrix
|
||||||
|
box += (page.cropbox_position.x, page.cropbox_position.y,
|
||||||
|
page.cropbox_position.x, page.cropbox_position.y)
|
||||||
|
page.set_cropbox(box)
|
||||||
|
|
||||||
|
|
||||||
|
def process_pdf(source: Path, destination: Path, review: Path | None,
|
||||||
|
dpi: float, padding_mm: float, min_crop_mm: float,
|
||||||
|
progress=None) -> list[dict]:
|
||||||
|
digest = hashlib.sha256(source.read_bytes()).hexdigest()
|
||||||
|
rows = []
|
||||||
|
with pymupdf.open(source) as doc:
|
||||||
|
for index, page in enumerate(doc):
|
||||||
|
pix = page.get_pixmap(dpi=round(dpi), colorspace=pymupdf.csRGB, alpha=False)
|
||||||
|
rgb = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
|
||||||
|
result = detect_bounds(rgb, dpi, padding_mm, min_crop_mm)
|
||||||
|
top, bottom = result.pop('top_px'), result.pop('bottom_px')
|
||||||
|
row = dict(file=source.name, page=index+1, **result,
|
||||||
|
top_removed_mm=round(top/pix.height*page.rect.height*25.4/72, 2),
|
||||||
|
bottom_removed_mm=round((pix.height-bottom)/pix.height*page.rect.height*25.4/72, 2),
|
||||||
|
original_cropbox=list(page.cropbox), rotation=page.rotation,
|
||||||
|
source_sha256=digest)
|
||||||
|
if review is not None:
|
||||||
|
preview = Image.fromarray(rgb)
|
||||||
|
preview.thumbnail((500, 700))
|
||||||
|
overlay = Image.new('RGBA', preview.size)
|
||||||
|
draw = ImageDraw.Draw(overlay)
|
||||||
|
y0, y1 = top/pix.height*preview.height, bottom/pix.height*preview.height
|
||||||
|
if top:
|
||||||
|
draw.rectangle((0, 0, preview.width, y0), fill=(255, 40, 40, 85))
|
||||||
|
draw.line((0, y0, preview.width, y0), fill=(230, 0, 0, 255), width=2)
|
||||||
|
if bottom < pix.height:
|
||||||
|
draw.rectangle((0, y1, preview.width, preview.height), fill=(255, 40, 40, 85))
|
||||||
|
draw.line((0, y1, preview.width, y1), fill=(230, 0, 0, 255), width=2)
|
||||||
|
preview = Image.alpha_composite(preview.convert('RGBA'), overlay).convert('RGB')
|
||||||
|
preview.save(review/f'{source.stem}-{index+1:03}.jpg', quality=85)
|
||||||
|
if top or bottom < pix.height:
|
||||||
|
apply_bounds(page, top/pix.height, bottom/pix.height)
|
||||||
|
row['output_cropbox'] = list(page.cropbox)
|
||||||
|
rows.append(row)
|
||||||
|
if progress is not None:
|
||||||
|
progress(index+1, len(doc), row)
|
||||||
|
doc.save(destination, garbage=3, deflate=True)
|
||||||
|
with pymupdf.open(destination) as check:
|
||||||
|
if len(check) != len(rows):
|
||||||
|
raise RuntimeError(f'Page count changed: {source}')
|
||||||
|
for page in check:
|
||||||
|
if page.rect.is_empty:
|
||||||
|
raise RuntimeError(f'Empty output page: {destination}')
|
||||||
|
if hashlib.sha256(source.read_bytes()).hexdigest() != digest:
|
||||||
|
raise RuntimeError(f'Source changed during processing: {source}')
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument('input', type=Path)
|
||||||
|
parser.add_argument('output', type=Path)
|
||||||
|
parser.add_argument('--dpi', type=int, default=200)
|
||||||
|
parser.add_argument('--padding-mm', type=float, default=6)
|
||||||
|
parser.add_argument('--min-crop-mm', type=float, default=5)
|
||||||
|
args = parser.parse_args()
|
||||||
|
if args.dpi < 100 or args.padding_mm < 0 or args.min_crop_mm < 0:
|
||||||
|
parser.error('Use dpi >= 100 and nonnegative margins.')
|
||||||
|
sources = sorted(args.input.glob('*.pdf')) if args.input.is_dir() else [args.input]
|
||||||
|
if not sources or any(not p.is_file() for p in sources):
|
||||||
|
parser.error('No input PDFs found.')
|
||||||
|
if any(p.resolve() == (args.output/p.name).resolve() for p in sources):
|
||||||
|
parser.error('Output must not overwrite input PDFs.')
|
||||||
|
args.output.mkdir(parents=True, exist_ok=True)
|
||||||
|
review = args.output/'review'
|
||||||
|
review.mkdir(exist_ok=True)
|
||||||
|
cv2.setNumThreads(2)
|
||||||
|
rows = []
|
||||||
|
for i, source in enumerate(sources):
|
||||||
|
batch = process_pdf(source, args.output/source.name, review,
|
||||||
|
args.dpi, args.padding_mm, args.min_crop_mm)
|
||||||
|
rows.extend(batch)
|
||||||
|
print(f'[{i+1}/{len(sources)}] {source.name}: {len(batch)} pages, '
|
||||||
|
f'{sum(r["top_removed_mm"] > 0 or r["bottom_removed_mm"] > 0 for r in batch)} cropped',
|
||||||
|
flush=True)
|
||||||
|
(args.output/'report.json').write_text(json.dumps(rows, indent=2)+'\n')
|
||||||
|
with (args.output/'report.csv').open('w') as stream:
|
||||||
|
fieldnames = list(dict.fromkeys(key for row in rows for key in row))
|
||||||
|
writer = csv.DictWriter(stream, fieldnames=fieldnames)
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(rows)
|
||||||
|
cards = []
|
||||||
|
for row in rows:
|
||||||
|
stem = Path(row['file']).stem
|
||||||
|
cards.append(f'<article><a href="{html.escape(row["file"])}#page={row["page"]}">'
|
||||||
|
f'{html.escape(stem)} / {row["page"]}</a>'
|
||||||
|
f'<p>Top: {row["top_removed_mm"]} mm · Bottom: {row["bottom_removed_mm"]} mm'
|
||||||
|
f' · {row["status"]}</p>'
|
||||||
|
f'<img loading="lazy" src="review/{html.escape(stem)}-{row["page"]:03}.jpg"></article>')
|
||||||
|
(args.output/'index.html').write_text(
|
||||||
|
'<!doctype html><meta charset="utf-8"><title>Crop review</title>'
|
||||||
|
'<style>body{font:15px system-ui;background:#eee;margin:24px}'
|
||||||
|
'main{display:grid;grid-template-columns:repeat(auto-fill,minmax(330px,1fr));gap:20px}'
|
||||||
|
'article{background:white;padding:12px}img{width:100%}p{font-size:12px}</style>'
|
||||||
|
'<h1>Crop review</h1><p>Red shading shows the removed areas on the original scan. '
|
||||||
|
'Click a page title to open the processed PDF. Review statuses flag uncertain detections.</p>'
|
||||||
|
'<main>'+''.join(cards)+'</main>')
|
||||||
|
changed = sum(r['top_removed_mm'] > 0 or r['bottom_removed_mm'] > 0 for r in rows)
|
||||||
|
print(f'Done: {len(sources)} PDFs, {len(rows)} pages, {changed} cropped. Review: {args.output / "index.html"}')
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,298 @@
|
|||||||
|
"""Propose large bottom-only crops for PDFs already split into exercises."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import hashlib
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
import multiprocessing
|
||||||
|
import signal
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
import pymupdf
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
from copienator.crop_blank_margins import apply_bounds
|
||||||
|
from copienator.filesystem import staged_directory
|
||||||
|
from copienator.ink_detection import detect_bounds
|
||||||
|
|
||||||
|
|
||||||
|
LINES_PER_PAGE = 36
|
||||||
|
MINIMUM_HEIGHT_LINES = 10
|
||||||
|
IGNORED_BOTTOM_LINES = 0.75
|
||||||
|
MINIMUM_CROP_LINES = 4
|
||||||
|
|
||||||
|
|
||||||
|
def _displayed_media_height(page: pymupdf.Page) -> float:
|
||||||
|
"""Return the uncropped sheet height in the page's displayed orientation."""
|
||||||
|
return page.mediabox.width if page.rotation % 180 else page.mediabox.height
|
||||||
|
|
||||||
|
|
||||||
|
def _full_page_height(copy_pdf: Path) -> float:
|
||||||
|
with pymupdf.open(copy_pdf) as document:
|
||||||
|
if not len(document):
|
||||||
|
raise ValueError(f"PDF sans page : {copy_pdf}")
|
||||||
|
return max(_displayed_media_height(page) for page in document)
|
||||||
|
|
||||||
|
|
||||||
|
def _save_review(rgb: np.ndarray, bottom: int, destination: Path) -> None:
|
||||||
|
preview = Image.fromarray(rgb)
|
||||||
|
preview.thumbnail((500, 700))
|
||||||
|
overlay = Image.new("RGBA", preview.size)
|
||||||
|
draw = ImageDraw.Draw(overlay)
|
||||||
|
y = bottom / rgb.shape[0] * preview.height
|
||||||
|
draw.rectangle((0, y, preview.width, preview.height), fill=(255, 40, 40, 85))
|
||||||
|
draw.line((0, y, preview.width, y), fill=(230, 0, 0, 255), width=2)
|
||||||
|
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
Image.alpha_composite(preview.convert("RGBA"), overlay).convert("RGB").save(
|
||||||
|
destination, quality=88
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def process_exercise_pdf(
|
||||||
|
source: Path,
|
||||||
|
destination: Path,
|
||||||
|
review_dir: Path | None,
|
||||||
|
full_page_height: float,
|
||||||
|
*,
|
||||||
|
dpi: int = 200,
|
||||||
|
padding_mm: float = 6,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Crop qualifying pages and save the PDF only when at least one changes."""
|
||||||
|
digest = hashlib.sha256(source.read_bytes()).hexdigest()
|
||||||
|
line_points = full_page_height / LINES_PER_PAGE
|
||||||
|
records: list[dict] = []
|
||||||
|
changed = False
|
||||||
|
with pymupdf.open(source) as document:
|
||||||
|
page_count = len(document)
|
||||||
|
for index, page in enumerate(document):
|
||||||
|
visible_height = page.rect.height
|
||||||
|
record = {
|
||||||
|
"file": source.as_posix(),
|
||||||
|
"page": index + 1,
|
||||||
|
"page_count": page_count,
|
||||||
|
"source_sha256": digest,
|
||||||
|
"height_lines": round(visible_height / line_points, 2),
|
||||||
|
"bottom_removed_lines": 0.0,
|
||||||
|
"bottom_removed_mm": 0.0,
|
||||||
|
"status": "skipped-short",
|
||||||
|
}
|
||||||
|
if visible_height + 1e-6 < MINIMUM_HEIGHT_LINES * line_points:
|
||||||
|
records.append(record)
|
||||||
|
continue
|
||||||
|
|
||||||
|
pixmap = page.get_pixmap(
|
||||||
|
dpi=dpi, colorspace=pymupdf.csRGB, alpha=False
|
||||||
|
)
|
||||||
|
rgb = np.frombuffer(pixmap.samples, np.uint8).reshape(
|
||||||
|
pixmap.height, pixmap.width, 3
|
||||||
|
)
|
||||||
|
pixels_per_point = pixmap.height / visible_height
|
||||||
|
ignored_pixels = min(
|
||||||
|
pixmap.height - 1,
|
||||||
|
round(IGNORED_BOTTOM_LINES * line_points * pixels_per_point),
|
||||||
|
)
|
||||||
|
analysis_bottom = pixmap.height - ignored_pixels
|
||||||
|
detection = detect_bounds(
|
||||||
|
rgb[:analysis_bottom], dpi=dpi, padding_mm=padding_mm, min_crop_mm=0
|
||||||
|
)
|
||||||
|
proposed_bottom = detection["bottom_px"]
|
||||||
|
removed_points = (pixmap.height - proposed_bottom) / pixels_per_point
|
||||||
|
removed_lines = removed_points / line_points
|
||||||
|
record["detector_status"] = detection["status"]
|
||||||
|
record["proposed_bottom_removed_lines"] = round(removed_lines, 2)
|
||||||
|
if removed_lines + 1e-6 < MINIMUM_CROP_LINES:
|
||||||
|
record["status"] = "unchanged-small-crop"
|
||||||
|
records.append(record)
|
||||||
|
continue
|
||||||
|
|
||||||
|
apply_bounds(page, 0, proposed_bottom / pixmap.height)
|
||||||
|
record.update(
|
||||||
|
bottom_removed_lines=round(removed_lines, 2),
|
||||||
|
bottom_removed_mm=round(removed_points * 25.4 / 72, 2),
|
||||||
|
status="cropped",
|
||||||
|
output_height_points=round(page.rect.height, 3),
|
||||||
|
)
|
||||||
|
if review_dir is not None:
|
||||||
|
record["output"] = destination.relative_to(review_dir.parent).as_posix()
|
||||||
|
review_path = review_dir / source.parent.name / (
|
||||||
|
f"{source.stem}-p{index + 1:02}.jpg"
|
||||||
|
)
|
||||||
|
_save_review(rgb, proposed_bottom, review_path)
|
||||||
|
record["review"] = review_path.relative_to(review_dir.parent).as_posix()
|
||||||
|
changed = True
|
||||||
|
records.append(record)
|
||||||
|
|
||||||
|
if changed:
|
||||||
|
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
document.save(destination, garbage=3, deflate=True)
|
||||||
|
|
||||||
|
if changed:
|
||||||
|
with pymupdf.open(destination) as check:
|
||||||
|
if len(check) != page_count:
|
||||||
|
raise RuntimeError(f"Nombre de pages modifié : {source}")
|
||||||
|
for page in check:
|
||||||
|
if page.rect.is_empty:
|
||||||
|
raise RuntimeError(f"Page vide produite : {destination}")
|
||||||
|
page.get_pixmap(matrix=pymupdf.Matrix(0.25, 0.25))
|
||||||
|
if hashlib.sha256(source.read_bytes()).hexdigest() != digest:
|
||||||
|
raise RuntimeError(f"PDF source modifié pendant le rognage : {source}")
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def _initialize_worker() -> None:
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
||||||
|
|
||||||
|
|
||||||
|
def _process_job(job: tuple[Path, Path, Path, float, int, float]) -> list[dict]:
|
||||||
|
source, destination, review_dir, full_height, dpi, padding = job
|
||||||
|
records = process_exercise_pdf(
|
||||||
|
source, destination, review_dir, full_height, dpi=dpi, padding_mm=padding
|
||||||
|
)
|
||||||
|
changed = sum(record["status"] == "cropped" for record in records)
|
||||||
|
print(f"{source.parent.name}/{source.name} : {changed}/{len(records)} page(s) rognée(s)",
|
||||||
|
flush=True)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def _write_index(output: Path, records: list[dict]) -> None:
|
||||||
|
changed = [record for record in records if record["status"] == "cropped"]
|
||||||
|
changed_files: dict[str, list[int]] = {}
|
||||||
|
for record in changed:
|
||||||
|
changed_files.setdefault(record["file"], []).append(record["page"])
|
||||||
|
(output / "cropped-files.txt").write_text(
|
||||||
|
"".join(
|
||||||
|
f"{path} - page(s) {', '.join(map(str, pages))}\n"
|
||||||
|
for path, pages in changed_files.items()
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
cards = []
|
||||||
|
for record in changed:
|
||||||
|
relative = Path(record["file"])
|
||||||
|
output_pdf = quote(record["output"])
|
||||||
|
preview = quote(record["review"])
|
||||||
|
name = html.escape(f"{relative.parent.name}/{relative.name} - page {record['page']}")
|
||||||
|
cards.append(
|
||||||
|
f'<article><a href="{output_pdf}#page={record["page"]}">{name}</a>'
|
||||||
|
f'<p>{record["bottom_removed_lines"]:.2f} lignes '
|
||||||
|
f'({record["bottom_removed_mm"]:.1f} mm) retirées</p>'
|
||||||
|
f'<img loading="lazy" src="{preview}"></article>'
|
||||||
|
)
|
||||||
|
(output / "index.html").write_text(
|
||||||
|
'<!doctype html><meta charset="utf-8"><title>Rognage bas des exercices</title>'
|
||||||
|
'<style>body{font:15px system-ui;background:#eee;margin:24px}'
|
||||||
|
'main{display:grid;grid-template-columns:repeat(auto-fill,minmax(330px,1fr));gap:20px}'
|
||||||
|
'article{background:white;padding:12px}img{width:100%}p{font-size:12px}</style>'
|
||||||
|
f'<h1>{len(changed)} pages rognées</h1>'
|
||||||
|
'<p>Le rouge montre la zone retirée. Seuls les PDF modifiés sont présents dans cropped/.</p>'
|
||||||
|
'<main>' + ''.join(cards) + '</main>',
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run(
|
||||||
|
input_path: Path,
|
||||||
|
output: Path,
|
||||||
|
*,
|
||||||
|
dpi: int = 200,
|
||||||
|
padding_mm: float = 6,
|
||||||
|
workers: int = 5,
|
||||||
|
) -> list[dict]:
|
||||||
|
copies = input_path / "Copies" if (input_path / "Copies").is_dir() else input_path
|
||||||
|
if not copies.is_dir():
|
||||||
|
raise ValueError(f"Dossier Copies introuvable : {input_path}")
|
||||||
|
sources = sorted(
|
||||||
|
copies.glob("Copie*/*.pdf"),
|
||||||
|
key=lambda path: (path.parent.name.casefold(), path.name.casefold()),
|
||||||
|
)
|
||||||
|
if not sources:
|
||||||
|
raise ValueError(f"Aucun PDF d'exercice trouvé dans {copies}")
|
||||||
|
if workers < 1:
|
||||||
|
raise ValueError("Le nombre de traitements parallèles doit être positif")
|
||||||
|
|
||||||
|
heights: dict[str, float] = {}
|
||||||
|
for copy_name in sorted({source.parent.name for source in sources}):
|
||||||
|
copy_pdf = copies / f"{copy_name}.pdf"
|
||||||
|
if not copy_pdf.is_file():
|
||||||
|
raise ValueError(f"PDF source introuvable : {copy_pdf}")
|
||||||
|
heights[copy_name] = _full_page_height(copy_pdf)
|
||||||
|
|
||||||
|
with staged_directory(output) as staging:
|
||||||
|
cropped_dir = staging / "cropped"
|
||||||
|
review_dir = staging / "review"
|
||||||
|
jobs = [
|
||||||
|
(
|
||||||
|
source,
|
||||||
|
cropped_dir / source.relative_to(copies),
|
||||||
|
review_dir,
|
||||||
|
heights[source.parent.name],
|
||||||
|
dpi,
|
||||||
|
padding_mm,
|
||||||
|
)
|
||||||
|
for source in sources
|
||||||
|
]
|
||||||
|
count = min(workers, len(jobs))
|
||||||
|
if count == 1:
|
||||||
|
previous_threads = cv2.getNumThreads()
|
||||||
|
cv2.setNumThreads(1)
|
||||||
|
try:
|
||||||
|
batches = [_process_job(job) for job in jobs]
|
||||||
|
finally:
|
||||||
|
cv2.setNumThreads(previous_threads)
|
||||||
|
else:
|
||||||
|
with multiprocessing.get_context("spawn").Pool(
|
||||||
|
count, _initialize_worker
|
||||||
|
) as pool:
|
||||||
|
batches = list(pool.imap_unordered(_process_job, jobs))
|
||||||
|
order = {source.as_posix(): i for i, source in enumerate(sources)}
|
||||||
|
records = sorted(
|
||||||
|
(record for batch in batches for record in batch),
|
||||||
|
key=lambda record: (order[record["file"]], record["page"]),
|
||||||
|
)
|
||||||
|
(staging / "report.json").write_text(
|
||||||
|
json.dumps(records, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
with (staging / "report.csv").open("w", encoding="utf-8", newline="") as stream:
|
||||||
|
fields = sorted({key for record in records for key in record})
|
||||||
|
writer = csv.DictWriter(stream, fieldnames=fields)
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(records)
|
||||||
|
_write_index(staging, records)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("input", type=Path, help="Évaluation ou dossier Copies")
|
||||||
|
parser.add_argument("output", type=Path)
|
||||||
|
parser.add_argument("--dpi", type=int, default=200)
|
||||||
|
parser.add_argument("--padding-mm", type=float, default=6)
|
||||||
|
parser.add_argument("--workers", type=int, default=5)
|
||||||
|
arguments = parser.parse_args()
|
||||||
|
if arguments.dpi < 100 or arguments.padding_mm < 0:
|
||||||
|
parser.error("Utilisez dpi >= 100 et une marge positive ou nulle")
|
||||||
|
try:
|
||||||
|
records = run(
|
||||||
|
arguments.input,
|
||||||
|
arguments.output,
|
||||||
|
dpi=arguments.dpi,
|
||||||
|
padding_mm=arguments.padding_mm,
|
||||||
|
workers=arguments.workers,
|
||||||
|
)
|
||||||
|
except ValueError as error:
|
||||||
|
parser.error(str(error))
|
||||||
|
cropped = sum(record["status"] == "cropped" for record in records)
|
||||||
|
files = len({record["file"] for record in records if record["status"] == "cropped"})
|
||||||
|
print(f"Terminé : {cropped} page(s) dans {files} PDF rognée(s). Revue : "
|
||||||
|
f"{arguments.output / 'index.html'}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -17,6 +17,10 @@ COMMANDS: dict[str, Command] = {
|
|||||||
"statement-personal": Command("enonce_info", "Generate personal statement metadata"),
|
"statement-personal": Command("enonce_info", "Generate personal statement metadata"),
|
||||||
"copies": Command("copies_tools", "Rotate or rename scanned copies"),
|
"copies": Command("copies_tools", "Rotate or rename scanned copies"),
|
||||||
"page-split": Command("page_splitter", "Split and reorder scanned PDF pages"),
|
"page-split": Command("page_splitter", "Split and reorder scanned PDF pages"),
|
||||||
|
"crop-margins": Command("crop_margins", "Trim blank top and bottom margins in Copies"),
|
||||||
|
"crop-answer-bottoms": Command(
|
||||||
|
"crop_exercise_bottoms", "Trim large blank bottoms from split answers"
|
||||||
|
),
|
||||||
"crop-labels": Command("cutleft", "Crop the label margin from copies"),
|
"crop-labels": Command("cutleft", "Crop the label margin from copies"),
|
||||||
"labels": Command("gemini_for_labels", "Detect question labels with Gemini"),
|
"labels": Command("gemini_for_labels", "Detect question labels with Gemini"),
|
||||||
"review-labels": Command("plotting", "Review detected labels interactively"),
|
"review-labels": Command("plotting", "Review detected labels interactively"),
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
"""Geometry checks shared by correction and annotation rendering."""
|
||||||
|
|
||||||
|
import math
|
||||||
|
from numbers import Real
|
||||||
|
|
||||||
|
|
||||||
|
def valid_feedback_box(box) -> bool:
|
||||||
|
return (
|
||||||
|
isinstance(box, (list, tuple)) and len(box) == 4
|
||||||
|
and all(isinstance(value, Real) and not isinstance(value, bool)
|
||||||
|
and math.isfinite(value) for value in box)
|
||||||
|
and box[0] < box[2] and box[1] < box[3]
|
||||||
|
)
|
||||||
@@ -0,0 +1,273 @@
|
|||||||
|
"""Detect top and bottom content bounds in scanned student work.
|
||||||
|
|
||||||
|
Strong coloured strokes are never erased merely because they coincide with
|
||||||
|
paper ruling. The detector supports coloured handwriting and dark ruled scans;
|
||||||
|
it remains conservative for pencil-only scans and heavily saturated grids.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from copienator.paper_background import _skew, _ruling, _foreground, _content_mask
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=8)
|
||||||
|
def _paper_kernel(sigma: float) -> np.ndarray:
|
||||||
|
"""Match OpenCV's uint8 Gaussian coefficients, including error diffusion.
|
||||||
|
|
||||||
|
See getGaussianKernelFixedPoint_ED in OpenCV's smooth.dispatch.cpp.
|
||||||
|
The 8-bit coefficients allow exact sums in the faster float filter path.
|
||||||
|
"""
|
||||||
|
size = round(sigma*6+1) | 1
|
||||||
|
kernel = cv2.getGaussianKernel(size, sigma).ravel()
|
||||||
|
fixed = np.zeros(size, np.float32)
|
||||||
|
error = 0.0
|
||||||
|
for i in range(size//2):
|
||||||
|
value = kernel[i]*256+error
|
||||||
|
weight = round(value)
|
||||||
|
error = value-weight
|
||||||
|
fixed[i] = fixed[-1-i] = weight
|
||||||
|
fixed[size//2] = 256-fixed.sum()
|
||||||
|
fixed /= 256
|
||||||
|
return fixed
|
||||||
|
|
||||||
|
|
||||||
|
def _paper_blur(gray: np.ndarray, sigma: float) -> np.ndarray:
|
||||||
|
kernel = _paper_kernel(sigma)
|
||||||
|
blurred = cv2.sepFilter2D(gray, cv2.CV_32F, kernel, kernel)
|
||||||
|
# GaussianBlur rounds positive half-integers upward, rather than to even.
|
||||||
|
np.add(blurred, .5, out=blurred)
|
||||||
|
np.floor(blurred, out=blurred)
|
||||||
|
return blurred.astype(np.uint8)
|
||||||
|
|
||||||
|
|
||||||
|
def _large_blank_ink(gray: np.ndarray, clean: np.ndarray, dpi: float):
|
||||||
|
"""Refine confirmed ruling, only accepting substantial extra blank margins.
|
||||||
|
|
||||||
|
Short directional openings tolerate locally bent/broken paper lines. A
|
||||||
|
physical component-size threshold rejects their remaining tiny fragments.
|
||||||
|
This is deliberately limited to already-confirmed ruled paper.
|
||||||
|
"""
|
||||||
|
px = dpi / 25.4
|
||||||
|
height, width = gray.shape
|
||||||
|
dark = 255-gray
|
||||||
|
length = max(9, round(2.5*px))
|
||||||
|
tolerance = max(3, round(.6*px) | 1)
|
||||||
|
lines = []
|
||||||
|
for horizontal in (True, False):
|
||||||
|
broadened = cv2.dilate(dark, np.ones(
|
||||||
|
(tolerance, 1) if horizontal else (1, tolerance), np.uint8))
|
||||||
|
lines.append(cv2.morphologyEx(broadened, cv2.MORPH_OPEN, np.ones(
|
||||||
|
(1, length) if horizontal else (length, 1), np.uint8)))
|
||||||
|
residual = cv2.subtract(dark, np.maximum(*lines))
|
||||||
|
n, labels, stats, centers = cv2.connectedComponentsWithStats(
|
||||||
|
(residual > 35).astype(np.uint8), 8)
|
||||||
|
keep = np.zeros(n, bool)
|
||||||
|
xx = stats[1:, 0]
|
||||||
|
ww, hh, area = stats[1:, 2], stats[1:, 3], stats[1:, 4]
|
||||||
|
density = area / (ww*hh)
|
||||||
|
shortest, longest = np.minimum(ww, hh), np.maximum(ww, hh)
|
||||||
|
compact_ink = ((area >= .8*px*px) & (shortest >= .6*px)
|
||||||
|
& (density > .3) & (longest < 4*shortest))
|
||||||
|
# Ruling suppression can fragment faint pencil handwriting into sparse,
|
||||||
|
# elongated components. Admit those moderately more readily in the central
|
||||||
|
# 80% of the sheet. The outermost 9% deliberately uses a stricter filter:
|
||||||
|
# punched holes, torn binding edges, and page numbers usually occur there.
|
||||||
|
center_x = xx + ww/2
|
||||||
|
central = (center_x > width*.1) & (center_x < width*.9)
|
||||||
|
central_ink = (central & (area >= .55*px*px) & (shortest >= .45*px)
|
||||||
|
& (density > .18) & (longest < 6*shortest))
|
||||||
|
outer = (center_x < width*.09) | (center_x > width*.91)
|
||||||
|
outer_ink = (outer & (area >= 1.2*px*px) & (shortest >= .8*px)
|
||||||
|
& (density >= .4) & (longest < 3*shortest))
|
||||||
|
keep[1:] = np.where(outer, outer_ink, compact_ink | central_ink)
|
||||||
|
# Use side columns as a prior, then require repeated size and alignment. An
|
||||||
|
# isolated note in the same column remains eligible to protect the margin.
|
||||||
|
candidates = np.flatnonzero(
|
||||||
|
((xx < 15*px) | (xx+ww > width-15*px))
|
||||||
|
& (ww > px) & (ww < 9*px) & (hh > px) & (hh < 12*px)) + 1
|
||||||
|
if len(candidates) > 128:
|
||||||
|
# Bound the matching cost and retain ambiguous, very noisy margins.
|
||||||
|
return clean, 0, False
|
||||||
|
holes = set()
|
||||||
|
for i in candidates:
|
||||||
|
matches = []
|
||||||
|
for j in candidates:
|
||||||
|
if (abs(centers[i, 0]-centers[j, 0]) < 2*px
|
||||||
|
and .6 < stats[j, 2]/stats[i, 2] < 1.6
|
||||||
|
and .6 < stats[j, 3]/stats[i, 3] < 1.6):
|
||||||
|
matches.append(j)
|
||||||
|
if len(matches) >= 3 and np.ptp(centers[matches, 1]) > height*.35:
|
||||||
|
holes.update(matches)
|
||||||
|
keep[list(holes)] = False
|
||||||
|
refined = keep[labels]
|
||||||
|
# Directional opening also removes long fraction bars. Protect very dark,
|
||||||
|
# thick straight strokes independently, even if ruling crosses their ends.
|
||||||
|
long_strokes = cv2.morphologyEx((gray < 50).astype(np.uint8), cv2.MORPH_OPEN,
|
||||||
|
np.ones((1, max(9, round(width*.1))), np.uint8))
|
||||||
|
count, lab, st, _ = cv2.connectedComponentsWithStats(long_strokes, 8)
|
||||||
|
bars = np.zeros(count, bool)
|
||||||
|
bw, bh, ba = st[1:, 2], st[1:, 3], st[1:, 4]
|
||||||
|
bars[1:] = ((bw > width*.1) & (bh >= .3*px) & (bh < height*.015)
|
||||||
|
& (bw > 8*bh) & (ba/(bw*bh) > .5))
|
||||||
|
strong_ruling, strong_lines = _ruling((gray < 50).astype(np.uint8)*255, True)
|
||||||
|
if strong_lines:
|
||||||
|
# A family of equally dark parallel lines is paper, not fraction bars.
|
||||||
|
overlap = np.bincount(lab[strong_ruling > 0], minlength=count)
|
||||||
|
bars &= overlap < st[:, 4]*.5
|
||||||
|
refined |= bars[lab]
|
||||||
|
ys = np.flatnonzero(np.any(refined, axis=1))
|
||||||
|
original = np.flatnonzero(np.any(clean, axis=1))
|
||||||
|
if not len(ys) or not len(original):
|
||||||
|
return clean, 0, False
|
||||||
|
# Leave ordinary small crops to the more permissive detector. Keep a
|
||||||
|
# recovery neighbourhood around the refined bounds for broken/faint strokes.
|
||||||
|
top, bottom = max(0, int(ys.min()-2*px)), min(height, int(ys.max()+1+2*px))
|
||||||
|
result = clean.copy()
|
||||||
|
changed = False
|
||||||
|
if top-original.min() >= 30*px:
|
||||||
|
result[:top] = 0
|
||||||
|
changed = True
|
||||||
|
if original.max()+1-bottom >= 30*px:
|
||||||
|
result[bottom:] = 0
|
||||||
|
changed = True
|
||||||
|
return result, len(holes), changed
|
||||||
|
|
||||||
|
|
||||||
|
def _neutral_paper_foreground(gray: np.ndarray, chroma: np.ndarray, dpi: float):
|
||||||
|
"""Clean confirmed dark ruling before it can seed whole pages.
|
||||||
|
|
||||||
|
Returns None for ordinary ink components or unconfirmed paper geometry.
|
||||||
|
The mask is transformed back to the original displayed pixel coordinates.
|
||||||
|
"""
|
||||||
|
height, width = gray.shape
|
||||||
|
# Only neutral darkness is relevant here: long blue equations are not
|
||||||
|
# evidence of dark paper. Broken ruling can form several medium-sized
|
||||||
|
# components instead of one page-spanning component. The broader threshold
|
||||||
|
# also admits faded gray grids; periodic ruling must still be confirmed.
|
||||||
|
_, _, stats, _ = cv2.connectedComponentsWithStats(
|
||||||
|
((gray < 160) & (chroma < 30)).astype(np.uint8), 8)
|
||||||
|
spans = np.maximum(stats[1:,2]/width, stats[1:,3]/height)
|
||||||
|
if not (np.any(spans > .35) or np.count_nonzero(spans > .1) >= 3):
|
||||||
|
return None, dict(paper_cleanup=False)
|
||||||
|
angle = _skew(gray, angle_step=.5)
|
||||||
|
matrix = cv2.getRotationMatrix2D((width/2, height/2), angle, 1)
|
||||||
|
corners = np.array([[0,0],[width,0],[0,height],[width,height]], dtype=float)
|
||||||
|
corners = cv2.transform(corners[None], matrix)[0]
|
||||||
|
origin = np.floor(corners.min(axis=0))
|
||||||
|
size = np.ceil(corners.max(axis=0)-origin).astype(int)
|
||||||
|
matrix[:, 2] -= origin
|
||||||
|
deskewed = cv2.warpAffine(gray, matrix, tuple(size), borderValue=255)
|
||||||
|
background = _paper_blur(deskewed, dpi/8)
|
||||||
|
normalized = cv2.divide(deskewed, np.maximum(background,1), scale=255)
|
||||||
|
block = max(15, int(dpi/5) | 1)
|
||||||
|
binary = cv2.adaptiveThreshold(normalized,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
|
||||||
|
cv2.THRESH_BINARY_INV,block,9)
|
||||||
|
horizontal, nh = _ruling(binary,True)
|
||||||
|
vertical, nv = _ruling(binary,False)
|
||||||
|
if not (nh or nv):
|
||||||
|
return None, dict(paper_cleanup=False)
|
||||||
|
# The permissive mask retains faint, isolated marks.
|
||||||
|
# Bands identify where ruling is expected, but only actual dark pixels
|
||||||
|
# inside them may be suppressed. Erasing the complete band loses faint
|
||||||
|
# writing alongside a dark grid line.
|
||||||
|
paper_pixels = (horizontal|vertical) & ((normalized < 160).astype(np.uint8)*255)
|
||||||
|
clean, holes = _content_mask(_foreground(normalized,dpi,9,paper_pixels),dpi,nh,nv)
|
||||||
|
clean, repeated_holes, refined = _large_blank_ink(deskewed, clean, dpi)
|
||||||
|
restored = cv2.warpAffine(clean, cv2.invertAffineTransform(matrix),
|
||||||
|
(width,height), flags=cv2.INTER_NEAREST, borderValue=0)
|
||||||
|
return restored > 0, dict(paper_cleanup=True, angle_deg=round(angle,3),
|
||||||
|
horizontal_lines=nh,vertical_lines=nv,
|
||||||
|
edge_artifacts=holes+repeated_holes,
|
||||||
|
large_blank_refinement=refined)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_mask(mask: np.ndarray, px: float) -> np.ndarray:
|
||||||
|
n, labels, stats, _ = cv2.connectedComponentsWithStats(mask.astype(np.uint8), 8)
|
||||||
|
keep = np.zeros(n, bool)
|
||||||
|
keep[1:] = ((stats[1:, 4] >= max(4, .12*px*px))
|
||||||
|
& (stats[1:, 2] >= max(2, round(.35*px)))
|
||||||
|
& (stats[1:, 3] >= max(2, round(.35*px))))
|
||||||
|
return keep[labels]
|
||||||
|
|
||||||
|
|
||||||
|
def detect_bounds(rgb: np.ndarray, dpi: float = 200, padding_mm: float = 6,
|
||||||
|
min_crop_mm: float = 5) -> dict:
|
||||||
|
"""Locate strong ink, recover adjacent faint strokes, and retain padding.
|
||||||
|
|
||||||
|
Neutral punched-hole shadows usually have neither sufficient chroma nor
|
||||||
|
sufficient darkness to seed a region. Nothing is discarded merely because
|
||||||
|
it is in a side margin. Two seed thresholds expose unstable boundaries.
|
||||||
|
"""
|
||||||
|
px = dpi/25.4
|
||||||
|
h, w = rgb.shape[:2]
|
||||||
|
red, green, blue = cv2.split(rgb)
|
||||||
|
lowest = cv2.min(cv2.min(red, green), blue)
|
||||||
|
chroma = cv2.subtract(cv2.max(cv2.max(red, green), blue), lowest)
|
||||||
|
gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
|
||||||
|
darkness = 255-lowest
|
||||||
|
length = max(15, round(5.2*px)) | 1
|
||||||
|
horizontal = cv2.morphologyEx(darkness, cv2.MORPH_OPEN,
|
||||||
|
np.ones((1, length), np.uint8))
|
||||||
|
vertical = cv2.morphologyEx(darkness, cv2.MORPH_OPEN,
|
||||||
|
np.ones((length, 1), np.uint8))
|
||||||
|
residual = cv2.subtract(darkness, np.maximum(horizontal, vertical))
|
||||||
|
# Strong strokes bypass the line-background estimate entirely: even a long
|
||||||
|
# isolated black or coloured fraction bar must survive. Local contrast is
|
||||||
|
# used only to recover weaker surrounding strokes.
|
||||||
|
neutral_ink = gray < 95
|
||||||
|
cleaned, paper_info = _neutral_paper_foreground(gray, chroma, dpi)
|
||||||
|
if cleaned is not None:
|
||||||
|
neutral_ink = cleaned
|
||||||
|
weak = ((chroma > 60) | ((gray < 175) & (residual > 25))).astype(np.uint8)
|
||||||
|
radius = max(1, round(2*px))
|
||||||
|
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2*radius+1, 2*radius+1))
|
||||||
|
extents = []
|
||||||
|
counts = []
|
||||||
|
same_thresholds = not np.any((chroma > 100) & (chroma <= 115) & ~neutral_ink)
|
||||||
|
for threshold in (100, 115):
|
||||||
|
if threshold == 115 and same_thresholds:
|
||||||
|
counts.append(counts[0])
|
||||||
|
if extents:
|
||||||
|
extents.append(extents[0])
|
||||||
|
continue
|
||||||
|
seeds = _seed_mask((chroma > threshold) | neutral_ink, px)
|
||||||
|
counts.append(int(np.count_nonzero(seeds)))
|
||||||
|
if not np.any(seeds):
|
||||||
|
continue
|
||||||
|
# Limit weak recovery to a physical neighbourhood: faint grid lines
|
||||||
|
# connected to a letter cannot grow into a full-page foreground mask.
|
||||||
|
nearby = cv2.dilate(seeds.astype(np.uint8), kernel)
|
||||||
|
candidate = (weak & nearby) | seeds.astype(np.uint8)
|
||||||
|
n, labels = cv2.connectedComponents(candidate, 8)
|
||||||
|
seeded_labels = np.zeros(n, bool)
|
||||||
|
seeded_labels[np.unique(labels[seeds])] = True
|
||||||
|
seeded_labels[0] = False
|
||||||
|
ys = np.flatnonzero(np.any(seeded_labels[labels], axis=1))
|
||||||
|
extents.append((int(ys.min()), int(ys.max())+1))
|
||||||
|
result = dict(top_px=0, bottom_px=h, angle_deg=None,
|
||||||
|
horizontal_lines=0, vertical_lines=0, edge_artifacts=0,
|
||||||
|
detector="ink", seed_pixels=counts, status="review-no-ink-seeds")
|
||||||
|
result.update(paper_info)
|
||||||
|
if not extents:
|
||||||
|
return result
|
||||||
|
pad = padding_mm*px
|
||||||
|
top = max(0, int(np.floor(min(e[0] for e in extents)-pad)))
|
||||||
|
bottom = min(h, int(np.ceil(max(e[1] for e in extents)+pad)))
|
||||||
|
uncertain = []
|
||||||
|
if len(extents) < 2:
|
||||||
|
uncertain.append('seed-threshold')
|
||||||
|
else:
|
||||||
|
for edge, name in ((0, 'top'), (1, 'bottom')):
|
||||||
|
if abs(extents[0][edge]-extents[1][edge]) > max(pad, 3*px):
|
||||||
|
uncertain.append(name)
|
||||||
|
if top < min_crop_mm*px:
|
||||||
|
top = 0
|
||||||
|
if h-bottom < min_crop_mm*px:
|
||||||
|
bottom = h
|
||||||
|
result.update(top_px=top, bottom_px=bottom,
|
||||||
|
status=('review-'+'-'.join(uncertain) if uncertain else
|
||||||
|
'cropped' if top or bottom < h else 'unchanged'))
|
||||||
|
return result
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
"""Shared geometry and foreground helpers for scanned paper backgrounds."""
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
def _runs(values: np.ndarray) -> list[tuple[int, int]]:
|
||||||
|
edges = np.diff(np.r_[False, values, False].astype(np.int8))
|
||||||
|
return list(zip(np.flatnonzero(edges == 1), np.flatnonzero(edges == -1)))
|
||||||
|
|
||||||
|
|
||||||
|
def _skew(gray: np.ndarray, angle_step: float = .1) -> float:
|
||||||
|
"""Use the dominant near-horizontal/vertical Hough angle, at reduced size."""
|
||||||
|
scale = min(1.0, 1200 / max(gray.shape))
|
||||||
|
small = cv2.resize(gray, None, fx=scale, fy=scale)
|
||||||
|
edges = cv2.Canny(small, 40, 120)
|
||||||
|
lines = cv2.HoughLinesP(edges, 1, np.deg2rad(angle_step), 60,
|
||||||
|
minLineLength=min(small.shape) * .16, maxLineGap=12)
|
||||||
|
if lines is None:
|
||||||
|
return 0.0
|
||||||
|
angles, weights = [], []
|
||||||
|
for x0, y0, x1, y1 in lines.reshape(-1, 4):
|
||||||
|
a = (np.degrees(np.arctan2(y1-y0, x1-x0)) + 45) % 90 - 45
|
||||||
|
if abs(a) <= 5:
|
||||||
|
angles.append(a)
|
||||||
|
weights.append(np.hypot(x1-x0, y1-y0))
|
||||||
|
return _dominant_angle(angles, weights)
|
||||||
|
|
||||||
|
|
||||||
|
def _dominant_angle(angles, weights) -> float:
|
||||||
|
if len(angles) < 4:
|
||||||
|
return 0.0
|
||||||
|
angles, weights = np.array(angles), np.array(weights)
|
||||||
|
bins = np.arange(-5.125, 5.126, .25)
|
||||||
|
hist, _ = np.histogram(angles, bins, weights=weights)
|
||||||
|
peak = (bins[hist.argmax()] + bins[hist.argmax()+1]) / 2
|
||||||
|
near = abs(angles-peak) < .4
|
||||||
|
if weights[near].sum() < .35 * weights.sum():
|
||||||
|
return 0.0
|
||||||
|
return float(np.average(angles[near], weights=weights[near]))
|
||||||
|
|
||||||
|
|
||||||
|
def _ruling(binary: np.ndarray, horizontal: bool) -> tuple[np.ndarray, int]:
|
||||||
|
"""Accept a family of long lines only when positions are largely periodic."""
|
||||||
|
h, w = binary.shape
|
||||||
|
length = max(25, int((w if horizontal else h) * .12))
|
||||||
|
kernel = cv2.getStructuringElement(cv2.MORPH_RECT,
|
||||||
|
(length, 1) if horizontal else (1, length))
|
||||||
|
connected = cv2.morphologyEx(binary, cv2.MORPH_CLOSE,
|
||||||
|
np.ones((1, 3) if horizontal else (3, 1), np.uint8))
|
||||||
|
lines = cv2.morphologyEx(connected, cv2.MORPH_OPEN, kernel)
|
||||||
|
counts = np.count_nonzero(lines, axis=1 if horizontal else 0)
|
||||||
|
bands = _runs(counts > (w if horizontal else h) * .18)
|
||||||
|
if len(bands) < 5:
|
||||||
|
return np.zeros_like(binary), 0
|
||||||
|
centers = np.array([(a+b)/2 for a, b in bands])
|
||||||
|
gaps = np.diff(centers)
|
||||||
|
# Missing lines and major/minor rulings may have integer-multiple spacing.
|
||||||
|
candidates = gaps[gaps >= 4]
|
||||||
|
regular = any(np.mean(abs(gaps / d - np.round(gaps / d)) < .16) >= .75
|
||||||
|
for d in candidates)
|
||||||
|
if not regular:
|
||||||
|
return np.zeros_like(binary), 0
|
||||||
|
accepted = np.zeros_like(binary)
|
||||||
|
for a, b in bands:
|
||||||
|
if horizontal:
|
||||||
|
accepted[max(0, a-2):b+2] = 255
|
||||||
|
else:
|
||||||
|
accepted[:, max(0, a-2):b+2] = 255
|
||||||
|
# Real scans have local warp as well as global skew. Recover shorter line
|
||||||
|
# segments close to the established ruling family, without extending the
|
||||||
|
# entire family into large empty gaps.
|
||||||
|
short = max(25, int((w if horizontal else h)*.035))
|
||||||
|
joined = cv2.morphologyEx(binary, cv2.MORPH_CLOSE,
|
||||||
|
np.ones((1, 7) if horizontal else (7, 1), np.uint8))
|
||||||
|
tolerant = cv2.dilate(joined, np.ones((3, 1) if horizontal else (1, 3), np.uint8))
|
||||||
|
fragments = cv2.morphologyEx(tolerant, cv2.MORPH_OPEN,
|
||||||
|
np.ones((1, short) if horizontal else (short, 1), np.uint8))
|
||||||
|
nearby = cv2.dilate(accepted, np.ones((15, 1) if horizontal else (1, 15), np.uint8))
|
||||||
|
accepted |= fragments & nearby if horizontal else fragments
|
||||||
|
return accepted, len(bands)
|
||||||
|
|
||||||
|
|
||||||
|
def _foreground(gray: np.ndarray, dpi: float, threshold: int,
|
||||||
|
ruling: np.ndarray) -> np.ndarray:
|
||||||
|
block = max(15, int(dpi / 5) | 1)
|
||||||
|
binary = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
|
||||||
|
cv2.THRESH_BINARY_INV, block, threshold)
|
||||||
|
# Only suppress near the fitted paper lines. Preserve unusually dark ink
|
||||||
|
# crossing pale ruling by comparing against the typical line intensity.
|
||||||
|
mask = cv2.dilate(ruling, np.ones((3, 3), np.uint8)) > 0
|
||||||
|
if np.any(ruling):
|
||||||
|
samples = ((ruling > 0) & (binary > 0)).astype(np.float32)
|
||||||
|
window = max(31, int(dpi*.4) | 1)
|
||||||
|
weight = cv2.boxFilter(samples, -1, (window, window))
|
||||||
|
total = cv2.boxFilter(gray.astype(np.float32)*samples, -1, (window, window))
|
||||||
|
typical = total / np.maximum(weight, 1e-6)
|
||||||
|
excess_ink = (gray.astype(float) < typical - 40).astype(np.uint8)
|
||||||
|
# A dark, one-pixel remnant of a paper line is still paper. Only retain
|
||||||
|
# locally thicker excess strokes inside the suppression mask.
|
||||||
|
excess_ink = cv2.erode(excess_ink, np.ones((2, 2), np.uint8)) > 0
|
||||||
|
mask &= ~excess_ink
|
||||||
|
binary[mask] = 0
|
||||||
|
return binary
|
||||||
|
|
||||||
|
|
||||||
|
def _content_mask(binary: np.ndarray, dpi: float, nh: int = 0,
|
||||||
|
nv: int = 0) -> tuple[np.ndarray, int]:
|
||||||
|
"""Filter only tiny speckles and repeated, matching edge-hole components."""
|
||||||
|
px = dpi / 25.4
|
||||||
|
# Small closing reconnects strokes interrupted by paper-line suppression.
|
||||||
|
grouped = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, np.ones((3, 3), np.uint8))
|
||||||
|
n, labels, stats, _ = cv2.connectedComponentsWithStats(grouped, 8)
|
||||||
|
keep = np.zeros(n, dtype=bool)
|
||||||
|
x, y, w, h, area = stats[1:].T
|
||||||
|
keep[1:] = (area >= max(4, .035*px*px)) & (np.maximum(w, h) >= .35*px)
|
||||||
|
# Thin straight residuals on confirmed ruled paper are not credible ink.
|
||||||
|
thin = max(2, round(.3*px))
|
||||||
|
if nv:
|
||||||
|
keep[1:] &= ~((w <= thin) & (h >= 3*w))
|
||||||
|
if nh:
|
||||||
|
keep[1:] &= ~((h <= thin) & (w >= 3*h))
|
||||||
|
# Hole shadows form recurring shapes close to a physical side edge. Never
|
||||||
|
# discard an entire margin strip: other writing there must remain visible.
|
||||||
|
height, width = binary.shape
|
||||||
|
candidates = np.flatnonzero(keep[1:]
|
||||||
|
& ((x+w < 12*px) | (x > width-12*px))
|
||||||
|
& (w > .8*px) & (w < 9*px) & (h > px) & (h < 12*px)) + 1
|
||||||
|
holes = set()
|
||||||
|
for i in candidates:
|
||||||
|
x, y, w, h, area = stats[i]
|
||||||
|
similar = []
|
||||||
|
a = cv2.resize((labels[y:y+h, x:x+w] == i).astype(np.uint8), (24, 32)) > 0
|
||||||
|
for j in candidates:
|
||||||
|
xx, yy, ww, hh, aa = stats[j]
|
||||||
|
if abs(x-xx) > 2*px or not (.7 < ww/w < 1.4 and .7 < hh/h < 1.4):
|
||||||
|
continue
|
||||||
|
b = cv2.resize((labels[yy:yy+hh, xx:xx+ww] == j).astype(np.uint8), (24, 32)) > 0
|
||||||
|
if np.count_nonzero(a & b) / max(1, np.count_nonzero(a | b)) > .60:
|
||||||
|
similar.append(j)
|
||||||
|
if len(similar) >= 4 and np.ptp(stats[similar, 1]) > height*.45:
|
||||||
|
holes.update(similar)
|
||||||
|
if holes:
|
||||||
|
keep[list(holes)] = False
|
||||||
|
# Bound original residual ink, not the expanded/grouped mask.
|
||||||
|
return (keep[labels] & (binary > 0)).astype(np.uint8), len(holes)
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
"""Split a PDF along its cumulative visible page height, in reading order."""
|
||||||
|
|
||||||
|
import math
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.commands.splitting_int import _prepare_split_pages
|
||||||
|
|
||||||
|
# Percentages emitted by the GUI have six decimal places. Recover exact page
|
||||||
|
# boundaries despite rounding, without turning nearby in-page cuts into breaks.
|
||||||
|
BOUNDARY_TOLERANCE_PERCENT = 0.000001
|
||||||
|
|
||||||
|
|
||||||
|
def cut_position(heights: list[float], percent: float) -> tuple[int, float]:
|
||||||
|
"""Return (page index, offset); offset zero denotes an exact page break."""
|
||||||
|
if not heights or any(height <= 0 for height in heights):
|
||||||
|
raise ValueError("Le PDF doit contenir des pages non vides.")
|
||||||
|
if not math.isfinite(percent) or not 0 < percent < 100:
|
||||||
|
raise ValueError("Le pourcentage de coupe doit être strictement entre 0 et 100.")
|
||||||
|
total = sum(heights)
|
||||||
|
position = total * percent / 100
|
||||||
|
start = 0.0
|
||||||
|
for index, height in enumerate(heights):
|
||||||
|
if index and abs(percent - start / total * 100) <= BOUNDARY_TOLERANCE_PERCENT:
|
||||||
|
return index, 0.0
|
||||||
|
end = start + height
|
||||||
|
if index + 1 < len(heights) and abs(percent - end / total * 100) <= BOUNDARY_TOLERANCE_PERCENT:
|
||||||
|
return index + 1, 0.0
|
||||||
|
if position < end:
|
||||||
|
return index, position - start
|
||||||
|
start = end
|
||||||
|
raise ValueError("Coupe hors du document.")
|
||||||
|
|
||||||
|
|
||||||
|
def split_pdf(source: Path, percent: float, first_path: Path, second_path: Path) -> None:
|
||||||
|
"""Preserve whole pages; clip only the page actually crossed by the cut."""
|
||||||
|
with pymupdf.open(source) as document, pymupdf.open() as first, pymupdf.open() as second:
|
||||||
|
index, offset = cut_position([page.rect.height for page in document], percent)
|
||||||
|
if index:
|
||||||
|
first.insert_pdf(document, from_page=0, to_page=index - 1)
|
||||||
|
if offset == 0:
|
||||||
|
second.insert_pdf(document, from_page=index)
|
||||||
|
else:
|
||||||
|
with pymupdf.open() as page_document:
|
||||||
|
page_document.insert_pdf(document, from_page=index, to_page=index)
|
||||||
|
visible = _prepare_split_pages(page_document)[0]
|
||||||
|
for target, clip in (
|
||||||
|
(first, pymupdf.Rect(visible.x0, visible.y0, visible.x1, visible.y0 + offset)),
|
||||||
|
(second, pymupdf.Rect(visible.x0, visible.y0 + offset, visible.x1, visible.y1)),
|
||||||
|
):
|
||||||
|
page = target.new_page(width=clip.width, height=clip.height)
|
||||||
|
page.show_pdf_page(page.rect, page_document, 0, clip=clip)
|
||||||
|
if index + 1 < len(document):
|
||||||
|
second.insert_pdf(document, from_page=index + 1)
|
||||||
|
first.save(first_path)
|
||||||
|
second.save(second_path)
|
||||||
+23
-4
@@ -2,6 +2,11 @@ from pathlib import Path
|
|||||||
import io
|
import io
|
||||||
from . import utils
|
from . import utils
|
||||||
|
|
||||||
|
PERSPECTIVE_GUIDANCE = (
|
||||||
|
"Ce barème est indicatif, si une réponse est entièrement correcte mais utilise "
|
||||||
|
"une méthode différente, elle mérite quand même tous les points."
|
||||||
|
)
|
||||||
|
|
||||||
main_prompt = """Je te fournis une image contenant plusieurs réponses manuscrites à un examen.
|
main_prompt = """Je te fournis une image contenant plusieurs réponses manuscrites à un examen.
|
||||||
|
|
||||||
Chaque réponse est séparée de la précédente par une ligne horizontale noire.
|
Chaque réponse est séparée de la précédente par une ligne horizontale noire.
|
||||||
@@ -103,7 +108,13 @@ def make_prompt(input_dir,full_label):
|
|||||||
# print("Debug : l/t/c/p", full_label, text, corr, persp)
|
# print("Debug : l/t/c/p", full_label, text, corr, persp)
|
||||||
|
|
||||||
if persp:
|
if persp:
|
||||||
persp = "\n\nVoici des consignes de notation complémentaires : \n\n```\n" + persp +"\n```\n"
|
persp = (
|
||||||
|
"\n\nVoici des consignes de notation complémentaires :\n\n"
|
||||||
|
+ PERSPECTIVE_GUIDANCE
|
||||||
|
+ "\n\n```\n"
|
||||||
|
+ persp
|
||||||
|
+ "\n```\n"
|
||||||
|
)
|
||||||
return main_prompt.replace("<<text>>", text).replace("<<corr>>", corr).replace("<<persp>>", persp).replace("<<label>>", full_label)
|
return main_prompt.replace("<<text>>", text).replace("<<corr>>", corr).replace("<<persp>>", persp).replace("<<label>>", full_label)
|
||||||
|
|
||||||
|
|
||||||
@@ -182,6 +193,7 @@ def generate_request(input_dir, file, full_label):
|
|||||||
]
|
]
|
||||||
|
|
||||||
generate_content_config = types.GenerateContentConfig(
|
generate_content_config = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=1.0,
|
temperature=1.0,
|
||||||
top_p=0.95,
|
top_p=0.95,
|
||||||
seed=0,
|
seed=0,
|
||||||
@@ -259,6 +271,7 @@ Commentaires d'origine :
|
|||||||
]
|
]
|
||||||
|
|
||||||
config = types.GenerateContentConfig(
|
config = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
temperature=1.0,
|
temperature=1.0,
|
||||||
response_mime_type="application/json",
|
response_mime_type="application/json",
|
||||||
response_json_schema=TypeAdapter(List[FeedbackItem]).json_schema()
|
response_json_schema=TypeAdapter(List[FeedbackItem]).json_schema()
|
||||||
@@ -288,7 +301,10 @@ Voici les labels possibles. Ta réponse doit être l'un d'entre eux :
|
|||||||
contents = [types.Content(role="user", parts=[
|
contents = [types.Content(role="user", parts=[
|
||||||
types.Part.from_bytes(data=get_single_image_bytes(pdf_path), mime_type="image/jpeg"),
|
types.Part.from_bytes(data=get_single_image_bytes(pdf_path), mime_type="image/jpeg"),
|
||||||
types.Part.from_text(text=prompt)])]
|
types.Part.from_text(text=prompt)])]
|
||||||
config = types.GenerateContentConfig(temperature=1.0)
|
config = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
|
temperature=1.0,
|
||||||
|
)
|
||||||
return contents, config
|
return contents, config
|
||||||
|
|
||||||
def request_for_additional_answer(pdf_path, label, enonce, labels_txt):
|
def request_for_additional_answer(pdf_path, label, enonce, labels_txt):
|
||||||
@@ -316,6 +332,9 @@ d'entre eux :
|
|||||||
types.Part.from_bytes(data=get_single_image_bytes(pdf_path), mime_type="image/jpeg"),
|
types.Part.from_bytes(data=get_single_image_bytes(pdf_path), mime_type="image/jpeg"),
|
||||||
types.Part.from_text(text=prompt)
|
types.Part.from_text(text=prompt)
|
||||||
])]
|
])]
|
||||||
config = types.GenerateContentConfig(temperature=1.0, response_mime_type="application/json")
|
config = types.GenerateContentConfig(
|
||||||
|
automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True),
|
||||||
|
temperature=1.0,
|
||||||
|
response_mime_type="application/json",
|
||||||
|
)
|
||||||
return contents, config
|
return contents, config
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,125 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from copienator import atomic_write_json, configuration, read_json, utils
|
||||||
|
from copienator.filesystem import staged_directory
|
||||||
|
from copienator.platform import safe_filename
|
||||||
|
|
||||||
|
RETURN_ANSWER_OPTIONS_FILE = Path(".copienator") / "return_answers.json"
|
||||||
|
|
||||||
|
|
||||||
|
def configured_return_answer_options() -> dict[str, bool]:
|
||||||
|
return {
|
||||||
|
"context": bool(configuration.RETURN_ANSWERS_CONTEXT),
|
||||||
|
"question": bool(configuration.RETURN_ANSWERS_QUESTION),
|
||||||
|
"solution": bool(configuration.RETURN_ANSWERS_SOLUTION),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def save_return_answer_options(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
context: bool,
|
||||||
|
question: bool,
|
||||||
|
solution: bool,
|
||||||
|
) -> None:
|
||||||
|
path = Path(root) / RETURN_ANSWER_OPTIONS_FILE
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
atomic_write_json(
|
||||||
|
path,
|
||||||
|
{"context": context, "question": question, "solution": solution},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_return_answer_options(root: Path) -> dict[str, bool]:
|
||||||
|
options = configured_return_answer_options()
|
||||||
|
path = Path(root) / RETURN_ANSWER_OPTIONS_FILE
|
||||||
|
if not path.is_file():
|
||||||
|
return options
|
||||||
|
loaded = read_json(path)
|
||||||
|
if not isinstance(loaded, dict):
|
||||||
|
raise ValueError(f"Expected a return-answer options object in {path}")
|
||||||
|
for name in options:
|
||||||
|
if name in loaded:
|
||||||
|
if type(loaded[name]) is not bool:
|
||||||
|
raise ValueError(f"Expected a boolean for {name!r} in {path}")
|
||||||
|
options[name] = loaded[name]
|
||||||
|
return options
|
||||||
|
|
||||||
|
|
||||||
|
def publish_answer_returns(
|
||||||
|
root: Path,
|
||||||
|
source: Path,
|
||||||
|
destination: Path,
|
||||||
|
*,
|
||||||
|
answers_only: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""Publish reviewed answers, optionally without touching return metadata."""
|
||||||
|
scores = read_json(source / "score.json")
|
||||||
|
if not isinstance(scores, dict):
|
||||||
|
raise ValueError(f"Expected a score object in {source}")
|
||||||
|
info_path = source / "info.json"
|
||||||
|
if not info_path.is_file():
|
||||||
|
raise ValueError(f"Missing {info_path}; recompile annotations before giving-names")
|
||||||
|
info = read_json(info_path)
|
||||||
|
if not isinstance(info, dict) or set(info) != set(scores):
|
||||||
|
raise ValueError(f"Invalid question information in {info_path}; recompile annotations")
|
||||||
|
for label, entry in info.items():
|
||||||
|
if (
|
||||||
|
not isinstance(entry, dict)
|
||||||
|
or set(entry) != {"present", "not_empty", "touched", "score"}
|
||||||
|
or any(type(entry[key]) is not bool for key in ("present", "not_empty", "touched"))
|
||||||
|
or (entry["not_empty"] and not entry["present"])
|
||||||
|
or (entry["touched"] and not entry["not_empty"])
|
||||||
|
):
|
||||||
|
raise ValueError(f"Invalid question information in {info_path}; recompile annotations")
|
||||||
|
# Match score.json, including manual score edits awaiting recompilation.
|
||||||
|
entry["score"] = scores[label]
|
||||||
|
|
||||||
|
answers_dir = destination / "answers"
|
||||||
|
if answers_dir.is_symlink():
|
||||||
|
raise ValueError(f"Expected a real answer directory: {answers_dir}")
|
||||||
|
if configuration.RETURN_ANSWERS_ENABLED:
|
||||||
|
options = load_return_answer_options(root)
|
||||||
|
labels = [label for label, entry in info.items() if entry["present"] and entry["not_empty"]]
|
||||||
|
|
||||||
|
# Import the rendering backend only when individual images are requested.
|
||||||
|
from copienator.commands.annotating import make_base_image
|
||||||
|
from copienator.commands.reading_annotations import concatenate
|
||||||
|
|
||||||
|
all_labels = utils.read_all_labels(root)
|
||||||
|
with staged_directory(answers_dir) as staging:
|
||||||
|
for label in sorted(labels, key=utils.natural_key):
|
||||||
|
paths = []
|
||||||
|
if options["context"]:
|
||||||
|
paths.extend(utils.pdf_images_of_contexts(root, label, all_labels))
|
||||||
|
if options["question"]:
|
||||||
|
paths.append(utils.pdf_image_of_enonce(root, label))
|
||||||
|
if options["solution"]:
|
||||||
|
paths.append(utils.pdf_image_of_solution(root, label))
|
||||||
|
images = []
|
||||||
|
for path in paths:
|
||||||
|
if path:
|
||||||
|
supplement, _, _ = make_base_image(path)
|
||||||
|
if supplement is None:
|
||||||
|
raise ValueError(f"Could not render {path}")
|
||||||
|
images.append(supplement)
|
||||||
|
with Image.open(source / f"{label}.jpg") as answer:
|
||||||
|
images.append(answer.convert("RGB"))
|
||||||
|
image = concatenate(images)
|
||||||
|
output = staging / f"{safe_filename(label)}.jpg"
|
||||||
|
if output.exists():
|
||||||
|
raise ValueError(
|
||||||
|
f"Answer labels produce the same filename in {answers_dir}: {label}"
|
||||||
|
)
|
||||||
|
image.save(output)
|
||||||
|
elif answers_dir.exists():
|
||||||
|
# Replace the managed directory with an empty one to remove stale exports.
|
||||||
|
with staged_directory(answers_dir):
|
||||||
|
pass
|
||||||
|
if not answers_only:
|
||||||
|
atomic_write_json(destination / "info.json", info)
|
||||||
|
(destination / "touched.json").unlink(missing_ok=True)
|
||||||
+21
-6
@@ -160,23 +160,38 @@ def compile_to_pdf(text, output_pdf_path):
|
|||||||
# env['TEXINPUTS'] = f".:{current_dir}:"
|
# env['TEXINPUTS'] = f".:{current_dir}:"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
subprocess.run(
|
result = subprocess.run(
|
||||||
['pdflatex', '-interaction=nonstopmode', tex_filename],
|
['pdflatex', '-interaction=nonstopmode', tex_filename],
|
||||||
cwd=temp_dir,
|
cwd=temp_dir,
|
||||||
stdout=subprocess.DEVNULL,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.DEVNULL,
|
stderr=subprocess.STDOUT,
|
||||||
|
text=True,
|
||||||
check=False
|
check=False
|
||||||
)
|
)
|
||||||
if "minted" in text:
|
if "minted" in text:
|
||||||
subprocess.run(
|
result = subprocess.run(
|
||||||
['pdflatex', '-interaction=nonstopmode', tex_filename],
|
['pdflatex', '-interaction=nonstopmode', tex_filename],
|
||||||
cwd=temp_dir,
|
cwd=temp_dir,
|
||||||
stdout=subprocess.DEVNULL,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.DEVNULL,
|
stderr=subprocess.STDOUT,
|
||||||
|
text=True,
|
||||||
check=False)
|
check=False)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
error_lines = [
|
||||||
|
line.strip() for line in result.stdout.splitlines()
|
||||||
|
if line.lstrip().startswith("!")
|
||||||
|
]
|
||||||
|
detail = f": {error_lines[0]}" if error_lines else ""
|
||||||
|
print(
|
||||||
|
f"Warning: LaTeX compilation failed for {output_pdf_path} "
|
||||||
|
f"(exit code {result.returncode}){detail}"
|
||||||
|
)
|
||||||
|
|
||||||
generated_pdf = os.path.join(temp_dir, pdf_filename)
|
generated_pdf = os.path.join(temp_dir, pdf_filename)
|
||||||
if os.path.exists(generated_pdf):
|
if os.path.exists(generated_pdf):
|
||||||
shutil.move(generated_pdf, output_pdf_path)
|
shutil.move(generated_pdf, output_pdf_path)
|
||||||
|
else:
|
||||||
|
print(f"Warning: LaTeX compilation produced no PDF for {output_pdf_path}")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"Compilation error for {output_pdf_path}: {e}")
|
print(f"Compilation error for {output_pdf_path}: {e}")
|
||||||
|
|||||||
@@ -109,6 +109,10 @@ class EvaluationWorkspace:
|
|||||||
def correction_progress_file(self) -> Path:
|
def correction_progress_file(self) -> Path:
|
||||||
return self.root / "correction_progress.json"
|
return self.root / "correction_progress.json"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def correction_pending_responses_file(self) -> Path:
|
||||||
|
return self.root / "correction_pending_responses.json"
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def batch_jobs_file(self) -> Path:
|
def batch_jobs_file(self) -> Path:
|
||||||
return self.root / "batch_jobs.json"
|
return self.root / "batch_jobs.json"
|
||||||
|
|||||||
+649
-47
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,81 @@
|
|||||||
|
"""Non-blocking, cancellable batch checks while the desktop GUI is open."""
|
||||||
|
|
||||||
|
import queue
|
||||||
|
|
||||||
|
from .runner import ProcessRunner
|
||||||
|
|
||||||
|
CHECK_INTERVAL_MS = 5 * 60 * 1000
|
||||||
|
|
||||||
|
|
||||||
|
class BatchMonitor:
|
||||||
|
def __init__(self, scheduler, on_ready, on_status, on_output):
|
||||||
|
self.scheduler = scheduler
|
||||||
|
self.on_ready = on_ready
|
||||||
|
self.on_status = on_status
|
||||||
|
self.on_output = on_output
|
||||||
|
self.active = False
|
||||||
|
self.timer = None
|
||||||
|
self.runner = None
|
||||||
|
|
||||||
|
def start(self, command, cwd, environment, log_path):
|
||||||
|
if self.active:
|
||||||
|
return
|
||||||
|
self.arguments = (command, cwd, environment, log_path)
|
||||||
|
self.active = True
|
||||||
|
self._check()
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
self.active = False
|
||||||
|
if self.timer is not None:
|
||||||
|
self.scheduler.after_cancel(self.timer)
|
||||||
|
self.timer = None
|
||||||
|
if self.runner is not None:
|
||||||
|
try:
|
||||||
|
self.runner.force_stop()
|
||||||
|
except OSError as exc:
|
||||||
|
self.on_output(f"Arrêt de la vérification : {exc}\n")
|
||||||
|
self.runner = None
|
||||||
|
self.on_status("Vérification automatique arrêtée.")
|
||||||
|
|
||||||
|
def _later(self):
|
||||||
|
self.timer = self.scheduler.after(CHECK_INTERVAL_MS, self._check)
|
||||||
|
|
||||||
|
def _check(self):
|
||||||
|
self.timer = None
|
||||||
|
if not self.active:
|
||||||
|
return
|
||||||
|
self.runner = ProcessRunner()
|
||||||
|
self.on_status("Vérification des batchs en cours…")
|
||||||
|
try:
|
||||||
|
self.runner.start(*self.arguments)
|
||||||
|
except (OSError, RuntimeError) as exc:
|
||||||
|
self.on_output(f"Vérification impossible : {exc}\n")
|
||||||
|
self.runner = None
|
||||||
|
self.on_status("Échec de la vérification. Nouvel essai dans 5 minutes.")
|
||||||
|
self._later()
|
||||||
|
return
|
||||||
|
self.timer = self.scheduler.after(100, self._poll)
|
||||||
|
|
||||||
|
def _poll(self):
|
||||||
|
self.timer = None
|
||||||
|
if not self.active or self.runner is None:
|
||||||
|
return
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
event, payload = self.runner.events.get_nowait()
|
||||||
|
except queue.Empty:
|
||||||
|
break
|
||||||
|
if event in {"output", "runner_error"}:
|
||||||
|
self.on_output(str(payload))
|
||||||
|
elif event == "finished":
|
||||||
|
code, interrupted = payload
|
||||||
|
self.runner = None
|
||||||
|
if code == 0 and not interrupted:
|
||||||
|
self.active = False
|
||||||
|
self.on_status("Tous les résultats batch sont prêts.")
|
||||||
|
self.on_ready()
|
||||||
|
else:
|
||||||
|
self.on_status("Résultats pas encore prêts. Nouvelle vérification dans 5 minutes.")
|
||||||
|
self._later()
|
||||||
|
return
|
||||||
|
self.timer = self.scheduler.after(100, self._poll)
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import tkinter as tk
|
||||||
|
from pathlib import Path
|
||||||
|
from tkinter import messagebox, ttk
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
from PIL import Image, ImageTk
|
||||||
|
|
||||||
|
from copienator.pdf_cut import cut_position
|
||||||
|
|
||||||
|
PAGE_GAP = 28
|
||||||
|
SNAP_PIXELS = 12
|
||||||
|
|
||||||
|
|
||||||
|
def percentage_at_y(y: float, heights: list[float], scale: float) -> float:
|
||||||
|
"""Convert canvas y to document height, snapping across inter-page gaps."""
|
||||||
|
top = 0.0
|
||||||
|
cumulative = 0.0
|
||||||
|
total = sum(heights)
|
||||||
|
for index, height in enumerate(heights):
|
||||||
|
bottom = top + height * scale
|
||||||
|
if index + 1 < len(heights) and bottom - SNAP_PIXELS <= y <= bottom + PAGE_GAP + SNAP_PIXELS:
|
||||||
|
return (cumulative + height) / total * 100
|
||||||
|
if y <= bottom:
|
||||||
|
return max(0.0, min(100.0, (cumulative + (y - top) / scale) / total * 100))
|
||||||
|
cumulative += height
|
||||||
|
top = bottom + PAGE_GAP
|
||||||
|
return 100.0
|
||||||
|
|
||||||
|
|
||||||
|
def y_at_percentage(percent: float, heights: list[float], scale: float) -> float:
|
||||||
|
if percent <= 0:
|
||||||
|
return 0.0
|
||||||
|
if percent >= 100:
|
||||||
|
return sum(heights) * scale + (len(heights) - 1) * PAGE_GAP
|
||||||
|
index, offset = cut_position(heights, percent)
|
||||||
|
top = sum(heights[:index]) * scale + index * PAGE_GAP
|
||||||
|
return top - PAGE_GAP / 2 if index and offset == 0 else top + offset * scale
|
||||||
|
|
||||||
|
|
||||||
|
def cut_operator(percent: float, keep: int, mode: str) -> str:
|
||||||
|
value = f"{percent:.6f}".rstrip("0").rstrip(".")
|
||||||
|
return f"c{{{value}}}{keep}{mode}"
|
||||||
|
|
||||||
|
|
||||||
|
class CutHelper(tk.Toplevel):
|
||||||
|
def __init__(self, parent, path: Path, on_accept, initial=None):
|
||||||
|
# Load the source before creating a window so invalid PDFs leave no dialog.
|
||||||
|
with pymupdf.open(path) as document:
|
||||||
|
if not len(document):
|
||||||
|
raise ValueError("Le PDF est vide.")
|
||||||
|
self.heights = [page.rect.height for page in document]
|
||||||
|
width = min(850, parent.winfo_screenwidth() - 100)
|
||||||
|
self.scale = min(1.5, width / max(page.rect.width for page in document))
|
||||||
|
rendered = []
|
||||||
|
for page in document:
|
||||||
|
pix = page.get_pixmap(matrix=pymupdf.Matrix(self.scale, self.scale), alpha=False,
|
||||||
|
colorspace=pymupdf.csRGB)
|
||||||
|
rendered.append(Image.frombytes("RGB", (pix.width, pix.height), pix.samples))
|
||||||
|
super().__init__(parent)
|
||||||
|
self.title(f"Cut — {path.name}")
|
||||||
|
self.geometry(f"{width + 45}x{min(850, parent.winfo_screenheight() - 100)}")
|
||||||
|
self.transient(parent.winfo_toplevel())
|
||||||
|
self.on_accept = on_accept
|
||||||
|
self.percent = initial[0] if initial else 50.0
|
||||||
|
self.keep = tk.IntVar(value=initial[1] if initial else 1)
|
||||||
|
self.mode = tk.StringVar(value=initial[2] if initial else ">")
|
||||||
|
self.caption = tk.StringVar()
|
||||||
|
ttk.Label(self, text="Déplacez la barre rouge. Entrée : afficher la commande ; Échap : annuler.",
|
||||||
|
wraplength=width).pack(anchor="w", padx=8, pady=5)
|
||||||
|
controls = ttk.Frame(self)
|
||||||
|
controls.pack(fill="x", padx=8)
|
||||||
|
ttk.Label(controls, text="Conserver à la source :").pack(side="left")
|
||||||
|
ttk.Radiobutton(controls, text="1 — début", variable=self.keep, value=1).pack(side="left")
|
||||||
|
ttk.Radiobutton(controls, text="2 — fin", variable=self.keep, value=2).pack(side="left")
|
||||||
|
ttk.Radiobutton(controls, text="Ajouter à la cible", variable=self.mode, value=">").pack(side="left")
|
||||||
|
ttk.Radiobutton(controls, text="Remplacer", variable=self.mode, value="x").pack(side="left")
|
||||||
|
ttk.Label(self, textvariable=self.caption).pack(anchor="w", padx=8, pady=5)
|
||||||
|
viewport = ttk.Frame(self)
|
||||||
|
viewport.pack(fill="both", expand=True)
|
||||||
|
self.canvas = tk.Canvas(viewport, background="#555555", highlightthickness=0)
|
||||||
|
scrollbar = ttk.Scrollbar(viewport, command=self.canvas.yview)
|
||||||
|
self.canvas.configure(yscrollcommand=scrollbar.set)
|
||||||
|
scrollbar.pack(side="right", fill="y")
|
||||||
|
self.canvas.pack(fill="both", expand=True)
|
||||||
|
self.images = [ImageTk.PhotoImage(image, master=self) for image in rendered]
|
||||||
|
top = 0.0
|
||||||
|
self.width = width
|
||||||
|
for index, image in enumerate(self.images):
|
||||||
|
self.canvas.create_image(0, top, anchor="nw", image=image)
|
||||||
|
top += self.heights[index] * self.scale
|
||||||
|
if index + 1 < len(self.images):
|
||||||
|
top += PAGE_GAP
|
||||||
|
self.canvas.configure(scrollregion=(0, 0, width, top))
|
||||||
|
self.bar = self.canvas.create_line(0, 0, width, 0, fill="#ff3030", width=4)
|
||||||
|
self.canvas.bind("<Button-1>", self.move_bar)
|
||||||
|
self.canvas.bind("<B1-Motion>", self.move_bar)
|
||||||
|
self.canvas.bind("<Button-4>", lambda event: self.canvas.yview_scroll(-3, "units"))
|
||||||
|
self.canvas.bind("<Button-5>", lambda event: self.canvas.yview_scroll(3, "units"))
|
||||||
|
self.canvas.bind("<MouseWheel>", lambda event: self.canvas.yview_scroll(-1 if event.delta > 0 else 1, "units"))
|
||||||
|
self.bind("<Return>", self.accept)
|
||||||
|
self.bind("<Escape>", lambda event: self.destroy())
|
||||||
|
self.keep.trace_add("write", lambda *_: self.draw_bar())
|
||||||
|
self.mode.trace_add("write", lambda *_: self.draw_bar())
|
||||||
|
self.draw_bar()
|
||||||
|
self.canvas.yview_moveto(max(0, (y_at_percentage(self.percent, self.heights, self.scale) - 200) / top))
|
||||||
|
self.focus_set()
|
||||||
|
self.grab_set()
|
||||||
|
|
||||||
|
def move_bar(self, event):
|
||||||
|
self.percent = percentage_at_y(self.canvas.canvasy(event.y), self.heights, self.scale)
|
||||||
|
self.draw_bar()
|
||||||
|
|
||||||
|
def draw_bar(self):
|
||||||
|
y = y_at_percentage(self.percent, self.heights, self.scale)
|
||||||
|
self.canvas.coords(self.bar, 0, y, self.width, y)
|
||||||
|
text = cut_operator(self.percent, self.keep.get(), self.mode.get())
|
||||||
|
if 0 < self.percent < 100:
|
||||||
|
index, offset = cut_position(self.heights, self.percent)
|
||||||
|
text += f" — entre les pages {index} et {index + 1}" if offset == 0 else f" — page {index + 1}"
|
||||||
|
self.caption.set(text)
|
||||||
|
|
||||||
|
def accept(self, event=None):
|
||||||
|
operator = cut_operator(self.percent, self.keep.get(), self.mode.get())
|
||||||
|
rounded = float(operator.split("{")[1].split("}")[0])
|
||||||
|
if not 0 < rounded < 100:
|
||||||
|
messagebox.showerror("Coupe invalide", "Chaque partie doit contenir une portion du PDF.", parent=self)
|
||||||
|
return
|
||||||
|
self.destroy()
|
||||||
|
self.on_accept(operator)
|
||||||
@@ -51,7 +51,7 @@ def collect_diagnostics(
|
|||||||
"pdf2image": "pdf2image",
|
"pdf2image": "pdf2image",
|
||||||
"reportlab": "reportlab",
|
"reportlab": "reportlab",
|
||||||
"img2pdf": "img2pdf",
|
"img2pdf": "img2pdf",
|
||||||
"PyMuPDF": "fitz",
|
"PyMuPDF": "pymupdf",
|
||||||
"ftfy": "ftfy",
|
"ftfy": "ftfy",
|
||||||
"ezodf": "ezodf",
|
"ezodf": "ezodf",
|
||||||
"Google GenAI": "google.genai",
|
"Google GenAI": "google.genai",
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import tkinter as tk
|
||||||
|
from pathlib import Path
|
||||||
|
from tkinter import messagebox, ttk
|
||||||
|
|
||||||
|
from copienator import CliError
|
||||||
|
from copienator.commands.resolve_manual import get_actual_pdf, parse_instruction_text
|
||||||
|
from copienator.platform import open_path
|
||||||
|
from .cut_helper import CutHelper
|
||||||
|
|
||||||
|
|
||||||
|
HELP = """La correction propose x> pour un mauvais label et -> pour une réponse supplémentaire lorsque le PDF cible existe déjà. Vérifiez les deux PDF avant de choisir.
|
||||||
|
|
||||||
|
Format : Copie01 Label source OP Label cible|
|
||||||
|
Conservez Copie suivi du numéro et les labels exacts (sans .pdf). Les espaces dans les labels sont acceptés ; entourez l’opérateur d’espaces.
|
||||||
|
|
||||||
|
-> : fusionner dans la cible, conserver la source.
|
||||||
|
x> : fusionner dans la cible, archiver la source.
|
||||||
|
-x : remplacer la cible par une copie de la source, conserver la source.
|
||||||
|
xx : remplacer la cible par une copie de la source, archiver la source.
|
||||||
|
ss : conserver les deux PDF sans fusion.
|
||||||
|
sx : conserver la source, archiver la cible, sans fusion.
|
||||||
|
xs : archiver la source, conserver la cible, sans fusion.
|
||||||
|
|
||||||
|
c{43}1> : couper la source à 43 %, conserver le début et ajouter la fin à la cible.
|
||||||
|
c{43}2x : couper la source à 43 %, conserver la fin et remplacer la cible par le début.
|
||||||
|
1 conserve le début, 2 conserve la fin ; > fusionne l’autre partie avec la cible, x la remplace. Le pourcentage porte sur la hauteur cumulée des pages visibles. Les décimales sont acceptées. Une seule coupe par label source est autorisée.
|
||||||
|
Cut permet de placer la coupe, avec accrochage entre les pages. Entrée ferme l’aperçu et affiche la commande à recopier ci-dessus : aucun fichier n’est modifié. Une séparation entre pages conserve les pages entières. La source complète est archivée en _old ; les deux labels modifiés sont générés en _new et ajoutés à refaire.json.
|
||||||
|
|
||||||
|
Pour les fusions : « Source x> Cible| » place la cible avant la source ; « Source x> |Cible » place la source avant la cible. Même règle avec -> et c{…}1>/c{…}2> (pour la partie transférée) ; sans |, la cible vient en premier.
|
||||||
|
Vous pouvez changer l’opérateur, déplacer |, corriger les labels ou retirer une instruction. Les lignes vides et celles commençant par ### sont ignorées. Retirer/commenter une ligne ne résout pas son conflit.
|
||||||
|
|
||||||
|
Enregistrez dans l’éditeur, puis cliquez sur Recharger et enfin sur Exécuter. Seul le fichier enregistré est appliqué. L’archivage utilise le suffixe _old ; les PDF créés utilisent _new. Après succès, manual_resolutions.txt est supprimé et correction.json est mis à jour. Si des PDF sont créés, refaire.json est généré : relancez la correction avec --refaire."""
|
||||||
|
|
||||||
|
|
||||||
|
class ManualResolutionPanel(ttk.Frame):
|
||||||
|
def __init__(self, parent, get_evaluation):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.get_evaluation = get_evaluation
|
||||||
|
self.pdf_buttons = []
|
||||||
|
self.cut_buttons = []
|
||||||
|
actions = ttk.Frame(self)
|
||||||
|
actions.pack(fill="x")
|
||||||
|
self.editor_button = ttk.Button(
|
||||||
|
actions, text="Ouvrir dans un éditeur de texte", command=self.open_editor
|
||||||
|
)
|
||||||
|
self.editor_button.pack(side="left")
|
||||||
|
ttk.Button(actions, text="Recharger", command=self.reload).pack(side="left", padx=6)
|
||||||
|
self.cut_result = tk.StringVar()
|
||||||
|
result = ttk.Frame(self)
|
||||||
|
result.pack(fill="x", pady=4)
|
||||||
|
ttk.Entry(result, textvariable=self.cut_result, state="readonly").pack(side="left", fill="x", expand=True)
|
||||||
|
ttk.Button(result, text="Copier la commande Cut", command=self.copy_cut_command).pack(side="left", padx=6)
|
||||||
|
self.status = ttk.Label(self, wraplength=650)
|
||||||
|
self.status.pack(fill="x", pady=4)
|
||||||
|
preview = ttk.Frame(self)
|
||||||
|
preview.pack(fill="both", expand=True)
|
||||||
|
self.text = tk.Text(preview, height=10, width=50, wrap="word", state="disabled")
|
||||||
|
scroll = ttk.Scrollbar(preview, command=self.text.yview)
|
||||||
|
self.text.configure(yscrollcommand=scroll.set)
|
||||||
|
scroll.pack(side="right", fill="y")
|
||||||
|
self.text.pack(fill="both", expand=True)
|
||||||
|
ttk.Label(self, text=HELP, wraplength=650, justify="left").pack(fill="x", pady=8)
|
||||||
|
self.reload()
|
||||||
|
|
||||||
|
def manual_path(self) -> Path | None:
|
||||||
|
evaluation = self.get_evaluation()
|
||||||
|
return evaluation / "manual_resolutions.txt" if evaluation else None
|
||||||
|
|
||||||
|
def open_editor(self):
|
||||||
|
path = self.manual_path()
|
||||||
|
if path is None or not path.is_file():
|
||||||
|
self.reload()
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
# .txt is opened in the desktop's associated text editor.
|
||||||
|
open_path(path)
|
||||||
|
except (OSError, RuntimeError) as exc:
|
||||||
|
messagebox.showerror("Ouverture impossible", str(exc))
|
||||||
|
|
||||||
|
def open_pdf(self, evaluation, copy_id, label):
|
||||||
|
path = get_actual_pdf(evaluation / "Copies", copy_id, label)
|
||||||
|
try:
|
||||||
|
if not path.is_file():
|
||||||
|
raise FileNotFoundError(f"PDF introuvable : {path}")
|
||||||
|
open_path(path)
|
||||||
|
except (OSError, RuntimeError) as exc:
|
||||||
|
messagebox.showerror("Ouverture du PDF", str(exc))
|
||||||
|
|
||||||
|
def reload(self):
|
||||||
|
for button in self.pdf_buttons + self.cut_buttons:
|
||||||
|
button.destroy()
|
||||||
|
self.pdf_buttons.clear()
|
||||||
|
self.cut_buttons.clear()
|
||||||
|
self.cut_result.set("")
|
||||||
|
self.text.configure(state="normal")
|
||||||
|
self.text.delete("1.0", "end")
|
||||||
|
path = self.manual_path()
|
||||||
|
self.editor_button.configure(state="disabled")
|
||||||
|
try:
|
||||||
|
if path is None:
|
||||||
|
raise FileNotFoundError("Chargez une évaluation.")
|
||||||
|
content = path.read_text(encoding="utf-8")
|
||||||
|
except (OSError, UnicodeError) as exc:
|
||||||
|
self.status.configure(text=f"manual_resolutions.txt indisponible : {exc}")
|
||||||
|
else:
|
||||||
|
self.editor_button.configure(state="normal")
|
||||||
|
malformed = []
|
||||||
|
for number, raw in enumerate(content.splitlines(keepends=True), 1):
|
||||||
|
self.text.insert("end", raw.rstrip("\r\n"))
|
||||||
|
try:
|
||||||
|
instructions = parse_instruction_text(raw)
|
||||||
|
except CliError:
|
||||||
|
malformed.append(str(number))
|
||||||
|
else:
|
||||||
|
if instructions:
|
||||||
|
instruction = instructions[0]
|
||||||
|
for title, label in (("PDF source", instruction.old_label),
|
||||||
|
("PDF cible", instruction.new_label)):
|
||||||
|
button = ttk.Button(
|
||||||
|
self.text, text=title,
|
||||||
|
command=lambda label=label, copy_id=instruction.copy_id, root=path.parent: self.open_pdf(root, copy_id, label),
|
||||||
|
)
|
||||||
|
self.pdf_buttons.append(button)
|
||||||
|
self.text.window_create("end", window=button, padx=8)
|
||||||
|
button = ttk.Button(self.text, text="Cut",
|
||||||
|
command=lambda item=instruction, root=path.parent: self.open_cut(root, item))
|
||||||
|
self.cut_buttons.append(button)
|
||||||
|
self.text.window_create("end", window=button, padx=8)
|
||||||
|
if raw.endswith("\n"):
|
||||||
|
self.text.insert("end", "\n")
|
||||||
|
detail = (" — lignes invalides : " + ", ".join(malformed)) if malformed else (
|
||||||
|
f" — {len(self.pdf_buttons) // 2} instruction(s)"
|
||||||
|
)
|
||||||
|
self.status.configure(text=str(path) + detail)
|
||||||
|
finally:
|
||||||
|
self.text.configure(state="disabled")
|
||||||
|
|
||||||
|
def copy_cut_command(self):
|
||||||
|
if self.cut_result.get():
|
||||||
|
self.clipboard_clear()
|
||||||
|
self.clipboard_append(self.cut_result.get())
|
||||||
|
|
||||||
|
def open_cut(self, evaluation, instruction):
|
||||||
|
source = get_actual_pdf(evaluation / "Copies", instruction.copy_id, instruction.old_label)
|
||||||
|
def accepted(operator):
|
||||||
|
target = ("|" + instruction.new_label) if instruction.pipe_first else (instruction.new_label + "|")
|
||||||
|
self.cut_result.set(f"Copie{instruction.copy_id} {instruction.old_label} {operator} {target}")
|
||||||
|
try:
|
||||||
|
initial = (*instruction.cut, instruction.operator[-1]) if instruction.cut else None
|
||||||
|
CutHelper(self, source, accepted, initial)
|
||||||
|
except (OSError, RuntimeError, ValueError) as exc:
|
||||||
|
messagebox.showerror("Ouverture du PDF", str(exc))
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
"""Launch native desktop notifications without blocking Tk."""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from copienator.platform import find_executable
|
||||||
|
|
||||||
|
|
||||||
|
def notify_desktop(title: str, message: str) -> None:
|
||||||
|
if sys.platform.startswith("linux"):
|
||||||
|
executable = find_executable("notify-send")
|
||||||
|
if not executable:
|
||||||
|
raise RuntimeError("Installez notify-send (libnotify) pour les notifications de bureau.")
|
||||||
|
command = [executable, "--app-name=Copienator", "--", title, message]
|
||||||
|
elif sys.platform == "darwin":
|
||||||
|
command = ["/usr/bin/osascript", "-e",
|
||||||
|
f"display notification {json.dumps(message, ensure_ascii=False)} with title {json.dumps(title, ensure_ascii=False)}"]
|
||||||
|
elif os.name == "nt":
|
||||||
|
# Encode the script and quote strings as literals; no shell interpolation.
|
||||||
|
quote = lambda value: "'" + value.replace("'", "''") + "'"
|
||||||
|
script = (
|
||||||
|
"Add-Type -AssemblyName System.Windows.Forms;"
|
||||||
|
"$notice = New-Object System.Windows.Forms.NotifyIcon;"
|
||||||
|
"$notice.Icon = [System.Drawing.SystemIcons]::Information;"
|
||||||
|
"$notice.Visible = $true;"
|
||||||
|
f"$notice.ShowBalloonTip(10000, {quote(title)}, {quote(message)}, "
|
||||||
|
"[System.Windows.Forms.ToolTipIcon]::Info);"
|
||||||
|
"Start-Sleep -Seconds 12; $notice.Dispose()"
|
||||||
|
)
|
||||||
|
command = ["powershell.exe", "-NoProfile", "-NonInteractive", "-WindowStyle", "Hidden",
|
||||||
|
"-EncodedCommand", base64.b64encode(script.encode("utf-16-le")).decode("ascii")]
|
||||||
|
else:
|
||||||
|
raise RuntimeError("Notifications de bureau indisponibles sur ce système.")
|
||||||
|
subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||||
+336
-74
@@ -7,6 +7,9 @@ from collections.abc import Iterable
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from copienator import configuration
|
||||||
|
from copienator.configuration import ALWAYS_CROP
|
||||||
|
|
||||||
EVALUATION = "${evaluation}"
|
EVALUATION = "${evaluation}"
|
||||||
|
|
||||||
|
|
||||||
@@ -16,6 +19,7 @@ class ArgumentSpec:
|
|||||||
label: str
|
label: str
|
||||||
kind: str = "text" # text, int, bool, choice, path
|
kind: str = "text" # text, int, bool, choice, path
|
||||||
flag: str | None = None
|
flag: str | None = None
|
||||||
|
false_flag: str | None = None
|
||||||
default: object = ""
|
default: object = ""
|
||||||
choices: tuple[str, ...] = ()
|
choices: tuple[str, ...] = ()
|
||||||
help: str = ""
|
help: str = ""
|
||||||
@@ -33,6 +37,7 @@ class CommandVariant:
|
|||||||
fixed_args_before_positionals: bool = False
|
fixed_args_before_positionals: bool = False
|
||||||
dangerous: bool = False
|
dangerous: bool = False
|
||||||
danger_warning: str | None = None
|
danger_warning: str | None = None
|
||||||
|
supports_verbose: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -50,31 +55,72 @@ class StepDefinition:
|
|||||||
auto_start_first_visit: bool = False
|
auto_start_first_visit: bool = False
|
||||||
skip_for_live_correction: bool = False
|
skip_for_live_correction: bool = False
|
||||||
skip_without_manual_conflicts: bool = False
|
skip_without_manual_conflicts: bool = False
|
||||||
|
extra_arguments_help: str = ""
|
||||||
|
extra_arguments_variants: tuple[str, ...] = ()
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_manual(self) -> bool:
|
def is_manual(self) -> bool:
|
||||||
return all(variant.kind == "manual" for variant in self.variants)
|
return all(variant.kind == "manual" for variant in self.variants)
|
||||||
|
|
||||||
|
|
||||||
def arg_target(help_text: str = "Dossier d’évaluation ou fichier à traiter") -> ArgumentSpec:
|
def arg_target(help_text: str = "un dossier d’évaluation ou un fichier à traiter") -> ArgumentSpec:
|
||||||
return ArgumentSpec(
|
return ArgumentSpec(
|
||||||
"target",
|
"target",
|
||||||
"Cible",
|
"Cible",
|
||||||
kind="path",
|
kind="path",
|
||||||
default=EVALUATION,
|
default=EVALUATION,
|
||||||
positional=True,
|
positional=True,
|
||||||
help=help_text,
|
help=f"La cible peut être {help_text.rstrip('.')}.",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
||||||
python = lambda ident, label, script, **kwargs: CommandVariant(
|
def python(ident, label, script, **kwargs):
|
||||||
ident, label, script, "python", **kwargs
|
supports_verbose = kwargs.pop("supports_verbose", True)
|
||||||
)
|
return CommandVariant(
|
||||||
|
ident,
|
||||||
|
label,
|
||||||
|
script,
|
||||||
|
"python",
|
||||||
|
supports_verbose=supports_verbose,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
manual = lambda ident, label="Étape manuelle": CommandVariant(
|
manual = lambda ident, label="Étape manuelle": CommandVariant(
|
||||||
ident, label, None, "manual"
|
ident, label, None, "manual"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
return_answer_arguments = ()
|
||||||
|
if configuration.RETURN_ANSWERS_ENABLED:
|
||||||
|
return_answer_arguments = (
|
||||||
|
ArgumentSpec(
|
||||||
|
"return_answers_context",
|
||||||
|
"Inclure le contexte dans answers",
|
||||||
|
"bool",
|
||||||
|
"--return-answers-context",
|
||||||
|
"--no-return-answers-context",
|
||||||
|
default=configuration.RETURN_ANSWERS_CONTEXT,
|
||||||
|
help="Inclut les pages de contexte applicables avant chaque réponse individuelle publiée dans answers.",
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"return_answers_question",
|
||||||
|
"Inclure l’énoncé dans answers",
|
||||||
|
"bool",
|
||||||
|
"--return-answers-question",
|
||||||
|
"--no-return-answers-question",
|
||||||
|
default=configuration.RETURN_ANSWERS_QUESTION,
|
||||||
|
help="Inclut l’énoncé actuel avant chaque réponse individuelle publiée dans answers.",
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"return_answers_solution",
|
||||||
|
"Inclure la correction dans answers",
|
||||||
|
"bool",
|
||||||
|
"--return-answers-solution",
|
||||||
|
"--no-return-answers-solution",
|
||||||
|
default=configuration.RETURN_ANSWERS_SOLUTION,
|
||||||
|
help="Inclut la correction actuelle avant chaque réponse individuelle publiée dans answers.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
steps = [
|
steps = [
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"inputs",
|
"inputs",
|
||||||
@@ -90,23 +136,39 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Prétraitement de l’énoncé",
|
"Prétraitement de l’énoncé",
|
||||||
"Analyser l’énoncé",
|
"Analyser l’énoncé",
|
||||||
"Détecte les questions, leurs groupes, le corrigé et les indications de barème.",
|
"Détecte les questions, leurs groupes, le corrigé et les indications de barème.",
|
||||||
(
|
((python("personal", "Énoncés et solutions personnels (SHEETINFO)", "statement-personal"),)
|
||||||
python("gemini", "Analyse avec Gemini", "statement"),
|
if show_personal_steps else ())
|
||||||
python("personal", "Alternative personnelle", "statement-personal"),
|
+ (python("gemini", "Analyse avec Gemini", "statement"),),
|
||||||
),
|
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec(
|
ArgumentSpec(
|
||||||
"restart",
|
"restart",
|
||||||
"Ignorer le cache (--restart)",
|
"Ignorer le cache (--restart)",
|
||||||
kind="bool",
|
kind="bool",
|
||||||
flag="--restart",
|
flag="--restart",
|
||||||
|
help="Ignore les résultats Gemini mis en cache et recommence entièrement l’analyse de l’énoncé.",
|
||||||
variants=("gemini",),
|
variants=("gemini",),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
requires=("enonce.pdf", "enonce.tex", "correction.tex"),
|
requires=("enonce.pdf", "enonce.tex", "correction.tex"),
|
||||||
artifacts=("labels", "Text", "Sol", "Persp"),
|
artifacts=("labels", "Text", "Sol", "Persp"),
|
||||||
),
|
),
|
||||||
|
StepDefinition(
|
||||||
|
"statement_groups", "Prétraitement de l’énoncé", "Regrouper les questions avec Gemini",
|
||||||
|
"Facultatif après la génération : remplace les groupes par exercice par des groupes "
|
||||||
|
"proposés par Gemini, en conservant les labels, les énoncés, les solutions et les barèmes.",
|
||||||
|
(python("default", "Groupes Gemini", "statement", fixed_args=("--groups-only",)),),
|
||||||
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
|
optional=True, personal=True, requires=("labels", "Text2", "Sol2"),
|
||||||
|
),
|
||||||
|
StepDefinition(
|
||||||
|
"statement_persp", "Prétraitement de l’énoncé", "Remplacer les barèmes par Gemini",
|
||||||
|
"Facultatif : remplace Persp par des barèmes Gemini sur 4 points, pour les groupes actuels. "
|
||||||
|
"Les énoncés et les solutions personnels sont conservés.",
|
||||||
|
(python("default", "Barèmes Gemini", "statement", fixed_args=("--persp-only",)),),
|
||||||
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
|
optional=True, personal=True, requires=("labels", "label_groups", "Text2", "Sol2"),
|
||||||
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"review_persp",
|
"review_persp",
|
||||||
"Prétraitement de l’énoncé",
|
"Prétraitement de l’énoncé",
|
||||||
@@ -129,9 +191,10 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"python",
|
"python",
|
||||||
("rotate",),
|
("rotate",),
|
||||||
fixed_args_before_positionals=True,
|
fixed_args_before_positionals=True,
|
||||||
|
supports_verbose=True,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
optional=True,
|
optional=True,
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
@@ -147,9 +210,10 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"python",
|
"python",
|
||||||
("rename",),
|
("rename",),
|
||||||
fixed_args_before_positionals=True,
|
fixed_args_before_positionals=True,
|
||||||
|
supports_verbose=True,
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"page_splitter",
|
"page_splitter",
|
||||||
@@ -157,18 +221,66 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Séparer et réordonner les pages",
|
"Séparer et réordonner les pages",
|
||||||
"Ouvre l’outil interactif de découpage A3 vers A4. La cible peut être un dossier ou un PDF.",
|
"Ouvre l’outil interactif de découpage A3 vers A4. La cible peut être un dossier ou un PDF.",
|
||||||
(python("default", "Séparation des pages", "page-split"),),
|
(python("default", "Séparation des pages", "page-split"),),
|
||||||
arguments=(arg_target(),),
|
arguments=(
|
||||||
|
arg_target(),
|
||||||
|
ArgumentSpec(
|
||||||
|
"marked",
|
||||||
|
"Copies signalées uniquement",
|
||||||
|
"bool",
|
||||||
|
"--marked",
|
||||||
|
help="Limite l’opération aux copies signalées dans l’interface au lieu de traiter toutes les copies.",
|
||||||
|
),
|
||||||
|
),
|
||||||
artifacts=("Copies", "Copies Originales"),
|
artifacts=("Copies", "Copies Originales"),
|
||||||
),
|
),
|
||||||
|
StepDefinition(
|
||||||
|
"crop_blank_margins",
|
||||||
|
"Prétraitement des copies",
|
||||||
|
"Rogner les zones vides",
|
||||||
|
"Facultatif : détecte les zones vides en haut et en bas malgré les lignes et les perforations, "
|
||||||
|
"puis remplace les PDF dans Copies. Les versions non rognées sont sauvegardées. "
|
||||||
|
"À effectuer avant la détection des labels. Traite plusieurs copies en parallèle.",
|
||||||
|
(python("default", "Rognage des zones vides", "crop-margins"),),
|
||||||
|
arguments=(
|
||||||
|
arg_target("le dossier de l’évaluation ou un PDF du dossier Copies"),
|
||||||
|
ArgumentSpec(
|
||||||
|
"workers",
|
||||||
|
"Copies traitées en parallèle",
|
||||||
|
"int",
|
||||||
|
"--workers",
|
||||||
|
default=5,
|
||||||
|
help="Fixe le nombre maximal de copies rognées simultanément. Une valeur élevée accélère le traitement si la machine possède assez de cœurs et de mémoire.",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
optional=True,
|
||||||
|
requires=("Copies",),
|
||||||
|
auto_start_first_visit=ALWAYS_CROP,
|
||||||
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"cutleft",
|
"cutleft",
|
||||||
"Prétraitement des copies",
|
"Prétraitement des copies",
|
||||||
"Découper la marge des labels",
|
"Découper une partie à gauche pour détection des labels",
|
||||||
"Produit les images de la partie gauche des copies. Une copie précise peut être ciblée.",
|
"Produit les images de la partie gauche des copies. Dans la fenêtre de découpe : "
|
||||||
|
"n décale la zone de 50 px vers la droite, N de 100 px, t de 50 px vers la gauche, "
|
||||||
|
"l l’élargit de 50 px, 1 utilise les pages entières, Entrée valide et s signale une erreur. "
|
||||||
|
"Une copie précise peut être ciblée.",
|
||||||
(python("default", "Découpe", "crop-labels"),),
|
(python("default", "Découpe", "crop-labels"),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target(),
|
arg_target(),
|
||||||
ArgumentSpec("fullpage", "Toujours utiliser la page entière", "bool", "--fullpage"),
|
ArgumentSpec(
|
||||||
|
"fullpage",
|
||||||
|
"Toujours utiliser la page entière",
|
||||||
|
"bool",
|
||||||
|
"--fullpage",
|
||||||
|
help="Désactive la découpe habituelle de la marge gauche et transmet chaque page entière à la détection des labels.",
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"marked",
|
||||||
|
"Copies signalées uniquement",
|
||||||
|
"bool",
|
||||||
|
"--marked",
|
||||||
|
help="Limite l’opération aux copies signalées dans l’interface au lieu de traiter toutes les copies.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
requires=("Copies",),
|
requires=("Copies",),
|
||||||
artifacts=("Cutleft",),
|
artifacts=("Cutleft",),
|
||||||
@@ -181,16 +293,27 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
(python("default", "Détection des labels", "labels"),),
|
(python("default", "Détection des labels", "labels"),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target(),
|
arg_target(),
|
||||||
ArgumentSpec("overwrite", "Régénérer les résultats", "bool", "--overwrite"),
|
ArgumentSpec(
|
||||||
|
"overwrite",
|
||||||
|
"Régénérer les résultats",
|
||||||
|
"bool",
|
||||||
|
"--overwrite",
|
||||||
|
help="Relance la détection même lorsqu’un fichier JSON de labels existe déjà pour la copie.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
requires=("labels", "Copies", "Cutleft"),
|
requires=("labels", "Copies", "Cutleft"),
|
||||||
artifacts=("Copies/*.json",),
|
artifacts=("Copies/*.json",),
|
||||||
|
extra_arguments_help=(
|
||||||
|
"PDF de copie ou images Cutleft supplémentaires de cette évaluation, "
|
||||||
|
"séparés par des espaces. Mettez entre guillemets les chemins contenant des espaces."
|
||||||
|
),
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"plotting",
|
"plotting",
|
||||||
"Labels et regroupement",
|
"Labels et regroupement",
|
||||||
"Vérifier visuellement les labels",
|
"Vérifier visuellement les labels",
|
||||||
"Ouvre la fenêtre de vérification. La cible peut être toute l’évaluation ou une copie.",
|
"Ouvre la fenêtre de vérification. La cible peut être toute l’évaluation ou une copie. "
|
||||||
|
"Le raccourci p revient à l’image précédente, y compris dans la copie précédente.",
|
||||||
(python("default", "Vérification", "review-labels"),),
|
(python("default", "Vérification", "review-labels"),),
|
||||||
arguments=(arg_target(),),
|
arguments=(arg_target(),),
|
||||||
requires=("labels", "Cutleft"),
|
requires=("labels", "Cutleft"),
|
||||||
@@ -207,32 +330,48 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
artifacts=("Copies/Copie*/*",),
|
artifacts=("Copies/Copie*/*",),
|
||||||
auto_start_first_visit=True,
|
auto_start_first_visit=True,
|
||||||
),
|
),
|
||||||
|
StepDefinition(
|
||||||
|
"crop_exercise_bottoms",
|
||||||
|
"Labels et regroupement",
|
||||||
|
"Rogner le bas des réponses",
|
||||||
|
"Facultatif après le découpage par labels : rogne uniquement les grands espaces "
|
||||||
|
"vides au bas des réponses. Les PDF modifiés sont remplacés, les originaux sont "
|
||||||
|
"sauvegardés et plusieurs fichiers sont analysés en parallèle.",
|
||||||
|
(python("default", "Rognage du bas", "crop-answer-bottoms"),),
|
||||||
|
arguments=(
|
||||||
|
arg_target("le dossier de l’évaluation"),
|
||||||
|
ArgumentSpec(
|
||||||
|
"workers",
|
||||||
|
"PDF traités en parallèle",
|
||||||
|
"int",
|
||||||
|
"--workers",
|
||||||
|
default=5,
|
||||||
|
help="Fixe le nombre maximal de PDF de réponse analysés simultanément. Une valeur élevée accélère le traitement si la machine possède assez de cœurs et de mémoire.",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
optional=True,
|
||||||
|
requires=("Copies/Copie*/*.pdf",),
|
||||||
|
auto_start_first_visit=ALWAYS_CROP,
|
||||||
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"grouping",
|
"grouping",
|
||||||
"Labels et regroupement",
|
"Labels et regroupement",
|
||||||
"Regrouper les réponses",
|
"Regrouper les réponses",
|
||||||
"Regroupe les réponses portant le même label pour préparer les requêtes.",
|
"Regroupe les réponses portant le même label pour préparer les requêtes.",
|
||||||
(python("default", "Regroupement", "group-answers"),),
|
(python("default", "Regroupement", "group-answers"),),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
requires=("Copies",),
|
requires=("Copies",),
|
||||||
artifacts=("Par label",),
|
artifacts=("Par label",),
|
||||||
auto_start_first_visit=True,
|
auto_start_first_visit=True,
|
||||||
),
|
),
|
||||||
StepDefinition(
|
|
||||||
"verify_groups",
|
|
||||||
"Labels et regroupement",
|
|
||||||
"Vérifier les groupes produits",
|
|
||||||
"Vérifie que chaque réponse PDF apparaît dans les métadonnées des groupes.",
|
|
||||||
(python("default", "Vérification des groupes", "verify-groups"),),
|
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
|
||||||
optional=True,
|
|
||||||
requires=("Copies", "Par label"),
|
|
||||||
),
|
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"correction",
|
"correction",
|
||||||
"Correction",
|
"Correction",
|
||||||
"Lancer ou intégrer la correction",
|
"Lancer ou intégrer la correction",
|
||||||
"Choisir une correction immédiate, batch, hybride, une recorrection, ou l’intégration d’un batch.",
|
"Choisir une correction immédiate, batch, hybride, une recorrection, ou l’intégration d’un batch. "
|
||||||
|
"En correction immédiate, la barre d’état affiche le nombre de groupes traités et le total. "
|
||||||
|
"Interrompre bloque les nouveaux appels Gemini, attend ceux déjà en cours, sauvegarde leurs résultats puis arrête la commande. "
|
||||||
|
"Au prochain lancement, les validations encore nécessaires (mauvais label ou contenu supplémentaire) reprennent sans refaire la requête principale.",
|
||||||
(
|
(
|
||||||
python("live", "Correction immédiate", "correct"),
|
python("live", "Correction immédiate", "correct"),
|
||||||
python("batch", "Préparer toutes les requêtes batch", "correct", fixed_args=("--batch",)),
|
python("batch", "Préparer toutes les requêtes batch", "correct", fixed_args=("--batch",)),
|
||||||
@@ -247,13 +386,39 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
python("reset", "Réinitialiser les corrections", "correct", fixed_args=("--reset",), dangerous=True),
|
python("reset", "Réinitialiser les corrections", "correct", fixed_args=("--reset",), dangerous=True),
|
||||||
),
|
),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Évaluation ou image Group_X.jpg"),
|
arg_target("le dossier de l’évaluation ou une image Group_X.jpg"),
|
||||||
ArgumentSpec("overwrite", "Écraser les corrections existantes", "bool", "--overwrite", variants=("live",)),
|
ArgumentSpec(
|
||||||
ArgumentSpec("limit", "Limite d’appels Pro", "int", "--limit", variants=("live",)),
|
"overwrite",
|
||||||
ArgumentSpec("batch_from", "Premier label envoyé en batch", "text", "--batch-from", variants=("hybrid",)),
|
"Écraser les corrections existantes",
|
||||||
|
"bool",
|
||||||
|
"--overwrite",
|
||||||
|
help="Relance les corrections demandées même lorsqu’un résultat existe déjà.",
|
||||||
|
variants=("live", "batch", "hybrid", "integrate"),
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"limit",
|
||||||
|
"Limite d’appels Pro",
|
||||||
|
"int",
|
||||||
|
"--limit",
|
||||||
|
help="Limite le nombre d’appels au modèle Pro pendant cette exécution. Laissez ce champ vide pour ne pas imposer de limite.",
|
||||||
|
variants=("live", "hybrid", "refaire"),
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"batch_from",
|
||||||
|
"Premier label envoyé en batch",
|
||||||
|
"text",
|
||||||
|
"--batch-from",
|
||||||
|
help="Indique le premier label traité en batch ; les labels précédents sont corrigés immédiatement.",
|
||||||
|
variants=("hybrid",),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
requires=("Par label", "Persp", "labels"),
|
requires=("Par label", "Persp", "labels"),
|
||||||
artifacts=("correction.json", "batch_requests_*.jsonl"),
|
artifacts=("correction.json", "batch_requests_*.jsonl"),
|
||||||
|
extra_arguments_help=(
|
||||||
|
"Images Group_X.jpg supplémentaires de cette évaluation, séparées par des espaces. "
|
||||||
|
"Mettez entre guillemets les chemins contenant des espaces."
|
||||||
|
),
|
||||||
|
extra_arguments_variants=("live", "batch", "hybrid", "refaire"),
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"submit_batches",
|
"submit_batches",
|
||||||
@@ -261,7 +426,7 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Envoyer les batchs",
|
"Envoyer les batchs",
|
||||||
"Envoie à Gemini les fichiers JSONL produits par le mode batch.",
|
"Envoie à Gemini les fichiers JSONL produits par le mode batch.",
|
||||||
(python("default", "Envoi", "batch-submit"),),
|
(python("default", "Envoi", "batch-submit"),),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
optional=True,
|
optional=True,
|
||||||
artifacts=("batch_jobs.json",),
|
artifacts=("batch_jobs.json",),
|
||||||
skip_for_live_correction=True,
|
skip_for_live_correction=True,
|
||||||
@@ -270,9 +435,8 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"batch_status",
|
"batch_status",
|
||||||
"Correction",
|
"Correction",
|
||||||
"Consulter l’état des batchs",
|
"Consulter l’état des batchs",
|
||||||
"Affiche les jobs Gemini en cours. L’identifiant de téléchargement est facultatif.",
|
"Vérifie les jobs enregistrés pour l’évaluation. Passe à la récupération uniquement lorsque tous ont réussi et que leurs résultats sont disponibles ; sinon, reste sur cette étape.",
|
||||||
(python("default", "État des batchs", "batch-status"),),
|
(python("default", "État des batchs", "batch-status"),),
|
||||||
arguments=(ArgumentSpec("download", "Télécharger le job", "text", "--download"),),
|
|
||||||
optional=True,
|
optional=True,
|
||||||
skip_for_live_correction=True,
|
skip_for_live_correction=True,
|
||||||
),
|
),
|
||||||
@@ -282,30 +446,31 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Récupérer les résultats batch",
|
"Récupérer les résultats batch",
|
||||||
"Télécharge et rassemble les réponses des jobs terminés.",
|
"Télécharge et rassemble les réponses des jobs terminés.",
|
||||||
(python("default", "Récupération", "batch-fetch"),),
|
(python("default", "Récupération", "batch-fetch"),),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
optional=True,
|
optional=True,
|
||||||
skip_for_live_correction=True,
|
skip_for_live_correction=True,
|
||||||
),
|
),
|
||||||
StepDefinition(
|
|
||||||
"post_correction",
|
|
||||||
"Correction",
|
|
||||||
"Nettoyer la correction",
|
|
||||||
"Corrige certains problèmes d’encodage et prépare le texte pour LaTeX.",
|
|
||||||
(python("default", "Post-correction", "post-correction"),),
|
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
|
||||||
requires=("correction.json",),
|
|
||||||
),
|
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"manual_resolution",
|
"manual_resolution",
|
||||||
"Correction",
|
"Correction",
|
||||||
"Résoudre les conflits manuels",
|
"Résoudre les conflits manuels",
|
||||||
"Étape conditionnelle, uniquement si manual_resolutions.txt contient des conflits à traiter.",
|
"Étape conditionnelle, uniquement si manual_resolutions.txt contient des conflits à traiter. "
|
||||||
|
"Après succès, le GUI revient à la correction avec « Recorrection depuis refaire.json » sélectionné.",
|
||||||
(python("default", "Résolution", "resolve-manual"),),
|
(python("default", "Résolution", "resolve-manual"),),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
optional=True,
|
optional=True,
|
||||||
requires=("manual_resolutions.txt", "correction.json"),
|
requires=("manual_resolutions.txt", "correction.json"),
|
||||||
skip_without_manual_conflicts=True,
|
skip_without_manual_conflicts=True,
|
||||||
),
|
),
|
||||||
|
StepDefinition(
|
||||||
|
"post_correction",
|
||||||
|
"Correction",
|
||||||
|
"Nettoyer la correction",
|
||||||
|
"Sauvegarde correction.json dans correction_precleanup.json, puis corrige certains problèmes d’encodage et prépare le texte pour LaTeX. La sauvegarde est remplacée à chaque nettoyage.",
|
||||||
|
(python("default", "Post-correction", "post-correction"),),
|
||||||
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
|
requires=("correction.json",),
|
||||||
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"annotation",
|
"annotation",
|
||||||
"Génération des annotations",
|
"Génération des annotations",
|
||||||
@@ -317,9 +482,22 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
python("grouped", "Annotations groupées (BGnot)", "annotate-grouped"),
|
python("grouped", "Annotations groupées (BGnot)", "annotate-grouped"),
|
||||||
),
|
),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec("overwrite", "Écraser les sorties", "bool", "--overwrite"),
|
ArgumentSpec(
|
||||||
ArgumentSpec("refaire", "Mode refaire", "bool", "--refaire", variants=("checks",)),
|
"overwrite",
|
||||||
|
"Écraser les sorties",
|
||||||
|
"bool",
|
||||||
|
"--overwrite",
|
||||||
|
help="Remplace les annotations déjà générées dans le dossier de sortie sélectionné.",
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"refaire",
|
||||||
|
"Mode refaire",
|
||||||
|
"bool",
|
||||||
|
"--refaire",
|
||||||
|
help="Génère uniquement les copies et questions inscrites dans refaire.json, dans le dossier réservé à la reprise.",
|
||||||
|
variants=("checks", "grouped"),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
requires=("correction.json",),
|
requires=("correction.json",),
|
||||||
artifacts=("Anot", "Bnot", "BGnot"),
|
artifacts=("Anot", "Bnot", "BGnot"),
|
||||||
@@ -331,16 +509,23 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Exporte les annotations vers le dossier EXPORT_DIR défini dans config.py.",
|
"Exporte les annotations vers le dossier EXPORT_DIR défini dans config.py.",
|
||||||
(python("default", "Export", "export"),),
|
(python("default", "Export", "export"),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec(
|
ArgumentSpec(
|
||||||
"annotation_dir",
|
"annotation_dir",
|
||||||
"Dossier d’annotations",
|
"Dossier d’annotations",
|
||||||
"choice",
|
"choice",
|
||||||
default="BGnot",
|
default="BGnot",
|
||||||
choices=("BGnot", "Bnot", "Anot"),
|
choices=("BGnot", "Bnot", "Anot"),
|
||||||
|
help="Choisissez le dossier d’annotations à exporter : groupées, avec cases, ou simples.",
|
||||||
positional=True,
|
positional=True,
|
||||||
),
|
),
|
||||||
ArgumentSpec("refaire", "Mode refaire", "bool", "--refaire"),
|
ArgumentSpec(
|
||||||
|
"refaire",
|
||||||
|
"Mode refaire",
|
||||||
|
"bool",
|
||||||
|
"--refaire",
|
||||||
|
help="Exporte les annotations du passage de reprise BRnot au lieu du dossier principal.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
optional=True,
|
optional=True,
|
||||||
),
|
),
|
||||||
@@ -358,16 +543,23 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Copie les PDF présents dans IMPORT_DIR vers l’évaluation.",
|
"Copie les PDF présents dans IMPORT_DIR vers l’évaluation.",
|
||||||
(python("default", "Import", "import"),),
|
(python("default", "Import", "import"),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec(
|
ArgumentSpec(
|
||||||
"annotation_dir",
|
"annotation_dir",
|
||||||
"Dossier d’annotations",
|
"Dossier d’annotations",
|
||||||
"choice",
|
"choice",
|
||||||
default="BGnot",
|
default="BGnot",
|
||||||
choices=("BGnot", "Bnot", "Anot"),
|
choices=("BGnot", "Bnot", "Anot"),
|
||||||
|
help="Choisissez le dossier principal dans lequel importer les annotations manuscrites.",
|
||||||
positional=True,
|
positional=True,
|
||||||
),
|
),
|
||||||
ArgumentSpec("refaire", "Mode refaire", "bool", "--refaire"),
|
ArgumentSpec(
|
||||||
|
"refaire",
|
||||||
|
"Mode refaire",
|
||||||
|
"bool",
|
||||||
|
"--refaire",
|
||||||
|
help="Importe les annotations manuscrites dans BRnot pour le passage de reprise.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
@@ -380,27 +572,60 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
python("grouped", "Lecture BGnot", "read-grouped"),
|
python("grouped", "Lecture BGnot", "read-grouped"),
|
||||||
),
|
),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec("update_score", "Réappliquer les score.json", "bool", "--update-score"),
|
ArgumentSpec(
|
||||||
ArgumentSpec("refaire", "Mode refaire", "bool", "--refaire", variants=("grouped",)),
|
"update_score",
|
||||||
),
|
"Régénérer avec les nouveaux énoncés/corrigés et les score.json",
|
||||||
|
"bool",
|
||||||
|
"--update-score",
|
||||||
|
help="Régénère les images avec les PDF actuels d’énoncé et de correction ; pour chaque label, le score.json existant prévaut sur le score relu dans l’annotation manuscrite.",
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"refaire",
|
||||||
|
"Mode refaire",
|
||||||
|
"bool",
|
||||||
|
"--refaire",
|
||||||
|
help="Lit les annotations du passage de reprise BRnot et les fusionne avec les copies principales.",
|
||||||
|
variants=("grouped",),
|
||||||
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"annotation_dir",
|
||||||
|
"Passage principal du mode refaire",
|
||||||
|
"choice",
|
||||||
|
"--annotation-dir",
|
||||||
|
default="BGnot",
|
||||||
|
choices=("BGnot", "Bnot", "Anot"),
|
||||||
|
help="Indique le dossier d’annotations du passage principal dans lequel intégrer les questions refaites.",
|
||||||
|
variants=("grouped",),
|
||||||
|
),
|
||||||
|
) + return_answer_arguments,
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
"giving_names",
|
"giving_names",
|
||||||
"Finalisation",
|
"Finalisation",
|
||||||
"Attribuer les noms et préparer A Rendre",
|
"Attribuer les noms et préparer A Rendre",
|
||||||
"Crée le dossier A Rendre à partir du dossier d’annotations choisi.",
|
"Crée le dossier A Rendre à partir du dossier d’annotations choisi, sans passer automatiquement à l’étape suivante. "
|
||||||
|
"Après l’exécution, vous pouvez renommer chaque dossier, ainsi que son fichier .jpg et son fichier .pdf, avec un autre nom ; le suffixe (id) du dossier est conservé. "
|
||||||
|
"Les outils affichés permettent d’identifier et corriger les noms Unknown ou attribués à plusieurs copies. Cliquez ensuite sur « Marquer terminée ».",
|
||||||
(python("default", "Attribution des noms", "giving-names"),),
|
(python("default", "Attribution des noms", "giving-names"),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec(
|
ArgumentSpec(
|
||||||
"annotation_dir",
|
"annotation_dir",
|
||||||
"Dossier d’annotations",
|
"Dossier d’annotations",
|
||||||
"choice",
|
"choice",
|
||||||
default="BGnot",
|
default="BGnot",
|
||||||
choices=("BGnot", "Bnot", "Anot"),
|
choices=("BGnot", "Bnot", "Anot"),
|
||||||
|
help="Choisissez le dossier d’annotations utilisé pour construire les fichiers nommés dans A Rendre.",
|
||||||
positional=True,
|
positional=True,
|
||||||
),
|
),
|
||||||
|
ArgumentSpec(
|
||||||
|
"update",
|
||||||
|
"Mettre à jour uniquement les images de answers",
|
||||||
|
"bool",
|
||||||
|
"--update",
|
||||||
|
help="Met à jour uniquement les images du dossier answers de chaque élève existant, en identifiant la copie par le numéro final entre parenthèses et sans modifier le nom du dossier ni les autres fichiers.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
artifacts=("A Rendre",),
|
artifacts=("A Rendre",),
|
||||||
),
|
),
|
||||||
@@ -425,10 +650,16 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Étapes personnelles",
|
"Étapes personnelles",
|
||||||
"Mettre à jour le fichier ODS",
|
"Mettre à jour le fichier ODS",
|
||||||
"Transfère les scores, avec la possibilité de n’écrire que leur somme.",
|
"Transfère les scores, avec la possibilité de n’écrire que leur somme.",
|
||||||
(python("default", "Mise à jour ODS", "update-ods"),),
|
(python("default", "Mise à jour ODS", "update-ods", supports_verbose=False),),
|
||||||
arguments=(
|
arguments=(
|
||||||
arg_target("Dossier de l’évaluation"),
|
arg_target("le dossier de l’évaluation"),
|
||||||
ArgumentSpec("sum", "Écrire seulement la somme", "bool", "--sum"),
|
ArgumentSpec(
|
||||||
|
"sum",
|
||||||
|
"Écrire seulement la somme",
|
||||||
|
"bool",
|
||||||
|
"--sum",
|
||||||
|
help="Écrit uniquement la note totale de chaque élève dans le fichier ODS, sans détailler les scores par question.",
|
||||||
|
),
|
||||||
),
|
),
|
||||||
personal=True,
|
personal=True,
|
||||||
),
|
),
|
||||||
@@ -452,9 +683,11 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"final_score",
|
"final_score",
|
||||||
"Étapes personnelles",
|
"Étapes personnelles",
|
||||||
"Ajouter le score final",
|
"Ajouter le score final",
|
||||||
"Génère les fichiers de diffusion avec le score final.",
|
"Prérequis immédiat : gestion_classe wse doit avoir été exécuté juste avant. "
|
||||||
(python("default", "Score final", "add-final-score"),),
|
"Génère ensuite les dossiers de diffusion avec le score final, puis copie "
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
"gestion_classe/Staging/histogramme.pdf dans le dossier Server/copies de l’évaluation.",
|
||||||
|
(python("default", "Score final", "add-final-score", supports_verbose=False),),
|
||||||
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
personal=True,
|
personal=True,
|
||||||
),
|
),
|
||||||
StepDefinition(
|
StepDefinition(
|
||||||
@@ -480,7 +713,7 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"Nettoyer les fichiers intermédiaires",
|
"Nettoyer les fichiers intermédiaires",
|
||||||
"Supprime définitivement les fichiers permettant de reprendre le parcours. "
|
"Supprime définitivement les fichiers permettant de reprendre le parcours. "
|
||||||
"Conserve les PDF traités, les fichiers textuels de l’énoncé, correction.json, "
|
"Conserve les PDF traités, les fichiers textuels de l’énoncé, correction.json, "
|
||||||
"les journaux, ainsi que les images et score.json de A Rendre.",
|
"les journaux, ainsi que les images, PDF, score.json et info.json de A Rendre.",
|
||||||
(
|
(
|
||||||
python(
|
python(
|
||||||
"default",
|
"default",
|
||||||
@@ -494,13 +727,19 @@ def build_workflow(show_personal_steps: bool) -> list[StepDefinition]:
|
|||||||
"seront supprimés. Il ne sera plus possible de reprendre une "
|
"seront supprimés. Il ne sera plus possible de reprendre une "
|
||||||
"étape sans régénérer ses données.\n\n"
|
"étape sans régénérer ses données.\n\n"
|
||||||
"Les PDF traités, les fichiers textuels de l’énoncé, "
|
"Les PDF traités, les fichiers textuels de l’énoncé, "
|
||||||
"correction.json, les journaux, ainsi que les images et "
|
"correction.json, les journaux, ainsi que les images, PDF et "
|
||||||
"score.json de A Rendre seront conservés.\n\n"
|
"score.json et info.json de A Rendre seront conservés.\n\n"
|
||||||
"Continuer ?"
|
"Continuer ?"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
python(
|
||||||
|
"dry_run",
|
||||||
|
"Prévisualiser sans supprimer",
|
||||||
|
"clean",
|
||||||
|
fixed_args=("--dry-run",),
|
||||||
|
),
|
||||||
),
|
),
|
||||||
arguments=(arg_target("Dossier de l’évaluation"),),
|
arguments=(arg_target("le dossier de l’évaluation"),),
|
||||||
optional=True,
|
optional=True,
|
||||||
requires=("Copies", "correction.json", "A Rendre"),
|
requires=("Copies", "correction.json", "A Rendre"),
|
||||||
),
|
),
|
||||||
@@ -533,13 +772,28 @@ def build_refaire_workflow() -> list[StepDefinition]:
|
|||||||
for suffix, title, description, program, flags, optional in definitions:
|
for suffix, title, description, program, flags, optional in definitions:
|
||||||
arguments = (arg_target(),) if program else ()
|
arguments = (arg_target(),) if program else ()
|
||||||
if suffix == "merge":
|
if suffix == "merge":
|
||||||
arguments += (ArgumentSpec("annotation_dir", "Passage principal", "choice", "--annotation-dir", default="BGnot", choices=("BGnot", "Bnot", "Anot")),)
|
arguments += (ArgumentSpec(
|
||||||
|
"annotation_dir",
|
||||||
|
"Passage principal",
|
||||||
|
"choice",
|
||||||
|
"--annotation-dir",
|
||||||
|
default="BGnot",
|
||||||
|
choices=("BGnot", "Bnot", "Anot"),
|
||||||
|
help="Indique le dossier d’annotations du passage principal dans lequel intégrer les questions refaites.",
|
||||||
|
),)
|
||||||
requirements = ("refaire.json", "Copies", "labels", "correction.json")
|
requirements = ("refaire.json", "Copies", "labels", "correction.json")
|
||||||
if suffix in {"export", "tablet", "import", "merge"}:
|
if suffix in {"export", "tablet", "import", "merge"}:
|
||||||
requirements += ("BRnot",)
|
requirements += ("BRnot",)
|
||||||
steps.append(StepDefinition(
|
steps.append(StepDefinition(
|
||||||
f"refaire_{suffix}", SECTION, title, description,
|
f"refaire_{suffix}", SECTION, title, description,
|
||||||
(CommandVariant("default", title, program, "python" if program else "manual", flags),),
|
(CommandVariant(
|
||||||
|
"default",
|
||||||
|
title,
|
||||||
|
program,
|
||||||
|
"python" if program else "manual",
|
||||||
|
flags,
|
||||||
|
supports_verbose=program is not None,
|
||||||
|
),),
|
||||||
arguments=arguments, optional=optional, requires=requirements,
|
arguments=arguments, optional=optional, requires=requirements,
|
||||||
))
|
))
|
||||||
return steps
|
return steps
|
||||||
@@ -556,6 +810,7 @@ def build_command(
|
|||||||
values: dict[str, object],
|
values: dict[str, object],
|
||||||
evaluation_arg: str,
|
evaluation_arg: str,
|
||||||
extra_arguments: str = "",
|
extra_arguments: str = "",
|
||||||
|
verbose: bool = False,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
if variant.kind == "manual" or not variant.program:
|
if variant.kind == "manual" or not variant.program:
|
||||||
return []
|
return []
|
||||||
@@ -570,6 +825,9 @@ def build_command(
|
|||||||
|
|
||||||
positionals: list[str] = []
|
positionals: list[str] = []
|
||||||
options: list[str] = []
|
options: list[str] = []
|
||||||
|
if step.id == "batch_status":
|
||||||
|
# Use the loaded evaluation without introducing another input field.
|
||||||
|
options.extend(("--evaluation", evaluation_arg))
|
||||||
for spec in step.arguments:
|
for spec in step.arguments:
|
||||||
if spec.variants and variant.id not in spec.variants:
|
if spec.variants and variant.id not in spec.variants:
|
||||||
continue
|
continue
|
||||||
@@ -577,6 +835,8 @@ def build_command(
|
|||||||
if spec.kind == "bool":
|
if spec.kind == "bool":
|
||||||
if bool(value) and spec.flag:
|
if bool(value) and spec.flag:
|
||||||
options.append(spec.flag)
|
options.append(spec.flag)
|
||||||
|
elif not bool(value) and spec.false_flag:
|
||||||
|
options.append(spec.false_flag)
|
||||||
continue
|
continue
|
||||||
if value is None or str(value).strip() == "":
|
if value is None or str(value).strip() == "":
|
||||||
continue
|
continue
|
||||||
@@ -592,6 +852,8 @@ def build_command(
|
|||||||
if not variant.fixed_args_before_positionals:
|
if not variant.fixed_args_before_positionals:
|
||||||
command.extend(variant.fixed_args)
|
command.extend(variant.fixed_args)
|
||||||
command.extend(options)
|
command.extend(options)
|
||||||
|
if verbose and variant.supports_verbose:
|
||||||
|
command.append("--verbose")
|
||||||
if extra_arguments.strip():
|
if extra_arguments.strip():
|
||||||
command.extend(shlex.split(extra_arguments, posix=os.name != "nt"))
|
command.extend(shlex.split(extra_arguments, posix=os.name != "nt"))
|
||||||
return command
|
return command
|
||||||
|
|||||||
+18
-3
@@ -6,26 +6,39 @@ API_KEY = os.environ.get("GEMINI_API_KEY")
|
|||||||
EXPORT_DIR = Path("Export")
|
EXPORT_DIR = Path("Export")
|
||||||
IMPORT_DIR = Path("Import")
|
IMPORT_DIR = Path("Import")
|
||||||
|
|
||||||
|
# Fichiers à inclure dans A Rendre (les sources d'annotations sont conservées).
|
||||||
|
RETURN_JPEG_ENABLED = True
|
||||||
|
RETURN_PDF_ENABLED = True
|
||||||
|
RETURN_ANSWERS_ENABLED = False
|
||||||
|
RETURN_ANSWERS_CONTEXT = False
|
||||||
|
RETURN_ANSWERS_QUESTION = True
|
||||||
|
RETURN_ANSWERS_SOLUTION = False
|
||||||
|
|
||||||
# Les étapes gestion_classe, ODS et publication sont masquées par défaut.
|
# Les étapes gestion_classe, ODS et publication sont masquées par défaut.
|
||||||
SHOW_PERSONAL_STEPS = False
|
SHOW_PERSONAL_STEPS = False
|
||||||
|
|
||||||
|
# Lance automatiquement les deux étapes facultatives de rognage dans le GUI.
|
||||||
|
ALWAYS_CROP = False
|
||||||
|
|
||||||
# Chemins utilisés uniquement par les étapes personnelles.
|
# Chemins utilisés uniquement par les étapes personnelles.
|
||||||
CURRENT_SCORE_ODS_PATH = Path("current_eval.ods")
|
CURRENT_SCORE_ODS_PATH = Path("current_eval.ods")
|
||||||
FINAL_SCORE_ODS_PATH = Path("simple_eval.ods")
|
FINAL_SCORE_ODS_PATH = Path("simple_eval.ods")
|
||||||
FINAL_SCORE_OUTPUT_DIR = Path("Server") / "copies"
|
FINAL_SCORE_OUTPUT_DIR = Path("Server") / "copies"
|
||||||
|
FINAL_SCORE_HISTOGRAM_PATH = Path("histogramme.pdf")
|
||||||
FINAL_SCORE_FONT_PATH = None
|
FINAL_SCORE_FONT_PATH = None
|
||||||
|
|
||||||
# Modèle pour des choses très légères
|
# Modèle pour des choses très légères
|
||||||
MODEL_LITE_ID = "gemini-3.5-flash-lite"
|
MODEL_LITE_ID = "gemini-3.5-flash-lite"
|
||||||
|
|
||||||
# Modèle pour identifier visuellement les labels
|
# Modèle pour identifier visuellement les labels
|
||||||
MODEL_FOR_LABEL_ID = "gemini-3.5-flash-lite"
|
# MODEL_FOR_LABEL_ID = "gemini-3.5-flash-lite" # 3.5 flash lite marche moyennement, il insiste pour mettre les labels dans l'ordre, sans considérer ce qu'il y a écrit
|
||||||
|
MODEL_FOR_LABEL_ID = "gemini-3.8-flash"
|
||||||
|
|
||||||
# Modèle pour des choses normales
|
# Modèle pour des choses normales
|
||||||
MODEL_FLASH_ID = "gemini-3.6-flash"
|
MODEL_FLASH_ID = "gemini-3.8-flash"
|
||||||
|
|
||||||
# Modèle pour des choses dures
|
# Modèle pour des choses dures
|
||||||
MODEL_PRO_ID = "gemini-3.6-flash"
|
MODEL_PRO_ID = "gemini-3.8-flash"
|
||||||
# MODEL_PRO_ID = "gemini-3.1-pro-preview"
|
# MODEL_PRO_ID = "gemini-3.1-pro-preview"
|
||||||
|
|
||||||
PAGE_SPLITTER_KB = {
|
PAGE_SPLITTER_KB = {
|
||||||
@@ -41,6 +54,7 @@ PAGE_SPLITTER_KB = {
|
|||||||
"next_page": "s",
|
"next_page": "s",
|
||||||
"discard_page": "z",
|
"discard_page": "z",
|
||||||
"send_end": "a", # Send this page to the end
|
"send_end": "a", # Send this page to the end
|
||||||
|
"reverse_pages": "i", # Reverse page order and restart at the new first page
|
||||||
"restart_file": "T",
|
"restart_file": "T",
|
||||||
"arranger": "A", # Call `pdf arranger` software, if available
|
"arranger": "A", # Call `pdf arranger` software, if available
|
||||||
"prev_file": "P",
|
"prev_file": "P",
|
||||||
@@ -69,6 +83,7 @@ LATEX_BEFORE = r"""\documentclass[varwidth=24.8cm,margin=0.4cm]{standalone}
|
|||||||
\usepackage{minted}
|
\usepackage{minted}
|
||||||
\usepackage{graphicx}
|
\usepackage{graphicx}
|
||||||
\usepackage{enumitem}
|
\usepackage{enumitem}
|
||||||
|
\usepackage{multicol}
|
||||||
\begin{document}
|
\begin{document}
|
||||||
\begin{minipage}{24.8cm}
|
\begin{minipage}{24.8cm}
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
# Scanned PDF margin cropping
|
||||||
|
|
||||||
|
Copienator has one automatic crop detector. It finds strongly coloured or dark
|
||||||
|
ink, recovers nearby weaker strokes, and removes substantial blank areas above
|
||||||
|
and below the detected content. It is designed for scanned student work on
|
||||||
|
plain, lined, or gridded paper, including mildly skewed pages and recurring
|
||||||
|
punched-hole artifacts.
|
||||||
|
|
||||||
|
## Review utility
|
||||||
|
|
||||||
|
Run from the repository root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python -m copienator.crop_blank_margins Interro01/Copies tmp/cropped-copies
|
||||||
|
```
|
||||||
|
|
||||||
|
The input can be a directory or one PDF. Directory processing includes only
|
||||||
|
PDFs directly inside that directory. The utility writes processed PDFs, an HTML
|
||||||
|
comparison gallery, JPEG previews, and JSON/CSV reports into the output
|
||||||
|
directory. Source PDFs are never modified.
|
||||||
|
|
||||||
|
Available options:
|
||||||
|
|
||||||
|
- `--dpi 200`: analysis resolution.
|
||||||
|
- `--padding-mm 6`: space retained around detected content.
|
||||||
|
- `--min-crop-mm 5`: minimum worthwhile removal at either edge.
|
||||||
|
|
||||||
|
The output retains filenames, page order, page count, colour, rotation, and the
|
||||||
|
embedded scan data. Cropping changes the PDF CropBox rather than rasterizing the
|
||||||
|
page. Red shading in `index.html` shows the removed part of each original page.
|
||||||
|
A `review-*` status records uncertainty; one edge can still be cropped while the
|
||||||
|
other remains unchanged.
|
||||||
|
|
||||||
|
## Detection
|
||||||
|
|
||||||
|
The detector uses colour and darkness as strong ink seeds. It recovers connected
|
||||||
|
weak strokes within a 2 mm neighbourhood using directional contrast, which
|
||||||
|
limits growth along paper lines. Two seed thresholds are compared so unstable
|
||||||
|
boundaries can be flagged for review.
|
||||||
|
|
||||||
|
For dark neutral paper, it deskews the scan and confirms repeated horizontal or
|
||||||
|
vertical ruling before suppressing paper-line pixels. It uses short directional
|
||||||
|
openings to tolerate broken or bent grid lines. Very dark fraction bars and
|
||||||
|
diagram axes remain protected. Repeated components with similar size and
|
||||||
|
alignment in the outer 15 mm are treated as punched holes only when at least
|
||||||
|
three span a substantial part of the page. Writing in the same side column still
|
||||||
|
protects its margin.
|
||||||
|
|
||||||
|
On confirmed ruled paper, faint sparse strokes in the central 80% of the page
|
||||||
|
use a moderately more sensitive component filter. The outermost 9% on each side
|
||||||
|
uses a stricter filter because punched holes, torn binding edges, and page
|
||||||
|
numbers normally appear there.
|
||||||
|
|
||||||
|
The large-blank refinement changes an edge only when it finds at least 30 mm of
|
||||||
|
additional empty paper. A 2 mm recovery neighbourhood is applied before the
|
||||||
|
normal padding. Apparently blank pages and pages without reliable ink seeds are
|
||||||
|
kept at full height.
|
||||||
|
|
||||||
|
This remains a heuristic. Extremely faint isolated pencil marks, unusually
|
||||||
|
damaged ruling, and repeated handwriting shaped like hole artifacts can be
|
||||||
|
ambiguous. Review crops before generating answer coordinates.
|
||||||
|
|
||||||
|
## Optional GUI step
|
||||||
|
|
||||||
|
After **Séparer et réordonner les pages**, the GUI offers **Rogner les zones
|
||||||
|
vides**. It can process the whole evaluation or a selected PDF. It runs at 200
|
||||||
|
dpi with 6 mm padding and uses five worker processes by default. The CLI form is:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python -m copienator crop-margins EVALUATION --workers 5
|
||||||
|
```
|
||||||
|
|
||||||
|
The batch is fully prepared before any source is replaced. Detection failures
|
||||||
|
and interruptions leave the working PDFs intact; replacement errors roll back.
|
||||||
|
Each successful run saves the untrimmed PDFs and its report under
|
||||||
|
`.copienator/runs/crop-margins-*/`. The **Archivage** step removes these backups
|
||||||
|
and reports while keeping the cropped copies and execution logs.
|
||||||
|
|
||||||
|
Cropping must run before label detection. If a selected PDF already has a
|
||||||
|
same-named JSON coordinate file, the command stops before changing any PDFs.
|
||||||
|
When `ALWAYS_CROP` is true in `config.py`, this facultative step starts
|
||||||
|
automatically when first reached; the default configuration keeps it manual.
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
Separate worker processes isolate MuPDF and each worker uses one OpenCV thread.
|
||||||
|
The detector uses native channel operations, vectorized component filtering,
|
||||||
|
cached separable background filtering, and a coarser Hough voting step for skew
|
||||||
|
candidates. Report rows remain ordered by copy and page regardless of worker
|
||||||
|
completion order.
|
||||||
|
|
||||||
|
On the Ryzen 7 PRO 7840U, an end-to-end benchmark took 56.48 seconds for 48 PDFs
|
||||||
|
of 10 pages, including rendering, detection, PDF writing, backups, and
|
||||||
|
replacement. The fixture uses Interro01 and DS08VA scans and cycles pages in
|
||||||
|
shorter copies, so it does not contain 480 distinct scans. Runtime depends on the
|
||||||
|
CPU, storage, and scan content.
|
||||||
|
|
||||||
|
Run the focused checks with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python -m unittest tests.test_crop_blank_margins tests.test_ink_detection tests.test_crop_margins_command -v
|
||||||
|
```
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Bottom cropping after exercise splitting
|
||||||
|
|
||||||
|
This review utility processes the exercise PDFs stored directly under
|
||||||
|
`Copies/CopieXX/`. It never changes the source files. Modified PDFs are written
|
||||||
|
to a matching tree under the chosen output directory; unchanged PDFs are not
|
||||||
|
copied.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python -m copienator.crop_exercise_bottoms Interro01 tmp/exercise-bottom-crop
|
||||||
|
```
|
||||||
|
|
||||||
|
One line is 1/36 of the uncropped full-page height recorded by the matching
|
||||||
|
`Copies/CopieXX.pdf`. Each exercise PDF page is considered independently:
|
||||||
|
|
||||||
|
1. Pages shorter than 10 lines are skipped.
|
||||||
|
2. The bottom 0.75 line is excluded from detection so a fragment of the next
|
||||||
|
label cannot keep a large blank area.
|
||||||
|
3. The existing scan detector locates the last ink above that strip and keeps
|
||||||
|
6 mm of padding.
|
||||||
|
4. The bottom CropBox changes only if the proposed removal is at least 4 lines.
|
||||||
|
The top CropBox is always retained.
|
||||||
|
|
||||||
|
The ignored 0.75-line strip is therefore not removed on its own. It is included
|
||||||
|
in the result only when the complete proposed crop passes the four-line
|
||||||
|
threshold.
|
||||||
|
|
||||||
|
The output contains `index.html`, previews with removed areas shaded red, a
|
||||||
|
plain `cropped-files.txt` list, `report.json`, and `report.csv`. The command uses
|
||||||
|
five worker processes by default; `--workers`, `--dpi`, and `--padding-mm` are
|
||||||
|
configurable.
|
||||||
|
|
||||||
|
## GUI integration
|
||||||
|
|
||||||
|
After **Découper les réponses par question**, the GUI offers the facultative
|
||||||
|
step **Rogner le bas des réponses**. It runs the same thresholds at 200 dpi and
|
||||||
|
uses five worker processes by default. Only PDFs with an accepted crop are
|
||||||
|
replaced. Their unmodified versions and the complete report are stored under
|
||||||
|
`.copienator/runs/crop-exercise-bottoms-*/`; a failure or interruption before
|
||||||
|
publication leaves every exercise PDF unchanged. The **Archivage** step removes
|
||||||
|
these retained originals and reports. When `ALWAYS_CROP` is true in `config.py`,
|
||||||
|
this facultative step starts automatically when first reached; the default
|
||||||
|
configuration keeps it manual.
|
||||||
@@ -0,0 +1,258 @@
|
|||||||
|
# Final output: `A Rendre`
|
||||||
|
|
||||||
|
This documents the current implementation, as of 2026-09-12. The return folder
|
||||||
|
referred to as « À rendre » is named **`A Rendre`** on disk. JPEG files use the
|
||||||
|
extension **`.jpg`**, not `.jpeg`.
|
||||||
|
|
||||||
|
## Files and their sources
|
||||||
|
|
||||||
|
After grouped correction and review:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python -m copienator read-grouped Interro
|
||||||
|
python -m copienator giving-names Interro BGnot
|
||||||
|
```
|
||||||
|
|
||||||
|
The expected layout for a copy is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Interro/A Rendre/
|
||||||
|
└── Student Name (01)/
|
||||||
|
├── Student Name.jpg
|
||||||
|
├── Student Name.pdf
|
||||||
|
├── score.json
|
||||||
|
├── info.json
|
||||||
|
└── answers/ # when individual answer export is enabled
|
||||||
|
├── 001 - Ex 1.jpg
|
||||||
|
└── 002 - Ex 2.jpg
|
||||||
|
```
|
||||||
|
|
||||||
|
The name comes from `Copies/Copie01.json` (`name`), with filename sanitization.
|
||||||
|
The copy ID distinguishes folders even when several copies have the same name.
|
||||||
|
`giving-names` links the full JPEG, PDF and score file (or copies them when links
|
||||||
|
are unavailable). It writes `info.json` and optionally composes the individual
|
||||||
|
answer JPEGs.
|
||||||
|
|
||||||
|
| Return file | Source under `BGnot/Copie01/` | Contents |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `Student Name.jpg` | `Concat.jpg` | Full continuous image of the compiled answers and corrections. |
|
||||||
|
| `Student Name.pdf` | `Concat_F.pdf` | Filtered, paginated correction with context, questions and solutions. |
|
||||||
|
| `score.json` | `score.json` | Per-question scores, including questions omitted from the filtered PDF. |
|
||||||
|
| `info.json` | `info.json` | Answer presence, empty-answer classification, PDF membership and score. |
|
||||||
|
| `answers/*.jpg` | Final per-label JPEGs selected by `info.json` | One annotated non-empty answer, with optional supplementary material. |
|
||||||
|
|
||||||
|
## Enabling or disabling outputs
|
||||||
|
|
||||||
|
Set these independent options in `config.py` (the defaults also apply when
|
||||||
|
absent from an older personal configuration):
|
||||||
|
|
||||||
|
```python
|
||||||
|
RETURN_JPEG_ENABLED = True
|
||||||
|
RETURN_PDF_ENABLED = True
|
||||||
|
RETURN_ANSWERS_ENABLED = False
|
||||||
|
RETURN_ANSWERS_CONTEXT = False
|
||||||
|
RETURN_ANSWERS_QUESTION = True
|
||||||
|
RETURN_ANSWERS_SOLUTION = False
|
||||||
|
```
|
||||||
|
|
||||||
|
The personal `config.py` enables `RETURN_ANSWERS_ENABLED`; the distributed
|
||||||
|
default is `False`. Set a full-output option to `False`, then rerun `giving-names` to omit that file from
|
||||||
|
`A Rendre`. For each prepared copy, any existing named return file of a disabled
|
||||||
|
type is removed, including a symlink or fallback copy. Its annotation source
|
||||||
|
remains intact. These options control return publication, not intermediate
|
||||||
|
rendering or scoring. `score.json` and `info.json` are always included and have
|
||||||
|
no disabling options.
|
||||||
|
|
||||||
|
Cleanup allows the JPEG to be absent when disabled, still requires `score.json`,
|
||||||
|
and preserves return PDFs when present.
|
||||||
|
|
||||||
|
## Individual answer JPEGs
|
||||||
|
|
||||||
|
With `RETURN_ANSWERS_ENABLED = True`, `giving-names` generates an `answers/`
|
||||||
|
subdirectory inside each student's return folder. It includes **every non-empty
|
||||||
|
compiled answer**, even a perfect answer omitted from the filtered PDF. Labels
|
||||||
|
marked `empty-answer` and labels without an answer are excluded. Every JPEG
|
||||||
|
contains the final annotated student answer, including retained feedback and
|
||||||
|
extracted handwriting.
|
||||||
|
|
||||||
|
The three supplementary options independently prepend, in this order:
|
||||||
|
|
||||||
|
1. Applicable context PDFs, if `RETURN_ANSWERS_CONTEXT` is enabled.
|
||||||
|
2. The question, if `RETURN_ANSWERS_QUESTION` is enabled (the default).
|
||||||
|
3. The model solution, if `RETURN_ANSWERS_SOLUTION` is enabled.
|
||||||
|
4. The annotated student answer, always.
|
||||||
|
|
||||||
|
These use the same `Text2`/`Sol2` sources as the filtered PDF. Missing supplements
|
||||||
|
are skipped; an unreadable existing file fails the export. They are concatenated
|
||||||
|
vertically on white, without PDF pagination or its black/blue borders. The
|
||||||
|
options affect only these individual images, not the full JPEG or filtered PDF.
|
||||||
|
Disabling all supplements produces just the annotated answer.
|
||||||
|
|
||||||
|
Filenames use natural label order, a three-digit minimum sequence number, and a
|
||||||
|
sanitized label (`001 - Ex 1.jpg`). Numbering prevents filename collisions when
|
||||||
|
labels differ only by characters forbidden in filenames. JSON keys retain exact
|
||||||
|
labels. The managed `answers/` directory is replaced on successful generation,
|
||||||
|
so removed/empty answers do not leave stale images; failures preserve the previous
|
||||||
|
directory. Disabling the option clears this directory on the next `giving-names`
|
||||||
|
run. Separate storage keeps these images out of the personal final-mark stamping
|
||||||
|
step, which reads only JPEGs directly inside the student's folder.
|
||||||
|
|
||||||
|
Recompile annotations once before exporting old evaluations with this option:
|
||||||
|
the compiler now saves every final answer block and writes `info.json`
|
||||||
|
next to them. For the grouped workflow, run `read-grouped`, then `giving-names`.
|
||||||
|
This avoids reconstructing a reviewed answer from outdated correction data.
|
||||||
|
|
||||||
|
## `info.json`: per-question information
|
||||||
|
|
||||||
|
Every label in `score.json` has an object containing exactly four fields:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Ex 1": {"present": true, "not_empty": true, "touched": false, "score": "4"},
|
||||||
|
"Ex 2": {"present": true, "not_empty": true, "touched": true, "score": "2"},
|
||||||
|
"Empty": {"present": true, "not_empty": false, "touched": false, "score": "0"},
|
||||||
|
"Absent": {"present": false, "not_empty": false, "touched": false, "score": ""}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `present`: an answer entry exists for this student and label in the compilation
|
||||||
|
data. A supplied answer judged empty still has `present: true`.
|
||||||
|
- `not_empty`: the answer was not marked `empty-answer` and was successfully
|
||||||
|
compiled. Absent answers have `not_empty: false`. When individual export is
|
||||||
|
enabled, a JPEG is generated if and only if both `present` and `not_empty` are
|
||||||
|
true. These fields describe the answer regardless of export settings.
|
||||||
|
- `touched`: the answer appears in the compiled filtered `Concat_F.pdf`, using
|
||||||
|
the actual selection including handwriting and selective redo preservation.
|
||||||
|
It is not a flag for human edits. Empty and absent answers have `false`.
|
||||||
|
- `score`: the same value as `score.json` (normally a numeric string, or `""`
|
||||||
|
for an unpopulated score). Editing scores requires recompilation to update the
|
||||||
|
images; return publication uses current `score.json` values for this field.
|
||||||
|
|
||||||
|
`info.json` is always exported, even when individual JPEGs or the named PDF are
|
||||||
|
disabled. `touched` describes the source filtered PDF. In normal `Anot` and
|
||||||
|
`Bnot` flows, no filtered PDF is produced and all `touched` values are false.
|
||||||
|
|
||||||
|
This file replaces `touched.json` and the internal `answer_labels.json` manifest.
|
||||||
|
Recompile old annotations, then run `giving-names`; successful regeneration and
|
||||||
|
publication remove the obsolete files from their respective folders. Missing
|
||||||
|
or malformed metadata requires recompilation rather than guessing answer presence
|
||||||
|
from scores. Cleanup preserves `info.json` alongside `score.json`.
|
||||||
|
|
||||||
|
## JPEG: the full compiled correction
|
||||||
|
|
||||||
|
The JPEG stacks the rendered answer blocks vertically in natural label order
|
||||||
|
(for example, Ex 2 precedes Ex 10). Each block contains the scanned answer, its
|
||||||
|
label and score, retained global and local feedback, and detected handwritten
|
||||||
|
review annotations. Local feedback can include red rectangles and comments in
|
||||||
|
the left margin. Review checkboxes are applied as actions rather than reproduced
|
||||||
|
as controls; internal error labels are hidden during recompilation.
|
||||||
|
|
||||||
|
The result is one RGB image of variable height, with no page breaks. It contains
|
||||||
|
all successfully compiled answer blocks, including answers scored 4 with no
|
||||||
|
remaining feedback. “Full” refers to those answer blocks, not the original scan
|
||||||
|
pages or every question in the statement. Missing/unrenderable answers cannot
|
||||||
|
be included, and the renderer normally suppresses `empty-answer` results.
|
||||||
|
The grouped compiler refuses to publish a new set when compilation is incomplete.
|
||||||
|
|
||||||
|
The JPEG does not prepend the question, context or model solution PDFs.
|
||||||
|
|
||||||
|
## PDF: a different selection and layout
|
||||||
|
|
||||||
|
**The PDF is not a PDF conversion of the JPEG.** During an ordinary full grouped
|
||||||
|
recompilation, an answer is omitted only when all three conditions hold:
|
||||||
|
|
||||||
|
- Its score is at least 4.
|
||||||
|
- Every feedback item is marked `to_delete` (also true for an empty feedback list).
|
||||||
|
- There are no significant detected handwritten annotations for that answer.
|
||||||
|
|
||||||
|
Thus, a 4/4 answer with retained feedback or handwriting still appears. Scores
|
||||||
|
for omitted answers remain in `score.json`, and their answer blocks remain in
|
||||||
|
the JPEG. Handwriting significance currently means more than 20 pixels with
|
||||||
|
alpha greater than 50 in the extracted annotation layer.
|
||||||
|
|
||||||
|
For each retained answer, the PDF stacks the following available material:
|
||||||
|
|
||||||
|
1. Applicable context PDFs from `Text2/CTXT first_label -> last_label.pdf`.
|
||||||
|
2. The question from `Text2/<label>.pdf`.
|
||||||
|
3. The model solution from `Sol2/<label>.pdf`.
|
||||||
|
4. The same compiled answer block used for the JPEG.
|
||||||
|
|
||||||
|
Missing supplementary PDFs are skipped. Contexts can repeat for successive
|
||||||
|
questions. Each question's complete group stays together on one page; groups
|
||||||
|
are packed until the next would exceed the target height. An oversized group
|
||||||
|
gets its own taller page, rather than being split. Pages are raster images saved
|
||||||
|
as PDF at 100 dpi, with variable heights, not fixed A4 sheets or searchable text.
|
||||||
|
|
||||||
|
The target height is `int(max_image_width * 1.414 * 1.25)` pixels. Small white
|
||||||
|
margins are added on the left and above/below the page. The first image of each
|
||||||
|
group receives a black border and the second a blue border. These borders are
|
||||||
|
assigned by position, so they do not consistently identify question and solution
|
||||||
|
when contexts are present or supplementary files are missing.
|
||||||
|
|
||||||
|
If nothing survives filtering, `read-grouped` removes old `Concat_F` outputs and
|
||||||
|
does not create a new PDF. During a selective `--refaire` merge, saved answer
|
||||||
|
images outside the selection are kept in the filtered output without reapplying
|
||||||
|
the perfect-answer filter; the resulting PDF can therefore retain more answers
|
||||||
|
than a full recompilation.
|
||||||
|
|
||||||
|
## JSON: per-question scores
|
||||||
|
|
||||||
|
`score.json` is a flat JSON object keyed by the exact question labels. For example
|
||||||
|
(illustrative data):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Ex 1 : 1)": "4",
|
||||||
|
"Ex 1 : 2)": "2.5",
|
||||||
|
"Ex 2": ""
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Values are **strings**, including numeric scores. The normal question scale is
|
||||||
|
0 to 4. `""` means no score was populated for that label; it is distinct from
|
||||||
|
`"0"`. The compiler initializes all labels from the evaluation's `labels` file
|
||||||
|
to `""`, then fills processed scores. This file contains no student identity,
|
||||||
|
feedback, annotation coordinates, grading weights, or overall final mark.
|
||||||
|
|
||||||
|
Scores incorporate review checkbox changes and, when requested, existing score
|
||||||
|
overrides via `read-grouped --update-score` (or `read-annotations --update-score`
|
||||||
|
for `Bnot`). Editing the JSON alone does not update the rendered scores. Overrides
|
||||||
|
are read from the annotation source folder: a return symlink points there, but
|
||||||
|
an independent fallback copy does not. After regeneration, rerun `giving-names`
|
||||||
|
to refresh copied return files.
|
||||||
|
|
||||||
|
`update-ods` reads these return JSON files; empty values become `NT` in the normal
|
||||||
|
per-question export. Its `--sum` option sums numeric values and skips nonnumeric
|
||||||
|
ones. Weighting and the final overall mark belong to the separate grading flow.
|
||||||
|
|
||||||
|
## Availability and later steps
|
||||||
|
|
||||||
|
- `giving-names` accepts `BGnot`, `Bnot`, or `Anot`. It selects the requested
|
||||||
|
source if `score.json` and either `Concat.jpg` or `info.json` exist,
|
||||||
|
otherwise it tries `Anot/CopieXX`. This also permits returns for an entirely
|
||||||
|
empty copy. It links the PDF only if `Concat_F.pdf` exists. The normal
|
||||||
|
`Bnot` reader produces a filtered `Concat_F.jpg`, and simple annotation produces
|
||||||
|
`Concat.jpg`; these paths do not guarantee a filtered PDF.
|
||||||
|
- Preparing returns removes disabled named outputs, and removes older named
|
||||||
|
outputs whose source is now absent, including broken symlinks.
|
||||||
|
- `add-final-score` writes to `FINAL_SCORE_OUTPUT_DIR/<evaluation>/`. It stamps
|
||||||
|
the overall mark from the configured ODS in red at the JPEG's upper right,
|
||||||
|
rounded down to one decimal place. It copies PDFs unchanged and does not export
|
||||||
|
either JSON file or the `answers/` directory. It does not add that mark to the
|
||||||
|
files inside `A Rendre`.
|
||||||
|
- The `clean` command retains return images (including individual answers), PDFs,
|
||||||
|
`score.json` and `info.json`, materializing
|
||||||
|
retained symlinks before deleting their sources.
|
||||||
|
|
||||||
|
## Implementation references
|
||||||
|
|
||||||
|
- [Naming and return links](../copienator/commands/giving_names.py)
|
||||||
|
- [Individual answer publication](../copienator/return_answers.py)
|
||||||
|
- [Grouped compilation, filtering and PDF pagination](../copienator/commands/reading_grouped_annotations.py)
|
||||||
|
- [Answer rendering](../copienator/commands/annotating.py)
|
||||||
|
- [Handwriting detection and per-copy compilation](../copienator/commands/reading_annotations.py)
|
||||||
|
- [Score and feedback actions](../copienator/annotation_actions.py)
|
||||||
|
- [Context, question and solution lookup](../copienator/utils.py)
|
||||||
|
- [ODS export](../copienator/commands/update_ods.py)
|
||||||
|
- [Final-mark stamping](../copienator/commands/add_final_score.py)
|
||||||
|
- [Cleanup retention](../copienator/commands/clean.py)
|
||||||
@@ -9,6 +9,7 @@ description = "Workflow assistant for preparing and correcting scanned exams"
|
|||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"numpy",
|
"numpy",
|
||||||
|
"opencv-python-headless>=4.8",
|
||||||
"pandas",
|
"pandas",
|
||||||
"matplotlib",
|
"matplotlib",
|
||||||
"Pillow",
|
"Pillow",
|
||||||
|
|||||||
Executable
+47
@@ -0,0 +1,47 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Applications opened from a file manager do not inherit variables exported by
|
||||||
|
# an interactive shell (for example GEMINI_API_KEY from ~/.zshrc). Re-enter
|
||||||
|
# through the user's shell once so its startup file can provide those values.
|
||||||
|
if [[ -z "${GEMINI_API_KEY:-}" && -z "${COPIENATOR_GUI_SHELL_LOADED:-}" && -x "${SHELL:-}" ]]; then
|
||||||
|
export COPIENATOR_GUI_SHELL_LOADED=1
|
||||||
|
exec "$SHELL" -ic 'exec "$@"' copienator-gui-shell "$0" "$@"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# With no arguments, choose the newest visible immediate subfolder by mtime.
|
||||||
|
if [[ $# -eq 0 ]]; then
|
||||||
|
newest=""
|
||||||
|
for candidate in "$PWD"/*/; do
|
||||||
|
[[ -d "$candidate" ]] || continue
|
||||||
|
folder="${candidate%/}"
|
||||||
|
folder="${folder##*/}"
|
||||||
|
case "$folder" in
|
||||||
|
copienator|copienator_gui|tests|OLD|__pycache__|build|dist|*.egg-info|venv|env|node_modules)
|
||||||
|
continue
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
if [[ -z "$newest" || "$candidate" -nt "$newest" ]]; then
|
||||||
|
newest="$candidate"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [[ -n "$newest" ]]; then
|
||||||
|
set -- "${newest%/}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Resolve the evaluation before changing to the project directory.
|
||||||
|
if [[ $# -gt 0 && "$1" != -* && "$1" != /* ]]; then
|
||||||
|
evaluation="$PWD/$1"
|
||||||
|
shift
|
||||||
|
set -- "$evaluation" "$@"
|
||||||
|
fi
|
||||||
|
repository="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
cd -- "$repository"
|
||||||
|
|
||||||
|
if [[ -x "$repository/.venv/bin/python" ]]; then
|
||||||
|
python_command="$repository/.venv/bin/python"
|
||||||
|
else
|
||||||
|
python_command="python3"
|
||||||
|
fi
|
||||||
|
exec "$python_command" -m copienator_gui "$@"
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from copienator.commands import add_final_score
|
||||||
|
|
||||||
|
|
||||||
|
class AddFinalScoreTests(unittest.TestCase):
|
||||||
|
def test_creates_complete_student_directory(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
source = root / "A Rendre" / "Student (01)"
|
||||||
|
answers = source / "answers"
|
||||||
|
answers.mkdir(parents=True)
|
||||||
|
Image.new("RGB", (300, 200), "white").save(source / "Student.jpg")
|
||||||
|
(source / "Student.pdf").write_bytes(b"pdf")
|
||||||
|
(source / "score.json").write_bytes(b'{"Ex 1": "4"}')
|
||||||
|
(source / "info.json").write_bytes(b'{"Ex 1": {}}')
|
||||||
|
(answers / "Ex 1.jpg").write_bytes(b"answer")
|
||||||
|
|
||||||
|
output = root / "output"
|
||||||
|
output.mkdir()
|
||||||
|
(output / "Student.jpg").write_bytes(b"legacy jpeg")
|
||||||
|
(output / "Student.pdf").write_bytes(b"legacy pdf")
|
||||||
|
stale_answers = output / "Student" / "answers"
|
||||||
|
stale_answers.mkdir(parents=True)
|
||||||
|
(stale_answers / "001 - Ex 1.jpg").write_bytes(b"stale")
|
||||||
|
scores = pd.DataFrame({0: ["Student"], 1: [12.39]})
|
||||||
|
histogram = root / "histogramme.pdf"
|
||||||
|
histogram.write_bytes(b"histogram")
|
||||||
|
with patch.object(add_final_score.pd, "read_excel", return_value=scores), \
|
||||||
|
patch.object(add_final_score, "HISTOGRAM_PATH", histogram), \
|
||||||
|
contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
add_final_score.process_images(root, output)
|
||||||
|
|
||||||
|
student = output / "Student"
|
||||||
|
self.assertEqual(
|
||||||
|
{path.name for path in student.iterdir()},
|
||||||
|
{"Student.jpg", "Student.pdf", "score.json", "info.json", "answers"},
|
||||||
|
)
|
||||||
|
self.assertEqual((student / "Student.pdf").read_bytes(), b"pdf")
|
||||||
|
self.assertEqual((student / "score.json").read_bytes(), b'{"Ex 1": "4"}')
|
||||||
|
self.assertEqual((student / "info.json").read_bytes(), b'{"Ex 1": {}}')
|
||||||
|
self.assertEqual(
|
||||||
|
(student / "answers" / "Ex 1.jpg").read_bytes(), b"answer"
|
||||||
|
)
|
||||||
|
self.assertFalse((output / "Student.jpg").exists())
|
||||||
|
self.assertFalse((output / "Student.pdf").exists())
|
||||||
|
self.assertFalse((student / "answers" / "001 - Ex 1.jpg").exists())
|
||||||
|
self.assertEqual((output / "histogramme.pdf").read_bytes(), b"histogram")
|
||||||
|
|
||||||
|
def test_omits_answers_directory_when_it_was_not_generated(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
source = root / "A Rendre" / "Student (01)"
|
||||||
|
source.mkdir(parents=True)
|
||||||
|
Image.new("RGB", (300, 200), "white").save(source / "Student.jpg")
|
||||||
|
(source / "Student.pdf").write_bytes(b"pdf")
|
||||||
|
(source / "score.json").write_text("{}")
|
||||||
|
(source / "info.json").write_text("{}")
|
||||||
|
|
||||||
|
output = root / "output"
|
||||||
|
scores = pd.DataFrame({0: ["Student"], 1: [10]})
|
||||||
|
with patch.object(add_final_score.pd, "read_excel", return_value=scores), \
|
||||||
|
contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
add_final_score.process_images(root, output)
|
||||||
|
|
||||||
|
self.assertFalse((output / "Student" / "answers").exists())
|
||||||
|
|
||||||
|
def test_missing_histogram_warns_without_discarding_student_outputs(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
source = root / "A Rendre" / "Student (01)"
|
||||||
|
source.mkdir(parents=True)
|
||||||
|
Image.new("RGB", (300, 200), "white").save(source / "Student.jpg")
|
||||||
|
scores = pd.DataFrame({0: ["Student"], 1: [10]})
|
||||||
|
output = root / "output"
|
||||||
|
messages = io.StringIO()
|
||||||
|
|
||||||
|
with patch.object(
|
||||||
|
add_final_score.pd, "read_excel", return_value=scores
|
||||||
|
), patch.object(
|
||||||
|
add_final_score, "HISTOGRAM_PATH", root / "missing.pdf"
|
||||||
|
), contextlib.redirect_stdout(messages):
|
||||||
|
add_final_score.process_images(root, output)
|
||||||
|
|
||||||
|
self.assertTrue((output / "Student" / "Student.jpg").is_file())
|
||||||
|
self.assertFalse((output / "histogramme.pdf").exists())
|
||||||
|
self.assertIn("Missing histogram", messages.getvalue())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,343 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import itertools
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from copienator import EvaluationWorkspace, atomic_write_json, configuration, read_json
|
||||||
|
from copienator.commands import annotating, giving_names
|
||||||
|
from copienator.commands import reading_grouped_annotations as reader
|
||||||
|
from copienator.return_answers import (
|
||||||
|
publish_answer_returns,
|
||||||
|
save_return_answer_options,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class AnswerReturnTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
self.source = self.root / "BGnot" / "Copie01"
|
||||||
|
self.source.mkdir(parents=True)
|
||||||
|
self.destination = self.root / "A Rendre" / "Student (01)"
|
||||||
|
self.destination.mkdir(parents=True)
|
||||||
|
(self.root / "labels").write_text("Ex 1\nEx 2\nEmpty\n")
|
||||||
|
atomic_write_json(self.source / "score.json", {"Ex 1": "4", "Ex 2": "2", "Empty": "0"})
|
||||||
|
atomic_write_json(self.source / "info.json", {
|
||||||
|
"Ex 1": {"present": True, "not_empty": True, "touched": False, "score": "4"},
|
||||||
|
"Ex 2": {"present": True, "not_empty": True, "touched": True, "score": "2"},
|
||||||
|
"Empty": {"present": True, "not_empty": False, "touched": False, "score": "0"},
|
||||||
|
})
|
||||||
|
for label in ("Ex 1", "Ex 2", "Empty"):
|
||||||
|
Image.new("RGB", (100, 30), "red").save(self.source / f"{label}.jpg")
|
||||||
|
for folder in ("Text2", "Sol2"):
|
||||||
|
(self.root / folder).mkdir()
|
||||||
|
for label in ("Ex 1", "Ex 2"):
|
||||||
|
(self.root / "Text2" / f"{label}.pdf").touch()
|
||||||
|
(self.root / "Sol2" / f"{label}.pdf").touch()
|
||||||
|
(self.root / "Text2" / "CTXT Ex 1 -> Ex 2.pdf").touch()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def supplement(path):
|
||||||
|
path = Path(path)
|
||||||
|
color = "blue" if path.name.startswith("CTXT") else "green" if path.parent.name == "Text2" else "yellow"
|
||||||
|
return Image.new("RGB", (100, 20), color), 0, 0
|
||||||
|
|
||||||
|
def test_all_supplement_combinations_always_include_annotated_answer(self):
|
||||||
|
for context, question, solution in itertools.product((False, True), repeat=3):
|
||||||
|
with self.subTest(context=context, question=question, solution=solution), patch.multiple(
|
||||||
|
configuration, RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=context, RETURN_ANSWERS_QUESTION=question,
|
||||||
|
RETURN_ANSWERS_SOLUTION=solution,
|
||||||
|
), patch.object(annotating, "make_base_image", side_effect=self.supplement):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
files = sorted((self.destination / "answers").glob("*.jpg"))
|
||||||
|
self.assertEqual([p.name for p in files], ["Ex 1.jpg", "Ex 2.jpg"])
|
||||||
|
with Image.open(files[0]) as image:
|
||||||
|
self.assertEqual(image.size, (100, 30 + 20 * sum((context, question, solution))))
|
||||||
|
colors = []
|
||||||
|
if context:
|
||||||
|
colors.append((0, 0, 255))
|
||||||
|
if question:
|
||||||
|
colors.append((0, 128, 0))
|
||||||
|
if solution:
|
||||||
|
colors.append((255, 255, 0))
|
||||||
|
colors.append((255, 0, 0))
|
||||||
|
for index, color in enumerate(colors):
|
||||||
|
pixel = image.getpixel((50, index * 20 + 10))
|
||||||
|
self.assertTrue(all(abs(a - b) < 10 for a, b in zip(pixel, color)))
|
||||||
|
self.assertEqual(read_json(self.destination / "info.json"), read_json(self.source / "info.json"))
|
||||||
|
|
||||||
|
def test_saved_reading_options_override_configuration_for_answers(self):
|
||||||
|
save_return_answer_options(
|
||||||
|
self.root,
|
||||||
|
context=True,
|
||||||
|
question=False,
|
||||||
|
solution=True,
|
||||||
|
)
|
||||||
|
with patch.multiple(
|
||||||
|
configuration,
|
||||||
|
RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=False,
|
||||||
|
RETURN_ANSWERS_QUESTION=True,
|
||||||
|
RETURN_ANSWERS_SOLUTION=False,
|
||||||
|
), patch.object(
|
||||||
|
annotating, "make_base_image", side_effect=self.supplement
|
||||||
|
):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
|
||||||
|
with Image.open(self.destination / "answers" / "Ex 1.jpg") as image:
|
||||||
|
self.assertEqual(image.size, (100, 70))
|
||||||
|
for y, color in (
|
||||||
|
(10, (0, 0, 255)),
|
||||||
|
(30, (255, 255, 0)),
|
||||||
|
(50, (255, 0, 0)),
|
||||||
|
):
|
||||||
|
self.assertTrue(
|
||||||
|
all(
|
||||||
|
abs(actual - expected) < 10
|
||||||
|
for actual, expected in zip(image.getpixel((50, y)), color)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_missing_supplement_is_optional_and_failure_preserves_old_answers(self):
|
||||||
|
with patch.multiple(configuration, RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=False, RETURN_ANSWERS_QUESTION=True,
|
||||||
|
RETURN_ANSWERS_SOLUTION=False), patch.object(
|
||||||
|
annotating, "make_base_image", side_effect=self.supplement
|
||||||
|
):
|
||||||
|
(self.root / "Text2" / "Ex 1.pdf").unlink()
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
path = self.destination / "answers" / "Ex 1.jpg"
|
||||||
|
with Image.open(path) as image:
|
||||||
|
self.assertEqual(image.size, (100, 30))
|
||||||
|
original = path.read_bytes()
|
||||||
|
(self.source / "Ex 2.jpg").unlink()
|
||||||
|
with self.assertRaises(FileNotFoundError):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
self.assertEqual(path.read_bytes(), original)
|
||||||
|
self.assertTrue((self.destination / "answers" / "Ex 2.jpg").exists())
|
||||||
|
|
||||||
|
def test_disabling_clears_individual_images_but_retains_info(self):
|
||||||
|
with patch.multiple(configuration, RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=False, RETURN_ANSWERS_QUESTION=False,
|
||||||
|
RETURN_ANSWERS_SOLUTION=False):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
with patch.object(configuration, "RETURN_ANSWERS_ENABLED", False):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
self.assertEqual(list((self.destination / "answers").iterdir()), [])
|
||||||
|
self.assertTrue(read_json(self.destination / "info.json")["Ex 2"]["touched"])
|
||||||
|
|
||||||
|
def test_missing_info_requires_recompilation_instead_of_guessing(self):
|
||||||
|
(self.source / "info.json").unlink()
|
||||||
|
(self.source / "Concat_F.pdf").touch()
|
||||||
|
with self.assertRaisesRegex(ValueError, "recompile"):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
|
||||||
|
def test_info_replaces_touched_and_tracks_manual_score_edits(self):
|
||||||
|
atomic_write_json(self.destination / "touched.json", {"obsolete": True})
|
||||||
|
atomic_write_json(self.source / "score.json", {"Ex 1": "3.5", "Ex 2": "2", "Empty": "0"})
|
||||||
|
with patch.object(configuration, "RETURN_ANSWERS_ENABLED", False):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
self.assertFalse((self.destination / "touched.json").exists())
|
||||||
|
self.assertEqual(read_json(self.destination / "info.json")["Ex 1"], {
|
||||||
|
"present": True, "not_empty": True, "touched": False, "score": "3.5"
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_invalid_info_preserves_previous_return(self):
|
||||||
|
atomic_write_json(self.destination / "info.json", {"previous": "keep"})
|
||||||
|
info = read_json(self.source / "info.json")
|
||||||
|
info["Empty"]["touched"] = True
|
||||||
|
atomic_write_json(self.source / "info.json", info)
|
||||||
|
with self.assertRaisesRegex(ValueError, "Invalid question information"):
|
||||||
|
publish_answer_returns(self.root, self.source, self.destination)
|
||||||
|
self.assertEqual(read_json(self.destination / "info.json"), {"previous": "keep"})
|
||||||
|
|
||||||
|
def test_publication_is_independent_of_full_jpeg_and_pdf_options(self):
|
||||||
|
workspace = EvaluationWorkspace(self.root)
|
||||||
|
workspace.copies_dir.mkdir()
|
||||||
|
atomic_write_json(workspace.copies_dir / "Copie01.json", {"name": "Student"})
|
||||||
|
(self.root / "names").write_text("Student\n")
|
||||||
|
with patch.multiple(configuration, RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=False, RETURN_ANSWERS_QUESTION=False,
|
||||||
|
RETURN_ANSWERS_SOLUTION=False, RETURN_JPEG_ENABLED=False,
|
||||||
|
RETURN_PDF_ENABLED=False), contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
self.assertEqual(giving_names.run(workspace, annotation_dir="BGnot"), 0)
|
||||||
|
self.assertEqual({p.name for p in self.destination.iterdir()}, {"answers", "score.json", "info.json"})
|
||||||
|
|
||||||
|
def test_update_uses_copy_id_from_renamed_folder_and_touches_only_answers(self):
|
||||||
|
workspace = EvaluationWorkspace(self.root)
|
||||||
|
workspace.copies_dir.mkdir()
|
||||||
|
atomic_write_json(workspace.copies_dir / "Copie01.json", {"name": "Student"})
|
||||||
|
renamed = self.destination.with_name("Nom modifié manuellement (01)")
|
||||||
|
self.destination.rename(renamed)
|
||||||
|
(renamed / "Nom personnalisé.jpg").write_bytes(b"keep-jpeg")
|
||||||
|
(renamed / "Nom personnalisé.pdf").write_bytes(b"keep-pdf")
|
||||||
|
(renamed / "score.json").write_bytes(b"keep-score")
|
||||||
|
(renamed / "info.json").write_bytes(b"keep-info")
|
||||||
|
answers = renamed / "answers"
|
||||||
|
answers.mkdir()
|
||||||
|
(answers / "obsolete.jpg").write_bytes(b"obsolete")
|
||||||
|
preserved = {
|
||||||
|
path.name: path.read_bytes()
|
||||||
|
for path in renamed.iterdir()
|
||||||
|
if path.is_file()
|
||||||
|
}
|
||||||
|
|
||||||
|
with patch.multiple(
|
||||||
|
configuration,
|
||||||
|
RETURN_ANSWERS_ENABLED=True,
|
||||||
|
RETURN_ANSWERS_CONTEXT=False,
|
||||||
|
RETURN_ANSWERS_QUESTION=False,
|
||||||
|
RETURN_ANSWERS_SOLUTION=False,
|
||||||
|
), contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
self.assertEqual(
|
||||||
|
giving_names.run(workspace, annotation_dir="BGnot", update=True),
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertFalse((workspace.return_dir / "Student (01)").exists())
|
||||||
|
self.assertEqual(
|
||||||
|
{
|
||||||
|
path.name: path.read_bytes()
|
||||||
|
for path in renamed.iterdir()
|
||||||
|
if path.is_file()
|
||||||
|
},
|
||||||
|
preserved,
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
sorted(path.name for path in answers.iterdir()),
|
||||||
|
["Ex 1.jpg", "Ex 2.jpg"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class CompiledMembershipTests(unittest.TestCase):
|
||||||
|
def test_update_score_preserves_file_and_manual_value_wins(self):
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
workspace = EvaluationWorkspace(Path(directory))
|
||||||
|
output = workspace.root / "BGnot" / "Copie01"
|
||||||
|
output.mkdir(parents=True)
|
||||||
|
original_score = b'{\n "Ex 1": "3.5"\n}\n'
|
||||||
|
(output / "score.json").write_bytes(original_score)
|
||||||
|
answer = workspace.root / "answer.pdf"
|
||||||
|
answer.touch()
|
||||||
|
data = {
|
||||||
|
"01": {
|
||||||
|
"Ex 1": {
|
||||||
|
"result": {"score": 1, "feedback": []},
|
||||||
|
"pdf_path": answer,
|
||||||
|
"coordinates": (0, 0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rendered_scores = []
|
||||||
|
|
||||||
|
def compose(base, label, result, *args, **kwargs):
|
||||||
|
rendered_scores.append(result["score"])
|
||||||
|
return Image.new("RGB", (100, 50), "white"), 0
|
||||||
|
|
||||||
|
with patch.object(
|
||||||
|
annotating, "make_base_image", return_value=(None, 0, 0)
|
||||||
|
), patch.object(
|
||||||
|
annotating, "compose_label_image", side_effect=compose
|
||||||
|
), patch.object(
|
||||||
|
reader, "get_extra_pdfs_as_images", return_value=[]
|
||||||
|
), patch.object(reader, "save_paginated_pdf"):
|
||||||
|
status, _ = reader.apply_actions_and_regenerate_grouped(
|
||||||
|
workspace,
|
||||||
|
data,
|
||||||
|
"01",
|
||||||
|
[{"label": "Ex 1", "type": "score", "value": "2"}],
|
||||||
|
{},
|
||||||
|
["Ex 1"],
|
||||||
|
update_score=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
self.assertEqual(rendered_scores, ["3.5"])
|
||||||
|
self.assertEqual((output / "score.json").read_bytes(), original_score)
|
||||||
|
self.assertEqual(read_json(output / "info.json")["Ex 1"]["score"], "3.5")
|
||||||
|
|
||||||
|
def test_membership_matches_actual_groups_and_saves_every_nonempty_answer(self):
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
root = Path(directory)
|
||||||
|
workspace = EvaluationWorkspace(root)
|
||||||
|
output = root / "BGnot" / "Copie01"
|
||||||
|
output.mkdir(parents=True)
|
||||||
|
answer = root / "answer.pdf"
|
||||||
|
answer.touch()
|
||||||
|
results = {
|
||||||
|
"Perfect": {"score": 4, "feedback": []},
|
||||||
|
"Low": {"score": 2, "feedback": []},
|
||||||
|
"Feedback": {"score": 4, "feedback": [{"text": "keep"}]},
|
||||||
|
"Deleted": {"score": 4, "feedback": [{"text": "delete", "to_delete": True}]},
|
||||||
|
"Handwriting": {"score": 4, "feedback": []},
|
||||||
|
"Empty": {"score": 0, "error": "empty-answer"},
|
||||||
|
}
|
||||||
|
data = {"01": {label: {"result": result, "pdf_path": answer, "coordinates": (0, 0)}
|
||||||
|
for label, result in results.items()}}
|
||||||
|
all_labels = [*results, "Absent"]
|
||||||
|
rendered = []
|
||||||
|
|
||||||
|
def compose(base, label, *args, **kwargs):
|
||||||
|
rendered.append(label)
|
||||||
|
return Image.new("RGB", (100, 50), "white"), 0
|
||||||
|
|
||||||
|
notes = {"Handwriting": {"img": Image.new("RGBA", (100, 50), "red"), "old_header_h": 0}}
|
||||||
|
with patch.object(annotating, "make_base_image", return_value=(None, 0, 0)), patch.object(
|
||||||
|
annotating, "compose_label_image", side_effect=compose
|
||||||
|
), patch.object(reader, "get_extra_pdfs_as_images", return_value=[]), patch.object(
|
||||||
|
reader, "save_paginated_pdf"
|
||||||
|
) as save_pdf:
|
||||||
|
status, _ = reader.apply_actions_and_regenerate_grouped(workspace, data, "01", [], notes, all_labels)
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
info = read_json(output / "info.json")
|
||||||
|
touched = {label: entry["touched"] for label, entry in info.items()}
|
||||||
|
self.assertEqual({label for label, value in touched.items() if value}, {"Low", "Feedback", "Handwriting"})
|
||||||
|
self.assertEqual(len(save_pdf.call_args.args[0]), sum(touched.values()))
|
||||||
|
self.assertEqual({label for label, entry in info.items() if entry["present"] and entry["not_empty"]}, set(results) - {"Empty"})
|
||||||
|
self.assertEqual(info["Empty"], {"present": True, "not_empty": False, "touched": False, "score": "0"})
|
||||||
|
self.assertEqual(info["Absent"], {"present": False, "not_empty": False, "touched": False, "score": ""})
|
||||||
|
self.assertEqual(info["Perfect"], {"present": True, "not_empty": True, "touched": False, "score": "4"})
|
||||||
|
self.assertNotIn("Empty", rendered)
|
||||||
|
for label in rendered:
|
||||||
|
self.assertTrue((output / f"{label}.jpg").is_file())
|
||||||
|
# Selective redo keeps unselected saved answers in the PDF,
|
||||||
|
# including previously perfect answers; touched must follow it.
|
||||||
|
status, _ = reader.apply_actions_and_regenerate_grouped(
|
||||||
|
workspace, data, "01", [], {}, all_labels, selected_labels={"Low"}
|
||||||
|
)
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
self.assertTrue(read_json(output / "info.json")["Perfect"]["touched"])
|
||||||
|
self.assertFalse(read_json(output / "info.json")["Empty"]["not_empty"])
|
||||||
|
|
||||||
|
def test_all_empty_removes_stale_concat_and_records_false(self):
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
workspace = EvaluationWorkspace(Path(directory))
|
||||||
|
output = workspace.root / "BGnot" / "Copie01"
|
||||||
|
output.mkdir(parents=True)
|
||||||
|
for name in ("Concat.jpg", "Concat_F.pdf"):
|
||||||
|
(output / name).write_bytes(b"stale")
|
||||||
|
atomic_write_json(output / "touched.json", {"Empty": True})
|
||||||
|
atomic_write_json(output / "answer_labels.json", ["Empty"])
|
||||||
|
status, _ = reader.apply_actions_and_regenerate_grouped(
|
||||||
|
workspace, {"01": {"Empty": {"result": {"score": 0, "error": "empty-answer"}}}},
|
||||||
|
"01", [], {}, ["Empty"]
|
||||||
|
)
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
self.assertEqual(read_json(output / "info.json"), {
|
||||||
|
"Empty": {"present": True, "not_empty": False, "touched": False, "score": "0"}
|
||||||
|
})
|
||||||
|
self.assertFalse((output / "Concat.jpg").exists())
|
||||||
|
self.assertFalse((output / "Concat_F.pdf").exists())
|
||||||
|
self.assertFalse((output / "touched.json").exists())
|
||||||
|
self.assertFalse((output / "answer_labels.json").exists())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
import queue
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from copienator_gui.batch_monitor import BatchMonitor, CHECK_INTERVAL_MS
|
||||||
|
from copienator_gui.notifications import notify_desktop
|
||||||
|
|
||||||
|
|
||||||
|
class BatchMonitorTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.scheduler = Mock()
|
||||||
|
self.scheduler.after.side_effect = lambda delay, callback: (delay, callback)
|
||||||
|
self.ready = Mock()
|
||||||
|
self.status = Mock()
|
||||||
|
self.monitor = BatchMonitor(self.scheduler, self.ready, self.status, Mock())
|
||||||
|
self.runner = Mock()
|
||||||
|
self.runner.events = queue.Queue()
|
||||||
|
self.factory = patch("copienator_gui.batch_monitor.ProcessRunner", return_value=self.runner)
|
||||||
|
self.factory.start()
|
||||||
|
self.addCleanup(self.factory.stop)
|
||||||
|
|
||||||
|
def start(self):
|
||||||
|
self.monitor.start(["check"], "/tmp", {}, None)
|
||||||
|
|
||||||
|
def test_checks_immediately_retries_in_five_minutes_and_notifies_once(self):
|
||||||
|
self.start()
|
||||||
|
self.runner.start.assert_called_once()
|
||||||
|
self.start()
|
||||||
|
self.runner.start.assert_called_once()
|
||||||
|
self.runner.events.put(("finished", (4, False)))
|
||||||
|
self.monitor._poll()
|
||||||
|
self.assertEqual(self.monitor.timer[0], CHECK_INTERVAL_MS)
|
||||||
|
self.assertEqual(CHECK_INTERVAL_MS, 300000)
|
||||||
|
self.ready.assert_not_called()
|
||||||
|
self.monitor.timer[1]()
|
||||||
|
self.assertEqual(self.runner.start.call_count, 2)
|
||||||
|
self.runner.events.put(("finished", (0, False)))
|
||||||
|
self.monitor._poll()
|
||||||
|
self.assertFalse(self.monitor.active)
|
||||||
|
self.assertIsNone(self.monitor.timer)
|
||||||
|
self.ready.assert_called_once()
|
||||||
|
self.monitor._poll()
|
||||||
|
self.ready.assert_called_once()
|
||||||
|
|
||||||
|
def test_stop_cancels_timer_and_running_check_without_notification(self):
|
||||||
|
self.start()
|
||||||
|
timer = self.monitor.timer
|
||||||
|
self.monitor.stop()
|
||||||
|
self.scheduler.after_cancel.assert_called_once_with(timer)
|
||||||
|
self.runner.force_stop.assert_called_once()
|
||||||
|
self.runner.events.put(("finished", (0, False)))
|
||||||
|
self.monitor._poll()
|
||||||
|
self.monitor._check()
|
||||||
|
self.ready.assert_not_called()
|
||||||
|
self.runner.start.assert_called_once()
|
||||||
|
|
||||||
|
def test_start_failure_retries_without_reporting_readiness(self):
|
||||||
|
self.runner.start.side_effect = OSError("unavailable")
|
||||||
|
self.start()
|
||||||
|
self.assertEqual(self.monitor.timer[0], CHECK_INTERVAL_MS)
|
||||||
|
self.ready.assert_not_called()
|
||||||
|
|
||||||
|
def test_linux_notification_passes_text_as_arguments(self):
|
||||||
|
with patch("copienator_gui.notifications.sys.platform", "linux"), patch(
|
||||||
|
"copienator_gui.notifications.find_executable", return_value="/usr/bin/notify-send"
|
||||||
|
), patch("copienator_gui.notifications.subprocess.Popen") as launch:
|
||||||
|
notify_desktop("Copienator", "Interro02 : résultats prêts")
|
||||||
|
self.assertEqual(launch.call_args.args[0], ["/usr/bin/notify-send", "--app-name=Copienator",
|
||||||
|
"--", "Copienator", "Interro02 : résultats prêts"])
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import Mock, call, patch
|
||||||
|
|
||||||
|
from copienator import CliError, EvaluationWorkspace, ExitCode, atomic_write_json
|
||||||
|
from copienator.commands.batch_status import check_evaluation_jobs, main
|
||||||
|
|
||||||
|
|
||||||
|
class BatchReadinessTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.workspace = EvaluationWorkspace(Path(self.temp.name))
|
||||||
|
self.client = Mock()
|
||||||
|
|
||||||
|
def manifest(self, jobs):
|
||||||
|
atomic_write_json(self.workspace.batch_jobs_file, {"jobs": jobs})
|
||||||
|
|
||||||
|
def test_only_recorded_jobs_are_checked_and_all_must_have_results(self):
|
||||||
|
self.manifest({"flash": {"name": "batches/flash"}, "pro": {"name": "batches/pro"}})
|
||||||
|
succeeded = SimpleNamespace(state="JOB_STATE_SUCCEEDED",
|
||||||
|
dest=SimpleNamespace(file_name="files/result"))
|
||||||
|
for state in ("JOB_STATE_PENDING", "JOB_STATE_RUNNING", "JOB_STATE_FAILED",
|
||||||
|
"JOB_STATE_CANCELLED", "JOB_STATE_EXPIRED", "UNKNOWN", "JOB_STATE_SUCCEEDED"):
|
||||||
|
with self.subTest(state=state):
|
||||||
|
self.client.reset_mock()
|
||||||
|
self.client.batches.get.side_effect = [succeeded, SimpleNamespace(
|
||||||
|
state=SimpleNamespace(name=state), dest=SimpleNamespace(file_name="files/pro"))]
|
||||||
|
result = check_evaluation_jobs(self.workspace, client=self.client)
|
||||||
|
self.assertEqual(result, ExitCode.SUCCESS if state == "JOB_STATE_SUCCEEDED" else ExitCode.PARTIAL)
|
||||||
|
self.assertEqual(self.client.batches.get.call_args_list,
|
||||||
|
[call(name="batches/flash"), call(name="batches/pro")])
|
||||||
|
self.client.batches.list.assert_not_called()
|
||||||
|
self.client.files.download.assert_not_called()
|
||||||
|
self.client.batches.get.side_effect = [succeeded, SimpleNamespace(
|
||||||
|
state="JOB_STATE_SUCCEEDED", dest=None)]
|
||||||
|
self.assertEqual(check_evaluation_jobs(self.workspace, client=self.client), ExitCode.PARTIAL)
|
||||||
|
|
||||||
|
def test_missing_empty_and_invalid_manifest_cannot_report_success(self):
|
||||||
|
self.assertEqual(check_evaluation_jobs(self.workspace, client=self.client), ExitCode.PARTIAL)
|
||||||
|
self.manifest({})
|
||||||
|
self.assertEqual(check_evaluation_jobs(self.workspace, client=self.client), ExitCode.PARTIAL)
|
||||||
|
for jobs in ([], {"flash": {}}, {"flash": {"name": ""}}, {"flash": None}):
|
||||||
|
self.manifest(jobs)
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
check_evaluation_jobs(self.workspace, client=self.client)
|
||||||
|
self.client.batches.get.assert_not_called()
|
||||||
|
|
||||||
|
def test_cli_returns_readiness_code_for_selected_evaluation(self):
|
||||||
|
with patch("copienator.commands.batch_status.check_evaluation_jobs", return_value=ExitCode.PARTIAL) as check:
|
||||||
|
self.assertEqual(main(["--evaluation", str(self.workspace.root)]), ExitCode.PARTIAL)
|
||||||
|
self.assertEqual(check.call_args.args[0].root, self.workspace.root)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
import unittest
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.crop_blank_margins import apply_bounds
|
||||||
|
|
||||||
|
|
||||||
|
class CropBlankMarginsTests(unittest.TestCase):
|
||||||
|
def test_cropbox_coordinates_with_rotation_and_existing_crop(self):
|
||||||
|
for rotation in (0, 90, 180, 270):
|
||||||
|
with self.subTest(rotation=rotation), pymupdf.open() as doc:
|
||||||
|
page = doc.new_page(width=600, height=800)
|
||||||
|
page.set_cropbox(pymupdf.Rect(30, 40, 570, 760))
|
||||||
|
page.set_rotation(rotation)
|
||||||
|
before = page.rect
|
||||||
|
page.insert_text((80, 220), 'Visible content', fontsize=20)
|
||||||
|
pix = page.get_pixmap()
|
||||||
|
original = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
|
||||||
|
apply_bounds(page, .2, .8)
|
||||||
|
self.assertAlmostEqual(page.rect.width, before.width)
|
||||||
|
self.assertAlmostEqual(page.rect.height, before.height*.6, places=3)
|
||||||
|
self.assertEqual(page.rotation, rotation)
|
||||||
|
pix = page.get_pixmap()
|
||||||
|
cropped = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
|
||||||
|
np.testing.assert_array_equal(cropped, original[int(before.height*.2):int(before.height*.8)])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.crop_exercise_bottoms import process_exercise_pdf
|
||||||
|
|
||||||
|
|
||||||
|
class CropExerciseBottomsTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
self.review = self.root / "review"
|
||||||
|
|
||||||
|
def make_pdf(self, name: str, height: float, text_y: float, footer=True) -> Path:
|
||||||
|
path = self.root / name
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
page = document.new_page(width=600, height=height)
|
||||||
|
page.insert_text((80, text_y), "student answer", fontsize=16)
|
||||||
|
if footer:
|
||||||
|
page.insert_text((20, height - 3), "Ex 2", fontsize=10)
|
||||||
|
document.save(path)
|
||||||
|
return path
|
||||||
|
|
||||||
|
def test_short_page_is_skipped(self):
|
||||||
|
source = self.make_pdf("short.pdf", 200, 100)
|
||||||
|
destination = self.root / "out" / source.name
|
||||||
|
records = process_exercise_pdf(
|
||||||
|
source, destination, self.review, 800, dpi=150
|
||||||
|
)
|
||||||
|
self.assertEqual(records[0]["status"], "skipped-short")
|
||||||
|
self.assertFalse(destination.exists())
|
||||||
|
|
||||||
|
def test_footer_fragment_is_ignored_for_a_large_bottom_crop(self):
|
||||||
|
source = self.make_pdf("large.pdf", 400, 100)
|
||||||
|
destination = self.root / "out" / source.name
|
||||||
|
records = process_exercise_pdf(
|
||||||
|
source, destination, self.review, 800, dpi=150
|
||||||
|
)
|
||||||
|
self.assertEqual(records[0]["status"], "cropped")
|
||||||
|
self.assertGreaterEqual(records[0]["bottom_removed_lines"], 4)
|
||||||
|
with pymupdf.open(destination) as result:
|
||||||
|
self.assertLess(result[0].rect.height, 250)
|
||||||
|
self.assertAlmostEqual(result[0].rect.width, 600)
|
||||||
|
|
||||||
|
def test_crop_smaller_than_four_lines_is_not_written(self):
|
||||||
|
source = self.make_pdf("small.pdf", 400, 330, footer=False)
|
||||||
|
destination = self.root / "out" / source.name
|
||||||
|
records = process_exercise_pdf(
|
||||||
|
source, destination, self.review, 800, dpi=150
|
||||||
|
)
|
||||||
|
self.assertEqual(records[0]["status"], "unchanged-small-crop")
|
||||||
|
self.assertFalse(destination.exists())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.cli import CliError
|
||||||
|
from copienator.commands import crop_exercise_bottoms
|
||||||
|
from copienator.commands.clean import apply_cleanup, build_cleanup_plan
|
||||||
|
from copienator.dispatcher import main
|
||||||
|
from copienator.workspace import EvaluationWorkspace
|
||||||
|
from copienator_gui.workflow import build_workflow
|
||||||
|
|
||||||
|
|
||||||
|
class CropExerciseBottomsCommandTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.workspace = EvaluationWorkspace(Path(self.temp.name) / "Évaluation")
|
||||||
|
self.workspace.copies_dir.mkdir(parents=True)
|
||||||
|
self.copy_pdf = self.workspace.copies_dir / "Copie01.pdf"
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
document.new_page(width=600, height=800)
|
||||||
|
document.save(self.copy_pdf)
|
||||||
|
self.answers = self.workspace.copies_dir / "Copie01"
|
||||||
|
self.answers.mkdir()
|
||||||
|
self.large = self.answers / "Ex 1.pdf"
|
||||||
|
self.short = self.answers / "Ex 2.pdf"
|
||||||
|
self._make_answer(self.large, 400, 100, footer=True)
|
||||||
|
self._make_answer(self.short, 200, 100, footer=False)
|
||||||
|
self.large_original = self.large.read_bytes()
|
||||||
|
self.short_original = self.short.read_bytes()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _make_answer(path: Path, height: float, text_y: float, *, footer: bool) -> None:
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
page = document.new_page(width=600, height=height)
|
||||||
|
page.insert_text((80, text_y), "student answer", fontsize=16)
|
||||||
|
if footer:
|
||||||
|
page.insert_text((20, height - 3), "Ex 2", fontsize=10)
|
||||||
|
document.save(path)
|
||||||
|
|
||||||
|
def test_dispatcher_replaces_only_changed_answers_and_backs_them_up(self):
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()) as log:
|
||||||
|
self.assertEqual(
|
||||||
|
main(["crop-answer-bottoms", str(self.workspace.root), "--workers", "2"]),
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
self.assertIn("1 PDF remplacé", log.getvalue())
|
||||||
|
self.assertIn("1 exercice(s) rogné(s)", log.getvalue())
|
||||||
|
self.assertIn("Rognage moyen des exercices modifiés", log.getvalue())
|
||||||
|
with pymupdf.open(self.large) as cropped:
|
||||||
|
self.assertLess(cropped[0].rect.height, 250)
|
||||||
|
self.assertEqual(self.short.read_bytes(), self.short_original)
|
||||||
|
backups = list(
|
||||||
|
self.workspace.runs_dir.glob(
|
||||||
|
"crop-exercise-bottoms-*/Copies/Copie01/Ex 1.pdf"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertEqual(len(backups), 1)
|
||||||
|
self.assertEqual(backups[0].read_bytes(), self.large_original)
|
||||||
|
self.assertFalse(
|
||||||
|
list(
|
||||||
|
self.workspace.runs_dir.glob(
|
||||||
|
"crop-exercise-bottoms-*/Copies/Copie01/Ex 2.pdf"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.assertTrue((backups[0].parents[2] / "report.json").is_file())
|
||||||
|
|
||||||
|
def test_crop_statistics_average_percentages_by_exercise(self):
|
||||||
|
records = [
|
||||||
|
{
|
||||||
|
"file": "Copies/Copie01/Ex 1.pdf",
|
||||||
|
"height_lines": 10.0,
|
||||||
|
"bottom_removed_lines": 4.0,
|
||||||
|
"status": "cropped",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "Copies/Copie01/Ex 1.pdf",
|
||||||
|
"height_lines": 10.0,
|
||||||
|
"bottom_removed_lines": 0.0,
|
||||||
|
"status": "skipped-short",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "Copies/Copie01/Ex 2.pdf",
|
||||||
|
"height_lines": 10.0,
|
||||||
|
"bottom_removed_lines": 5.0,
|
||||||
|
"status": "cropped",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "Copies/Copie01/Ex 3.pdf",
|
||||||
|
"height_lines": 10.0,
|
||||||
|
"bottom_removed_lines": 0.0,
|
||||||
|
"status": "unchanged-small-crop",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
count, average = crop_exercise_bottoms.crop_statistics(records)
|
||||||
|
self.assertEqual(count, 2)
|
||||||
|
self.assertAlmostEqual(average, 35.0)
|
||||||
|
|
||||||
|
def test_detection_failure_does_not_publish_an_earlier_result(self):
|
||||||
|
broken = self.answers / "Ex 3.pdf"
|
||||||
|
broken.write_bytes(b"not a PDF")
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()), self.assertRaises(Exception):
|
||||||
|
crop_exercise_bottoms.run(
|
||||||
|
self.workspace, self.workspace.root, workers=1
|
||||||
|
)
|
||||||
|
self.assertEqual(self.large.read_bytes(), self.large_original)
|
||||||
|
self.assertEqual(self.short.read_bytes(), self.short_original)
|
||||||
|
self.assertEqual(broken.read_bytes(), b"not a PDF")
|
||||||
|
self.assertFalse(self.workspace.runs_dir.exists())
|
||||||
|
|
||||||
|
def test_invalid_target_and_worker_count_are_rejected(self):
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
crop_exercise_bottoms.run(self.workspace, self.large, workers=1)
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
crop_exercise_bottoms.run(self.workspace, self.workspace.root, workers=0)
|
||||||
|
|
||||||
|
def test_archiving_removes_the_retained_originals_and_report(self):
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
crop_exercise_bottoms.run(
|
||||||
|
self.workspace, self.workspace.root, workers=1
|
||||||
|
)
|
||||||
|
run_dirs = list(
|
||||||
|
self.workspace.runs_dir.glob("crop-exercise-bottoms-*")
|
||||||
|
)
|
||||||
|
self.assertEqual(len(run_dirs), 1)
|
||||||
|
self.workspace.correction_file.write_text("{}")
|
||||||
|
student = self.workspace.return_dir / "Student"
|
||||||
|
student.mkdir(parents=True)
|
||||||
|
(student / "answer.jpg").write_bytes(b"return image")
|
||||||
|
(student / "score.json").write_text("{}")
|
||||||
|
plan = build_cleanup_plan(self.workspace)
|
||||||
|
self.assertTrue(
|
||||||
|
any(path.name == "report.json" for path in plan.deleted_files)
|
||||||
|
)
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
apply_cleanup(self.workspace, plan)
|
||||||
|
self.assertFalse(self.workspace.runs_dir.exists())
|
||||||
|
|
||||||
|
def test_optional_step_follows_splitting_and_precedes_grouping(self):
|
||||||
|
steps = build_workflow(False)
|
||||||
|
index = next(
|
||||||
|
i for i, step in enumerate(steps) if step.id == "crop_exercise_bottoms"
|
||||||
|
)
|
||||||
|
self.assertEqual(steps[index - 1].id, "splitting")
|
||||||
|
self.assertEqual(steps[index + 1].id, "grouping")
|
||||||
|
self.assertTrue(steps[index].optional)
|
||||||
|
self.assertTrue(steps[index].auto_start_first_visit)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.cli import CliError
|
||||||
|
from copienator.commands import crop_margins
|
||||||
|
from copienator.commands.clean import build_cleanup_plan, apply_cleanup
|
||||||
|
from copienator.dispatcher import main
|
||||||
|
from copienator.workspace import EvaluationWorkspace
|
||||||
|
from copienator_gui.workflow import build_workflow
|
||||||
|
|
||||||
|
|
||||||
|
class CropMarginsCommandTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.workspace = EvaluationWorkspace(Path(self.temp.name)/"Évaluation")
|
||||||
|
self.workspace.copies_dir.mkdir(parents=True)
|
||||||
|
self.source = self.workspace.copies_dir/"Copie01.pdf"
|
||||||
|
with pymupdf.open() as doc:
|
||||||
|
page = doc.new_page(width=300, height=420)
|
||||||
|
page.insert_text((50, 180), "answer = 42", fontsize=15)
|
||||||
|
doc.new_page(width=300, height=420)
|
||||||
|
doc.save(self.source)
|
||||||
|
self.original = self.source.read_bytes()
|
||||||
|
|
||||||
|
def test_dispatcher_replaces_copies_and_keeps_recoverable_original(self):
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()) as log:
|
||||||
|
self.assertEqual(main(["crop-margins", str(self.workspace.root)]), 0)
|
||||||
|
self.assertIn("Page 2/2", log.getvalue())
|
||||||
|
self.assertIn("1/2 pages rognées", log.getvalue())
|
||||||
|
self.assertIn("Rognage moyen des pages modifiées", log.getvalue())
|
||||||
|
self.assertIn("page(s) rognée(s) de plus de 30 %", log.getvalue())
|
||||||
|
with pymupdf.open(self.source) as result:
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertLess(result[0].rect.height, 150)
|
||||||
|
self.assertIn("answer = 42", result[0].get_text())
|
||||||
|
self.assertEqual(result[1].rect.height, 420)
|
||||||
|
for page in result:
|
||||||
|
self.assertEqual(page.rect.width, 300)
|
||||||
|
page.get_pixmap()
|
||||||
|
backups = list(self.workspace.runs_dir.glob("crop-margins-*/Copies/Copie01.pdf"))
|
||||||
|
self.assertEqual(len(backups), 1)
|
||||||
|
self.assertEqual(backups[0].read_bytes(), self.original)
|
||||||
|
self.assertTrue((backups[0].parent.parent/"report.json").is_file())
|
||||||
|
|
||||||
|
def test_crop_statistics_use_only_modified_pages(self):
|
||||||
|
records = [
|
||||||
|
{
|
||||||
|
"top_removed_mm": 10.0,
|
||||||
|
"bottom_removed_mm": 20.0,
|
||||||
|
"original_cropbox": [0, 0, 200, 300],
|
||||||
|
"rotation": 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"top_removed_mm": 40.0,
|
||||||
|
"bottom_removed_mm": 0.0,
|
||||||
|
"original_cropbox": [0, 0, 200, 300],
|
||||||
|
"rotation": 90,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"top_removed_mm": 0.0,
|
||||||
|
"bottom_removed_mm": 0.0,
|
||||||
|
"original_cropbox": [0, 0, 200, 300],
|
||||||
|
"rotation": 0,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
count, average, over_thirty = crop_margins.crop_statistics(records)
|
||||||
|
self.assertEqual(count, 2)
|
||||||
|
self.assertAlmostEqual(average, (28.3465 + 56.6929) / 2, places=3)
|
||||||
|
self.assertEqual(over_thirty, 1)
|
||||||
|
|
||||||
|
def test_failure_or_interruption_never_publishes_partial_batch(self):
|
||||||
|
second = self.workspace.copies_dir/"Copie02.pdf"
|
||||||
|
second.write_bytes(self.original)
|
||||||
|
real = crop_margins.process_pdf
|
||||||
|
for error in (OSError("bad scan"), KeyboardInterrupt()):
|
||||||
|
with self.subTest(error=type(error).__name__):
|
||||||
|
def process(source, *args, **kwargs):
|
||||||
|
if source == second:
|
||||||
|
raise error
|
||||||
|
return real(source, *args, **kwargs)
|
||||||
|
with patch.object(crop_margins, "process_pdf", side_effect=process):
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
with self.assertRaises(type(error)):
|
||||||
|
crop_margins.run(self.workspace, self.workspace.root, workers=1)
|
||||||
|
self.assertEqual(self.source.read_bytes(), self.original)
|
||||||
|
self.assertEqual(second.read_bytes(), self.original)
|
||||||
|
|
||||||
|
def test_existing_coordinates_are_not_silently_invalidated(self):
|
||||||
|
self.source.with_suffix(".json").write_text('{"list": []}')
|
||||||
|
with self.assertRaisesRegex(CliError, "coordonnées"):
|
||||||
|
crop_margins.run(self.workspace, self.workspace.root)
|
||||||
|
self.assertEqual(self.source.read_bytes(), self.original)
|
||||||
|
|
||||||
|
def test_archiving_removes_crop_backups_but_keeps_processed_pdf_and_logs(self):
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
crop_margins.run(self.workspace, self.workspace.root)
|
||||||
|
processed = self.source.read_bytes()
|
||||||
|
self.workspace.correction_file.write_text('{}')
|
||||||
|
student = self.workspace.return_dir/"Student"
|
||||||
|
student.mkdir(parents=True)
|
||||||
|
(student/"answer.jpg").write_bytes(b"return image")
|
||||||
|
(student/"score.json").write_text('{}')
|
||||||
|
self.workspace.logs_dir.mkdir(parents=True)
|
||||||
|
log = self.workspace.logs_dir/"crop_blank_margins.log"
|
||||||
|
log.write_text("Completed crop")
|
||||||
|
backups = list(self.workspace.runs_dir.glob("crop-margins-*/Copies/*.pdf"))
|
||||||
|
reports = list(self.workspace.runs_dir.glob("crop-margins-*/report.json"))
|
||||||
|
self.assertTrue(backups)
|
||||||
|
self.assertTrue(reports)
|
||||||
|
plan = build_cleanup_plan(self.workspace)
|
||||||
|
for path in backups+reports:
|
||||||
|
self.assertIn(path, plan.deleted_files)
|
||||||
|
self.assertIn(self.source, plan.kept_files)
|
||||||
|
self.assertIn(log, plan.kept_files)
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
apply_cleanup(self.workspace, plan)
|
||||||
|
self.assertFalse(self.workspace.runs_dir.exists())
|
||||||
|
self.assertEqual(self.source.read_bytes(), processed)
|
||||||
|
self.assertEqual(log.read_text(), "Completed crop")
|
||||||
|
self.assertTrue((student/"answer.jpg").exists())
|
||||||
|
self.assertTrue((student/"score.json").exists())
|
||||||
|
|
||||||
|
def test_single_copy_selection_cannot_target_original_scans(self):
|
||||||
|
self.assertEqual(crop_margins.selected_files(self.workspace, self.source), [self.source])
|
||||||
|
original = self.workspace.root/"Original.pdf"
|
||||||
|
original.write_bytes(self.original)
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
crop_margins.selected_files(self.workspace, original)
|
||||||
|
|
||||||
|
def test_parallel_workers_match_serial_results_in_copy_and_page_order(self):
|
||||||
|
second = self.workspace.copies_dir/"Copie02.pdf"
|
||||||
|
second.write_bytes(self.original)
|
||||||
|
serial, parallel = self.workspace.root/"serial", self.workspace.root/"parallel"
|
||||||
|
serial.mkdir()
|
||||||
|
parallel.mkdir()
|
||||||
|
files = [self.source, second]
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
expected = crop_margins.process_copies(files, serial, 1)
|
||||||
|
actual = crop_margins.process_copies(files, parallel, 2)
|
||||||
|
self.assertEqual(actual, expected)
|
||||||
|
self.assertEqual([(r['file'],r['page']) for r in actual],
|
||||||
|
[(p.name,i) for p in files for i in (1,2)])
|
||||||
|
for source in files:
|
||||||
|
with pymupdf.open(serial/source.name) as a, pymupdf.open(parallel/source.name) as b:
|
||||||
|
self.assertEqual([list(p.cropbox) for p in a], [list(p.cropbox) for p in b])
|
||||||
|
|
||||||
|
def test_worker_failure_does_not_replace_any_copy(self):
|
||||||
|
broken = self.workspace.copies_dir/"Copie02.pdf"
|
||||||
|
broken.write_bytes(b"not a PDF")
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
with self.assertRaises(Exception):
|
||||||
|
crop_margins.run(self.workspace, self.workspace.root, workers=2)
|
||||||
|
self.assertEqual(self.source.read_bytes(), self.original)
|
||||||
|
self.assertEqual(broken.read_bytes(), b"not a PDF")
|
||||||
|
self.assertFalse(list(self.workspace.root.glob(".Copies.*.files.tmp")))
|
||||||
|
|
||||||
|
def test_invalid_worker_count_is_rejected_before_processing(self):
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
crop_margins.run(self.workspace, self.workspace.root, workers=0)
|
||||||
|
self.assertEqual(self.source.read_bytes(), self.original)
|
||||||
|
|
||||||
|
@unittest.skipIf(os.name == "nt", "SIGINT subprocess check uses Unix signals")
|
||||||
|
def test_interrupt_stops_parallel_workers_before_removing_staging(self):
|
||||||
|
import select
|
||||||
|
with pymupdf.open() as doc:
|
||||||
|
for _ in range(30):
|
||||||
|
page = doc.new_page(width=595, height=842)
|
||||||
|
page.insert_text((100,400), "answer = 42", fontsize=20)
|
||||||
|
data = doc.tobytes()
|
||||||
|
self.source.write_bytes(data)
|
||||||
|
second = self.workspace.copies_dir/"Copie02.pdf"
|
||||||
|
second.write_bytes(data)
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable,"-u","-m","copienator","crop-margins",
|
||||||
|
str(self.workspace.root),"--workers","2"],
|
||||||
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
|
||||||
|
cwd=Path(__file__).resolve().parents[1])
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
ready, _, _ = select.select([proc.stdout], [], [], 20)
|
||||||
|
self.assertTrue(ready, "No progress from parallel crop command")
|
||||||
|
line = proc.stdout.readline()
|
||||||
|
self.assertTrue(line, "Crop command exited before processing a page")
|
||||||
|
if "Page " in line:
|
||||||
|
break
|
||||||
|
proc.send_signal(signal.SIGINT)
|
||||||
|
output, _ = proc.communicate(timeout=20)
|
||||||
|
self.assertEqual(proc.returncode, 130, output)
|
||||||
|
finally:
|
||||||
|
if proc.poll() is None:
|
||||||
|
proc.kill()
|
||||||
|
proc.communicate()
|
||||||
|
self.assertEqual(self.source.read_bytes(), data)
|
||||||
|
self.assertEqual(second.read_bytes(), data)
|
||||||
|
self.assertFalse(list(self.workspace.root.glob(".Copies.*.files.tmp")))
|
||||||
|
|
||||||
|
def test_optional_step_sits_between_page_split_and_label_crop(self):
|
||||||
|
steps = build_workflow(False)
|
||||||
|
index = next(i for i, step in enumerate(steps) if step.id == "crop_blank_margins")
|
||||||
|
self.assertEqual(steps[index-1].id, "page_splitter")
|
||||||
|
self.assertEqual(steps[index+1].id, "cutleft")
|
||||||
|
self.assertTrue(steps[index].optional)
|
||||||
|
self.assertTrue(steps[index].auto_start_first_visit)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
import copy
|
||||||
|
import json
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from copienator import EvaluationWorkspace, atomic_write_json
|
||||||
|
from copienator.annotation_data import GroupCoordinates, _scaled_result
|
||||||
|
from copienator.annotation_actions import apply_checkbox_actions
|
||||||
|
from copienator.commands import annotating, correction
|
||||||
|
from copienator.feedback_boxes import valid_feedback_box
|
||||||
|
|
||||||
|
|
||||||
|
class FeedbackBoxTests(unittest.TestCase):
|
||||||
|
def test_checkbox_for_promoted_feedback_deletes_the_correct_comment(self):
|
||||||
|
feedback = [{"text": "Invalid local", "box_2d": [753, 680, 287, 946]},
|
||||||
|
{"text": "Existing global", "box_2d": None}]
|
||||||
|
apply_checkbox_actions({"Ex 5": {"result": {"feedback": feedback}}},
|
||||||
|
[{"label": "Ex 5", "type": "del_global", "index": 0}], lambda _: None)
|
||||||
|
self.assertTrue(feedback[0]["to_delete"])
|
||||||
|
self.assertNotIn("to_delete", feedback[1])
|
||||||
|
|
||||||
|
def test_bad_boxes_keep_their_comment_as_global_feedback_without_mutating_data(self):
|
||||||
|
for box in ([753, 680, 287, 946], [10, 50, 20, 30], [10, 20, 10, 30],
|
||||||
|
[1, 2, 3], [None, 0, 10, 20], [float("nan"), 0, 10, 20]):
|
||||||
|
with self.subTest(box=box):
|
||||||
|
result = {"score": 2, "feedback": [{"text": "Important comment", "box_2d": box}]}
|
||||||
|
original_box = result["feedback"][0]["box_2d"]
|
||||||
|
scaled = _scaled_result(result, GroupCoordinates(2415, 3297, 1655, 3297))
|
||||||
|
callback = Mock()
|
||||||
|
render = Mock(return_value=Image.new("RGBA", (200, 40), "white"))
|
||||||
|
with patch.object(annotating, "render_score_text", return_value=Image.new("RGBA", (200, 40))):
|
||||||
|
image, _ = annotating.compose_label_image(Image.new("RGBA", (800, 100)), "Ex 5", scaled,
|
||||||
|
2415, render_fn=render, draw_callback=callback)
|
||||||
|
self.assertIsNotNone(image)
|
||||||
|
render.assert_called_once_with("Important comment", unittest.mock.ANY)
|
||||||
|
self.assertFalse(any(call.args[0] == "local_rect" for call in callback.call_args_list))
|
||||||
|
self.assertIs(result["feedback"][0]["box_2d"], original_box)
|
||||||
|
|
||||||
|
def test_valid_boxes_remain_local(self):
|
||||||
|
result = {"feedback": [{"text": "Local", "box_2d": [10, 20, 30, 40]}]}
|
||||||
|
before = copy.deepcopy(result)
|
||||||
|
callback = Mock()
|
||||||
|
with patch.object(annotating, "render_score_text", return_value=Image.new("RGBA", (200, 40))):
|
||||||
|
annotating.compose_label_image(Image.new("RGBA", (800, 100)), "Ex 5", result, 0,
|
||||||
|
render_fn=Mock(return_value=Image.new("RGBA", (200, 40))),
|
||||||
|
draw_callback=callback)
|
||||||
|
self.assertTrue(any(call.args[0] == "local_rect" for call in callback.call_args_list))
|
||||||
|
self.assertEqual(result, before)
|
||||||
|
self.assertTrue(valid_feedback_box([10, 20, 30, 40]))
|
||||||
|
|
||||||
|
def test_invalid_auxiliary_response_loses_only_its_rectangle(self):
|
||||||
|
returned = [{"text": "Keep this", "box_2d": [753, 680, 287, 946]}]
|
||||||
|
with patch.object(correction.prompting, "request_for_box_correction", return_value=([], {})), patch.object(
|
||||||
|
correction, "call_gemini_with_retries", return_value=json.dumps(returned)
|
||||||
|
):
|
||||||
|
feedback = correction.correct_boxes_with_gemini("26", "Ex 5", Path("unused.pdf"), [], 0, 1000, 1, 1000)
|
||||||
|
self.assertEqual(feedback, [{"text": "Keep this", "box_2d": None}])
|
||||||
|
|
||||||
|
def test_correction_requests_repair_for_inverted_boxes_and_falls_back_without_losing_comments(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
group = root / "Par label" / "Ex 5" / "Group_1.jpg"
|
||||||
|
group.parent.mkdir(parents=True)
|
||||||
|
group.touch()
|
||||||
|
atomic_write_json(group.with_suffix(".json"), [["26", 0, 1000, 1, "Ex 5"]])
|
||||||
|
args = correction.build_parser().parse_args([str(root)])
|
||||||
|
correction.configure_runtime(EvaluationWorkspace(root), [(str(group), "Ex 5")], args, api_client=Mock())
|
||||||
|
response = [{"id": "26", "result": {"score": 2, "error": "", "feedback": [
|
||||||
|
{"text": "First", "box_2d": [753, 680, 287, 946]},
|
||||||
|
{"text": "Second", "box_2d": [100, 900, 200, 100]}]}}]
|
||||||
|
with patch.object(correction.prompting, "generate_request", return_value=([], {})), patch.object(
|
||||||
|
correction, "correct_boxes_with_gemini", side_effect=RuntimeError("repair failed")
|
||||||
|
) as repair:
|
||||||
|
correction.process_single_task((str(group), "Ex 5"), json.dumps(response))
|
||||||
|
repair.assert_called_once()
|
||||||
|
feedback = correction.results["Ex 5"][0][0]["result"]["feedback"]
|
||||||
|
self.assertEqual([f["text"] for f in feedback], ["First", "Second"])
|
||||||
|
self.assertTrue(all(f["box_2d"] is None for f in feedback))
|
||||||
@@ -0,0 +1,371 @@
|
|||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from tkinter import ttk
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from copienator_gui.app import CopienatorApp
|
||||||
|
from copienator_gui.workflow import command_display
|
||||||
|
from copienator.commands.page_splitter import _selected_inputs
|
||||||
|
from copienator import EvaluationWorkspace
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(os.environ.get("DISPLAY"), "Tk requires a display")
|
||||||
|
class GuiConvenienceTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.repository = Path(self.temp.name)
|
||||||
|
self.evaluation = self.repository / "Évaluation avec espaces"
|
||||||
|
self.evaluation.mkdir()
|
||||||
|
self.app = CopienatorApp(self.repository, False, self.evaluation)
|
||||||
|
self.app.update()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
for callback in self.app.tk.splitlist(self.app.tk.call("after", "info")):
|
||||||
|
self.app.after_cancel(callback)
|
||||||
|
self.app.destroy()
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _label_texts(widget):
|
||||||
|
return [
|
||||||
|
text
|
||||||
|
for child in widget.winfo_children()
|
||||||
|
for text in (
|
||||||
|
[str(child.cget("text"))]
|
||||||
|
if isinstance(child, ttk.Label)
|
||||||
|
else GuiConvenienceTests._label_texts(child)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_reload_detects_added_and_removed_files_without_advancing(self):
|
||||||
|
for name in ("enonce.pdf", "enonce.tex", "correction.tex"):
|
||||||
|
(self.evaluation / name).touch()
|
||||||
|
self.app._reload_inputs()
|
||||||
|
self.assertIn("names", self.app.info_var.get())
|
||||||
|
(self.repository / "names").touch()
|
||||||
|
self.app._reload_inputs()
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.state_store.step("inputs")["status"], "success")
|
||||||
|
self.assertEqual(self.app.current_step.id, "inputs")
|
||||||
|
(self.evaluation / "enonce.pdf").unlink()
|
||||||
|
self.app._reload_inputs()
|
||||||
|
self.assertEqual(self.app.state_store.step("inputs")["status"], "ready")
|
||||||
|
self.assertIn("enonce.pdf", self.app.missing_requirements_label.cget("text"))
|
||||||
|
style = ttk.Style(self.app)
|
||||||
|
self.assertEqual(
|
||||||
|
style.lookup("MissingPrerequisite.TLabel", "foreground"), "#c62828"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_compact_evaluation_input_and_open_folder_button(self):
|
||||||
|
self.assertEqual(int(self.app.evaluation_entry.cget("width")), 42)
|
||||||
|
self.assertEqual(
|
||||||
|
int(self.app.open_evaluation_button.grid_info()["column"]),
|
||||||
|
int(self.app.evaluation_entry.grid_info()["column"]) + 1,
|
||||||
|
)
|
||||||
|
with patch("copienator_gui.app.open_path") as opened:
|
||||||
|
self.app.open_evaluation_button.invoke()
|
||||||
|
opened.assert_called_once_with(self.evaluation)
|
||||||
|
|
||||||
|
def test_manual_resolution_panel_follows_command_target(self):
|
||||||
|
manual = self.evaluation / "manual_resolutions.txt"
|
||||||
|
manual.write_text("Copie01 A -> B|\n", encoding="utf-8")
|
||||||
|
self.app.tree.selection_set("manual_resolution")
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(len(self.app.manual_panel.pdf_buttons), 2)
|
||||||
|
other = self.repository / "Other"
|
||||||
|
other.mkdir()
|
||||||
|
other_manual = other / "manual_resolutions.txt"
|
||||||
|
other_manual.write_text("Copie02 C xs D\n", encoding="utf-8")
|
||||||
|
self.app.arg_vars["target"].set(str(other))
|
||||||
|
self.app.update()
|
||||||
|
self.assertIn("Copie02", self.app.manual_panel.text.get("1.0", "end"))
|
||||||
|
with patch("copienator_gui.manual_resolution.open_path") as opened:
|
||||||
|
self.app.manual_panel.editor_button.invoke()
|
||||||
|
opened.assert_called_once_with(other_manual)
|
||||||
|
|
||||||
|
def test_successful_manual_resolution_returns_to_refaire_correction(self):
|
||||||
|
manual = self.evaluation / "manual_resolutions.txt"
|
||||||
|
manual.write_text("Copie01 A -> B|\n", encoding="utf-8")
|
||||||
|
self.app.state_store.update_step("manual_resolution", visited=True)
|
||||||
|
self.app.tree.selection_set("manual_resolution")
|
||||||
|
self.app.update()
|
||||||
|
|
||||||
|
self.app.active_step_id = "manual_resolution"
|
||||||
|
self.app._finish_process(0, False)
|
||||||
|
self.app.update()
|
||||||
|
|
||||||
|
self.assertEqual(self.app.current_step.id, "correction")
|
||||||
|
self.assertEqual(self.app.variant_var.get(), "refaire")
|
||||||
|
self.assertEqual(
|
||||||
|
self.app.state_store.step("correction")["variant"], "refaire"
|
||||||
|
)
|
||||||
|
self.assertNotIn("overwrite", self.app.arg_vars)
|
||||||
|
self.assertIn("--refaire", self.app._make_command())
|
||||||
|
|
||||||
|
def test_giving_names_waits_for_explicit_manual_completion(self):
|
||||||
|
return_dir = self.evaluation / "A Rendre" / "Student (01)"
|
||||||
|
return_dir.mkdir(parents=True)
|
||||||
|
(return_dir / "Student.jpg").write_bytes(b"jpg")
|
||||||
|
(return_dir / "Student.pdf").write_bytes(b"pdf")
|
||||||
|
self.app.tree.selection_set("giving_names")
|
||||||
|
self.app.update()
|
||||||
|
|
||||||
|
self.app.active_step_id = "giving_names"
|
||||||
|
self.app._finish_process(0, False)
|
||||||
|
self.app.update()
|
||||||
|
|
||||||
|
self.assertEqual(self.app.current_step.id, "giving_names")
|
||||||
|
self.assertEqual(
|
||||||
|
self.app.state_store.step("giving_names")["status"], "detected"
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
self.app.complete_giving_names_button.cget("text"),
|
||||||
|
"Marquer terminée",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.app.complete_giving_names_button.invoke()
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(
|
||||||
|
self.app.state_store.step("giving_names")["status"], "success"
|
||||||
|
)
|
||||||
|
self.assertNotEqual(self.app.current_step.id, "giving_names")
|
||||||
|
|
||||||
|
def test_batch_status_waits_until_all_jobs_are_ready(self):
|
||||||
|
self.app.state_store.update_step("correction", variant="batch")
|
||||||
|
self.app.state_store.update_step("batch_status", visited=True)
|
||||||
|
self.app.tree.selection_set("batch_status")
|
||||||
|
self.app.update()
|
||||||
|
command = self.app._make_command()
|
||||||
|
self.assertIn("--evaluation", command)
|
||||||
|
self.assertFalse(self.app.arg_vars)
|
||||||
|
self.app.active_step_id = "batch_status"
|
||||||
|
self.app._finish_process(4, False)
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.current_step.id, "batch_status")
|
||||||
|
self.app.active_step_id = "batch_status"
|
||||||
|
self.app._finish_process(0, False)
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.current_step.id, "fetch_batches")
|
||||||
|
|
||||||
|
def test_batch_watch_buttons_use_loaded_evaluation_and_stop_on_reload(self):
|
||||||
|
self.app.state_store.update_step("correction", variant="batch")
|
||||||
|
self.app.tree.selection_set("batch_status")
|
||||||
|
self.app.update()
|
||||||
|
with patch.object(self.app.batch_monitor, "start") as start:
|
||||||
|
self.app.watch_batches_button.invoke()
|
||||||
|
self.assertEqual(start.call_args.args[0][-2:], ["--evaluation", str(self.evaluation)])
|
||||||
|
self.app.batch_monitor.active = True
|
||||||
|
self.app._update_controls()
|
||||||
|
self.assertEqual(str(self.app.stop_watch_batches_button.cget("state")), "normal")
|
||||||
|
self.app.stop_watch_batches_button.invoke()
|
||||||
|
self.assertFalse(self.app.batch_monitor.active)
|
||||||
|
self.app.batch_monitor.active = True
|
||||||
|
self.app._load_evaluation()
|
||||||
|
self.assertFalse(self.app.batch_monitor.active)
|
||||||
|
|
||||||
|
def test_background_batch_readiness_notifies_without_changing_other_step(self):
|
||||||
|
self.app.tree.selection_set("inputs")
|
||||||
|
self.app.update()
|
||||||
|
with patch("copienator_gui.app.notify_desktop") as notify:
|
||||||
|
self.app._batch_results_ready()
|
||||||
|
notify.assert_called_once()
|
||||||
|
self.assertIn(self.evaluation.name, notify.call_args.args[1])
|
||||||
|
self.assertEqual(self.app.current_step.id, "inputs")
|
||||||
|
self.assertEqual(self.app.state_store.step("batch_status")["status"], "success")
|
||||||
|
|
||||||
|
def test_optional_blank_crop_can_target_a_copy_or_be_skipped(self):
|
||||||
|
copies = self.evaluation/"Copies"
|
||||||
|
copies.mkdir()
|
||||||
|
source = copies/"Copie01.pdf"
|
||||||
|
source.touch()
|
||||||
|
self.app.state_store.update_step("crop_blank_margins", visited=True)
|
||||||
|
self.app.tree.selection_set("crop_blank_margins")
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.current_step.id, "crop_blank_margins")
|
||||||
|
self.assertEqual(str(self.app.skip_button.cget("state")), "normal")
|
||||||
|
self.app.copy_var.set(source.name)
|
||||||
|
self.app._target_selected_copy()
|
||||||
|
command = self.app._make_command()
|
||||||
|
self.assertEqual(command[-4:], ["crop-margins", str(source), "--workers", "5"])
|
||||||
|
self.app._skip_step()
|
||||||
|
self.assertEqual(self.app.state_store.step("crop_blank_margins")["status"], "skipped")
|
||||||
|
|
||||||
|
def test_optional_answer_bottom_crop_uses_parallel_workers_and_can_be_skipped(self):
|
||||||
|
answers = self.evaluation / "Copies" / "Copie01"
|
||||||
|
answers.mkdir(parents=True)
|
||||||
|
(answers / "Ex 1.pdf").touch()
|
||||||
|
self.app.state_store.update_step("crop_exercise_bottoms", visited=True)
|
||||||
|
self.app.tree.selection_set("crop_exercise_bottoms")
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.current_step.id, "crop_exercise_bottoms")
|
||||||
|
self.assertEqual(str(self.app.skip_button.cget("state")), "normal")
|
||||||
|
command = self.app._make_command()
|
||||||
|
self.assertEqual(
|
||||||
|
command[-4:],
|
||||||
|
["crop-answer-bottoms", self.app._evaluation_arg(), "--workers", "5"],
|
||||||
|
)
|
||||||
|
self.app._skip_step()
|
||||||
|
self.assertEqual(
|
||||||
|
self.app.state_store.step("crop_exercise_bottoms")["status"], "skipped"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_redo_targets_selected_copy_and_copies_runnable_command(self):
|
||||||
|
for folder in ("Copies", "Copies Originales"):
|
||||||
|
(self.evaluation / folder).mkdir()
|
||||||
|
for name in ("Copie01.pdf", "Copie02.pdf"):
|
||||||
|
(self.evaluation / folder / name).touch()
|
||||||
|
self.app.tree.selection_set("page_splitter")
|
||||||
|
self.app.update()
|
||||||
|
self.app.copy_var.set("Copie02.pdf")
|
||||||
|
self.app._target_selected_copy()
|
||||||
|
command = self.app._make_command()
|
||||||
|
target = Path(command[-1])
|
||||||
|
self.assertEqual(target, self.evaluation / "Copies" / "Copie02.pdf")
|
||||||
|
self.assertEqual(_selected_inputs(EvaluationWorkspace(self.evaluation), target),
|
||||||
|
[self.evaluation / "Copies Originales" / "Copie02.pdf"])
|
||||||
|
self.app._copy_command()
|
||||||
|
self.assertEqual(self.app.clipboard_get(), command_display(command))
|
||||||
|
self.app._target_all_pages()
|
||||||
|
self.assertEqual(self.app.arg_vars["target"].get(), self.app._evaluation_arg())
|
||||||
|
|
||||||
|
def test_label_detection_can_target_one_copy(self):
|
||||||
|
copies = self.evaluation / "Copies"
|
||||||
|
copies.mkdir()
|
||||||
|
for name in ("Copie01.pdf", "Copie02.pdf"):
|
||||||
|
(copies / name).touch()
|
||||||
|
|
||||||
|
self.app.tree.selection_set("labels")
|
||||||
|
self.app.update()
|
||||||
|
self.assertEqual(self.app.copy_var.get(), "Copie01.pdf")
|
||||||
|
self.app.copy_var.set("Copie02.pdf")
|
||||||
|
self.app._target_selected_copy()
|
||||||
|
|
||||||
|
command = self.app._make_command()
|
||||||
|
self.assertEqual(Path(command[-1]), copies / "Copie02.pdf")
|
||||||
|
|
||||||
|
def button_texts(widget):
|
||||||
|
return [
|
||||||
|
text
|
||||||
|
for child in widget.winfo_children()
|
||||||
|
for text in (
|
||||||
|
[child.cget("text")]
|
||||||
|
if isinstance(child, ttk.Button)
|
||||||
|
else button_texts(child)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
controls = button_texts(self.app.form)
|
||||||
|
self.assertIn("Cibler la copie sélectionnée", controls)
|
||||||
|
self.assertIn("Cibler tout le dossier", controls)
|
||||||
|
|
||||||
|
def test_label_review_can_show_and_target_one_copy(self):
|
||||||
|
copies = self.evaluation / "Copies"
|
||||||
|
copies.mkdir()
|
||||||
|
for name in ("Copie01.pdf", "Copie02.pdf"):
|
||||||
|
(copies / name).touch()
|
||||||
|
|
||||||
|
self.app.tree.selection_set("plotting")
|
||||||
|
self.app.update()
|
||||||
|
self.app.copy_var.set("Copie02.pdf")
|
||||||
|
|
||||||
|
with patch("copienator_gui.app.open_path") as opened:
|
||||||
|
self.app._open_selected_copy()
|
||||||
|
opened.assert_called_once_with(copies / "Copie02.pdf")
|
||||||
|
|
||||||
|
self.app._target_selected_copy()
|
||||||
|
command = self.app._make_command()
|
||||||
|
self.assertEqual(Path(command[-1]), copies / "Copie02.pdf")
|
||||||
|
labels = self._label_texts(self.app.form)
|
||||||
|
self.assertTrue(
|
||||||
|
any("Seule la copie ciblée sera vérifiée" in text for text in labels)
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_free_form_arguments_are_shown_only_when_documented(self):
|
||||||
|
self.app.tree.selection_set("labels")
|
||||||
|
self.app.update()
|
||||||
|
labels_text = self._label_texts(self.app.form)
|
||||||
|
extra_label = next(
|
||||||
|
child
|
||||||
|
for child in self.app.form.winfo_children()
|
||||||
|
if isinstance(child, ttk.Label)
|
||||||
|
and str(child.cget("text")).startswith("Arguments supplémentaires")
|
||||||
|
)
|
||||||
|
tooltip = extra_label._copienator_tooltip
|
||||||
|
self.assertIn("Cutleft", tooltip.text)
|
||||||
|
self.assertFalse(any("Cutleft" in text for text in labels_text))
|
||||||
|
|
||||||
|
tooltip.show()
|
||||||
|
self.app.update()
|
||||||
|
self.assertIsNotNone(tooltip.window)
|
||||||
|
self.assertIn("Cutleft", tooltip.window.winfo_children()[0].cget("text"))
|
||||||
|
tooltip.hide()
|
||||||
|
|
||||||
|
self.app.tree.selection_set("plotting")
|
||||||
|
self.app.update()
|
||||||
|
self.assertFalse(
|
||||||
|
any(
|
||||||
|
text.startswith("Arguments supplémentaires")
|
||||||
|
for text in self._label_texts(self.app.form)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_each_visible_argument_label_has_a_tooltip(self):
|
||||||
|
self.app.tree.selection_set("correction")
|
||||||
|
self.app.update()
|
||||||
|
argument_labels = [
|
||||||
|
child
|
||||||
|
for child in self.app.form.winfo_children()
|
||||||
|
if isinstance(child, ttk.Label) and str(child.cget("text")).endswith("ⓘ")
|
||||||
|
]
|
||||||
|
self.assertGreaterEqual(len(argument_labels), 4)
|
||||||
|
for label in argument_labels:
|
||||||
|
with self.subTest(label=label.cget("text")):
|
||||||
|
tooltip = getattr(label, "_copienator_tooltip", None)
|
||||||
|
self.assertIsNotNone(tooltip)
|
||||||
|
self.assertTrue(tooltip.text.strip())
|
||||||
|
|
||||||
|
def test_verbose_checkbox_updates_supported_commands(self):
|
||||||
|
self.app.tree.selection_set("labels")
|
||||||
|
self.app.update()
|
||||||
|
self.assertNotIn("--verbose", self.app._make_command())
|
||||||
|
self.app.verbose_var.set(True)
|
||||||
|
self.assertIn("--verbose", self.app._make_command())
|
||||||
|
|
||||||
|
def test_console_selection_survives_output_and_is_read_only(self):
|
||||||
|
self.app._append_console("Première ligne\nDeuxième ligne\n")
|
||||||
|
self.app.console.tag_add("sel", "1.0", "1.end")
|
||||||
|
self.app._append_console("Suite\n")
|
||||||
|
self.app._copy_console_selection()
|
||||||
|
self.assertEqual(self.app.clipboard_get(), "Première ligne")
|
||||||
|
self.app._copy_console_all()
|
||||||
|
expected = "Première ligne\nDeuxième ligne\nSuite\n"
|
||||||
|
self.assertEqual(self.app.clipboard_get(), expected)
|
||||||
|
self.app.console.insert("end", "unwanted edit")
|
||||||
|
self.assertEqual(self.app.console.get("1.0", "end-1c"), expected)
|
||||||
|
self.app._select_console_all()
|
||||||
|
self.app._copy_console_selection()
|
||||||
|
self.assertEqual(self.app.clipboard_get(), expected)
|
||||||
|
|
||||||
|
def test_validate_rename_preserves_files_and_downstream_status(self):
|
||||||
|
pdf = self.evaluation / "Copie01.pdf"
|
||||||
|
pdf.write_bytes(b"unchanged PDF")
|
||||||
|
self.app.state_store.update_step("rename", status="stale")
|
||||||
|
self.app.state_store.update_step("page_splitter", status="success")
|
||||||
|
self.app.tree.selection_set("rename")
|
||||||
|
self.app.update()
|
||||||
|
with patch.object(self.app.runner, "start") as start:
|
||||||
|
self.app.validate_rename_button.invoke()
|
||||||
|
start.assert_not_called()
|
||||||
|
self.assertEqual(self.app.state_store.step("rename")["status"], "success")
|
||||||
|
self.assertEqual(self.app.state_store.step("page_splitter")["status"], "success")
|
||||||
|
self.assertEqual(pdf.read_bytes(), b"unchanged PDF")
|
||||||
|
self.assertEqual(list(self.evaluation.glob("*.pdf")), [pdf])
|
||||||
|
self.app.state_store.update_step("rename", status="stale")
|
||||||
|
self.app.active_step_id = "page_splitter"
|
||||||
|
self.app._update_controls()
|
||||||
|
self.assertIn("disabled", self.app.validate_rename_button.state())
|
||||||
|
self.app._validate_rename_step()
|
||||||
|
self.assertEqual(self.app.state_store.step("rename")["status"], "stale")
|
||||||
|
self.app.active_step_id = None
|
||||||
+905
-25
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,56 @@
|
|||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import tkinter as tk
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator_gui.cut_helper import CutHelper, PAGE_GAP
|
||||||
|
from copienator_gui.manual_resolution import ManualResolutionPanel
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(os.environ.get("DISPLAY"), "requires a display")
|
||||||
|
class CutHelperTests(unittest.TestCase):
|
||||||
|
def test_cut_button_drag_to_page_gap_and_enter_only_produces_command(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
directory = Path(temporary)
|
||||||
|
source = directory / "Copies" / "Copie16" / "A.pdf"
|
||||||
|
source.parent.mkdir(parents=True)
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
for height in (200, 400):
|
||||||
|
page = document.new_page(width=300, height=height)
|
||||||
|
page.insert_text((20, 40), "Student answer")
|
||||||
|
document.save(source)
|
||||||
|
manual = directory / "manual_resolutions.txt"
|
||||||
|
manual.write_text("Copie16 A -> B|\n")
|
||||||
|
original = source.read_bytes()
|
||||||
|
root = tk.Tk()
|
||||||
|
try:
|
||||||
|
panel = ManualResolutionPanel(root, lambda: directory)
|
||||||
|
panel.pack()
|
||||||
|
root.update()
|
||||||
|
self.assertEqual(len(panel.cut_buttons), 1)
|
||||||
|
panel.cut_buttons[0].invoke()
|
||||||
|
helper = next(child for child in panel.winfo_children() if isinstance(child, CutHelper))
|
||||||
|
helper.canvas.yview_moveto(0)
|
||||||
|
root.update()
|
||||||
|
helper.move_bar(SimpleNamespace(y=200 * helper.scale + PAGE_GAP / 2))
|
||||||
|
self.assertIn("entre les pages 1 et 2", helper.caption.get())
|
||||||
|
helper.keep.set(2)
|
||||||
|
helper.mode.set("x")
|
||||||
|
helper.focus_force()
|
||||||
|
root.update()
|
||||||
|
helper.event_generate("<Return>")
|
||||||
|
root.update()
|
||||||
|
self.assertFalse(helper.winfo_exists())
|
||||||
|
self.assertEqual(panel.cut_result.get(), "Copie16 A c{33.333333}2x B|")
|
||||||
|
panel.copy_cut_command()
|
||||||
|
self.assertEqual(root.clipboard_get(), panel.cut_result.get())
|
||||||
|
self.assertEqual(source.read_bytes(), original)
|
||||||
|
self.assertEqual(manual.read_text(), "Copie16 A -> B|\n")
|
||||||
|
panel.reload()
|
||||||
|
self.assertEqual(len(panel.cut_buttons), 1)
|
||||||
|
finally:
|
||||||
|
root.destroy()
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import tkinter as tk
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import call, patch
|
||||||
|
|
||||||
|
from copienator_gui.manual_resolution import ManualResolutionPanel
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(os.environ.get("DISPLAY"), "requires a display")
|
||||||
|
class ManualResolutionPanelTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.evaluation = Path(self.temp.name)
|
||||||
|
self.root = tk.Tk()
|
||||||
|
self.addCleanup(self.root.destroy)
|
||||||
|
self.path = self.evaluation / "manual_resolutions.txt"
|
||||||
|
|
||||||
|
def panel(self):
|
||||||
|
panel = ManualResolutionPanel(self.root, lambda: self.evaluation)
|
||||||
|
panel.pack()
|
||||||
|
self.root.update()
|
||||||
|
return panel
|
||||||
|
|
||||||
|
def test_preview_editor_and_each_pdf_pair_use_shared_resolution_rules(self):
|
||||||
|
content = "### Instructions\nCopie01 Ex 1 x> |Ex 2\n\nCopie02 Ex 3 ss Ex 4|\n"
|
||||||
|
self.path.write_text(content, encoding="utf-8")
|
||||||
|
paths = []
|
||||||
|
for copy, label in (("01", "Ex 1_new"), ("01", "Ex 2_old"),
|
||||||
|
("02", "Ex 3"), ("02", "Ex 4")):
|
||||||
|
path = self.evaluation / "Copies" / f"Copie{copy}" / f"{label}.pdf"
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.touch()
|
||||||
|
paths.append(path)
|
||||||
|
panel = self.panel()
|
||||||
|
self.assertEqual(panel.text.get("1.0", "end-1c"), content)
|
||||||
|
self.assertEqual(len(panel.pdf_buttons), 4)
|
||||||
|
self.assertEqual([button.cget("text") for button in panel.pdf_buttons],
|
||||||
|
["PDF source", "PDF cible"] * 2)
|
||||||
|
with patch("copienator_gui.manual_resolution.open_path") as opened:
|
||||||
|
panel.editor_button.invoke()
|
||||||
|
for button in panel.pdf_buttons:
|
||||||
|
button.invoke()
|
||||||
|
self.assertEqual(opened.call_args_list, [call(self.path), *map(call, paths)])
|
||||||
|
|
||||||
|
def test_reload_keeps_invalid_lines_visible_and_removes_stale_actions(self):
|
||||||
|
self.path.write_text("Copie01 A -> B|\n", encoding="utf-8")
|
||||||
|
panel = self.panel()
|
||||||
|
self.path.write_text("bad instruction\nCopie02 C sx D\n", encoding="utf-8")
|
||||||
|
panel.reload()
|
||||||
|
self.assertIn("bad instruction", panel.text.get("1.0", "end"))
|
||||||
|
self.assertIn("lignes invalides : 1", panel.status.cget("text"))
|
||||||
|
self.assertEqual(len(panel.pdf_buttons), 2)
|
||||||
|
self.path.unlink()
|
||||||
|
panel.reload()
|
||||||
|
self.assertFalse(panel.pdf_buttons)
|
||||||
|
self.assertEqual(str(panel.editor_button.cget("state")), "disabled")
|
||||||
|
|
||||||
|
def test_missing_source_does_not_prevent_opening_destination(self):
|
||||||
|
self.path.write_text("Copie01 A -> B|\n", encoding="utf-8")
|
||||||
|
destination = self.evaluation / "Copies" / "Copie01" / "B.pdf"
|
||||||
|
destination.parent.mkdir(parents=True)
|
||||||
|
destination.touch()
|
||||||
|
panel = self.panel()
|
||||||
|
with patch("copienator_gui.manual_resolution.open_path") as opened, patch(
|
||||||
|
"copienator_gui.manual_resolution.messagebox.showerror"
|
||||||
|
) as error:
|
||||||
|
panel.pdf_buttons[0].invoke()
|
||||||
|
opened.assert_not_called()
|
||||||
|
panel.pdf_buttons[1].invoke()
|
||||||
|
opened.assert_called_once_with(destination)
|
||||||
|
self.assertIn("A.pdf", error.call_args.args[1])
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
import unittest
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from copienator.ink_detection import detect_bounds, _paper_blur
|
||||||
|
|
||||||
|
|
||||||
|
class InkDetectionTests(unittest.TestCase):
|
||||||
|
def test_fast_background_blur_matches_opencv_pixel_for_pixel(self):
|
||||||
|
rng = np.random.default_rng(42)
|
||||||
|
for shape in ((97,131), (241,319)):
|
||||||
|
gray = rng.integers(0,256,shape,dtype=np.uint8)
|
||||||
|
for dpi in (100,150,200,300):
|
||||||
|
with self.subTest(shape=shape,dpi=dpi):
|
||||||
|
expected = cv2.GaussianBlur(gray,(0,0),dpi/8)
|
||||||
|
np.testing.assert_array_equal(_paper_blur(gray,dpi/8),expected)
|
||||||
|
|
||||||
|
def scan(self):
|
||||||
|
image = np.full((1200, 850, 3), 255, np.uint8)
|
||||||
|
for y in range(20, 1200, 20):
|
||||||
|
cv2.line(image, (0, y), (849, y), (160, 155, 205), 1)
|
||||||
|
for x in range(10, 850, 20):
|
||||||
|
cv2.line(image, (x, 0), (x, 1199), (160, 155, 205), 1)
|
||||||
|
for y in (100, 180, 400, 480, 800, 880, 1050, 1130):
|
||||||
|
cv2.ellipse(image, (25, y), (12, 20), 0, 0, 300, (155,155,155), 2)
|
||||||
|
cv2.putText(image, 'x + y = 2', (110, 400), cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1, (20, 90, 190), 2)
|
||||||
|
cv2.putText(image, 'answer = 42', (110, 700), cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1, (20, 90, 190), 2)
|
||||||
|
return image
|
||||||
|
|
||||||
|
def test_crops_past_holes_and_grid(self):
|
||||||
|
r = detect_bounds(self.scan(), dpi=150)
|
||||||
|
self.assertGreater(r['top_px'], 250)
|
||||||
|
self.assertLess(r['top_px'], 370)
|
||||||
|
self.assertGreater(r['bottom_px'], 700)
|
||||||
|
self.assertLess(r['bottom_px'], 800)
|
||||||
|
|
||||||
|
def test_isolated_margin_note_survives(self):
|
||||||
|
image = self.scan()
|
||||||
|
cv2.putText(image, '1', (4, 1110), cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
.65, (20, 90, 190), 2)
|
||||||
|
self.assertGreater(detect_bounds(image, dpi=150)['bottom_px'], 1110)
|
||||||
|
|
||||||
|
def test_long_fraction_bar_on_grid_survives(self):
|
||||||
|
for colour in ((20,90,190), (20,20,20), (190,20,20)):
|
||||||
|
with self.subTest(colour=colour):
|
||||||
|
image = self.scan()
|
||||||
|
cv2.line(image, (100, 1020), (700, 1020), colour, 3)
|
||||||
|
self.assertGreater(detect_bounds(image, dpi=150)['bottom_px'], 1020)
|
||||||
|
|
||||||
|
def test_black_annotation_survives(self):
|
||||||
|
image = self.scan()
|
||||||
|
cv2.putText(image, 'note', (600, 1100), cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
.7, (30,30,30), 2)
|
||||||
|
self.assertGreater(detect_bounds(image, dpi=150)['bottom_px'], 1100)
|
||||||
|
|
||||||
|
def test_blank_and_pencil_only_pages_are_retained(self):
|
||||||
|
for pencil in (False, True):
|
||||||
|
image = np.full((1200,850,3),255,np.uint8)
|
||||||
|
if pencil:
|
||||||
|
cv2.putText(image, 'pencil', (100,600), cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1, (195,195,195), 2)
|
||||||
|
r = detect_bounds(image, dpi=150)
|
||||||
|
self.assertEqual((r['top_px'],r['bottom_px']), (0,1200))
|
||||||
|
self.assertEqual(r['status'], 'review-no-ink-seeds')
|
||||||
|
|
||||||
|
def test_skewed_colour_scan(self):
|
||||||
|
matrix = cv2.getRotationMatrix2D((425,600),2,1)
|
||||||
|
image = cv2.warpAffine(self.scan(),matrix,(850,1200),borderValue=(255,255,255))
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertGreater(r['top_px'],250)
|
||||||
|
self.assertLess(r['top_px'],370)
|
||||||
|
self.assertGreater(r['bottom_px'],715)
|
||||||
|
self.assertLess(r['bottom_px'],820)
|
||||||
|
|
||||||
|
def test_weak_stroke_attached_to_ink_is_recovered(self):
|
||||||
|
image = np.full((1200,850,3),255,np.uint8)
|
||||||
|
cv2.rectangle(image,(400,500),(410,530),(20,90,190),-1)
|
||||||
|
cv2.rectangle(image,(400,531),(410,540),(130,170,205),-1)
|
||||||
|
r = detect_bounds(image,dpi=150,padding_mm=0,min_crop_mm=0)
|
||||||
|
self.assertGreaterEqual(r['bottom_px'],541)
|
||||||
|
|
||||||
|
def dark_grid(self):
|
||||||
|
image = np.full((1200,850,3),255,np.uint8)
|
||||||
|
for y in range(20,1200,20):
|
||||||
|
cv2.line(image,(0,y),(849,y),(65,65,65),1)
|
||||||
|
for x in range(10,850,20):
|
||||||
|
cv2.line(image,(x,0),(x,1199),(65,65,65),1)
|
||||||
|
cv2.putText(image,'x + y = 2',(100,400),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1,(20,20,20),2)
|
||||||
|
cv2.putText(image,'answer = 42',(100,700),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1,(20,20,20),2)
|
||||||
|
return image
|
||||||
|
|
||||||
|
def test_dark_grid_does_not_seed_entire_page(self):
|
||||||
|
r = detect_bounds(self.dark_grid(),dpi=150)
|
||||||
|
self.assertTrue(r['paper_cleanup'])
|
||||||
|
self.assertGreater(r['top_px'],250)
|
||||||
|
self.assertLess(r['top_px'],370)
|
||||||
|
self.assertGreater(r['bottom_px'],700)
|
||||||
|
self.assertLess(r['bottom_px'],820)
|
||||||
|
|
||||||
|
def test_faint_isolated_note_on_dark_grid_survives(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
cv2.putText(image,'pencil',(100,1100),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
.7,(190,190,190),2)
|
||||||
|
self.assertGreater(detect_bounds(image,dpi=150)['bottom_px'],1100)
|
||||||
|
|
||||||
|
def test_sparse_central_pencil_on_dark_grid_survives(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
# Thin, pale handwriting crossing the ruling is split into sparse
|
||||||
|
# components. Its central position distinguishes it from page holes.
|
||||||
|
cv2.putText(image,'result',(330,1090),cv2.FONT_HERSHEY_SCRIPT_SIMPLEX,
|
||||||
|
.85,(155,155,155),1)
|
||||||
|
self.assertGreater(detect_bounds(image,dpi=150)['bottom_px'],1090)
|
||||||
|
|
||||||
|
def test_large_unruled_diagram_is_not_paper(self):
|
||||||
|
image = np.full((1200,850,3),255,np.uint8)
|
||||||
|
cv2.rectangle(image,(100,100),(750,1100),(20,20,20),3)
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertFalse(r['paper_cleanup'])
|
||||||
|
self.assertLess(r['top_px'],100)
|
||||||
|
self.assertGreater(r['bottom_px'],1100)
|
||||||
|
|
||||||
|
def test_dark_grid_preserves_black_fraction_bar(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
cv2.line(image,(150,1020),(700,1020),(0,0,0),3)
|
||||||
|
self.assertGreater(detect_bounds(image,dpi=150)['bottom_px'],1020)
|
||||||
|
|
||||||
|
def test_disconnected_dark_grid_is_still_recognized(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
for x in range(170,850,170):
|
||||||
|
image[:,x:x+5] = 255
|
||||||
|
for y in range(200,1200,200):
|
||||||
|
image[y:y+5,:] = 255
|
||||||
|
cv2.putText(image,'x + y = 2',(100,400),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
1,(20,20,20),2)
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertTrue(r['paper_cleanup'])
|
||||||
|
self.assertGreater(r['top_px'],250)
|
||||||
|
self.assertLess(r['top_px'],370)
|
||||||
|
self.assertGreater(r['bottom_px'],700)
|
||||||
|
self.assertLess(r['bottom_px'],850)
|
||||||
|
|
||||||
|
def test_repeated_dark_holes_on_either_side(self):
|
||||||
|
for right in (False, True):
|
||||||
|
with self.subTest(right=right):
|
||||||
|
image = self.dark_grid()
|
||||||
|
x = 820 if right else 25
|
||||||
|
for y in (110, 370, 630, 890, 1130):
|
||||||
|
cv2.ellipse(image, (x,y), (12,20), 0, 0, 300, (35,35,35), 2)
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertLess(r['bottom_px'],850)
|
||||||
|
# The same column can contain handwriting as well as holes.
|
||||||
|
cv2.putText(image,'7',(x-5,1060),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
.7,(20,20,20),2)
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertGreater(r['bottom_px'],1060)
|
||||||
|
|
||||||
|
def test_faint_page_number_in_outer_band_does_not_block_crop(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
cv2.putText(image,'4/',(3,1160),cv2.FONT_HERSHEY_SIMPLEX,
|
||||||
|
.55,(130,130,130),1)
|
||||||
|
self.assertLess(detect_bounds(image,dpi=150)['bottom_px'],850)
|
||||||
|
|
||||||
|
def test_faded_neutral_grid(self):
|
||||||
|
image = self.dark_grid()
|
||||||
|
image[np.all(image == 65,axis=2)] = 145
|
||||||
|
r = detect_bounds(image,dpi=150)
|
||||||
|
self.assertTrue(r['paper_cleanup'])
|
||||||
|
self.assertLess(r['bottom_px'],850)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator import CliError, EvaluationWorkspace, atomic_write_json, read_json
|
||||||
|
from copienator.commands.resolve_manual import parse_instruction_text, resolve_manual
|
||||||
|
from copienator.pdf_cut import cut_position, split_pdf
|
||||||
|
from copienator_gui.cut_helper import PAGE_GAP, cut_operator, percentage_at_y, y_at_percentage
|
||||||
|
|
||||||
|
|
||||||
|
def make_pdf(path, heights=(300, 700), rotation=0, cropped=False):
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
for index, height in enumerate(heights):
|
||||||
|
page = document.new_page(width=200, height=height)
|
||||||
|
page.draw_rect(pymupdf.Rect(0, 0, 200, height / 2), color=None, fill=(1, 0, 0))
|
||||||
|
page.draw_rect(pymupdf.Rect(0, height / 2, 200, height), color=None, fill=(0, 0, 1))
|
||||||
|
page.insert_text((30, 30), f"Page {index + 1}")
|
||||||
|
if cropped:
|
||||||
|
page.set_cropbox(pymupdf.Rect(10, 20, 180, height - 30))
|
||||||
|
page.set_rotation(rotation)
|
||||||
|
document.save(path)
|
||||||
|
|
||||||
|
|
||||||
|
class ManualCutTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
|
||||||
|
def test_parser_accepts_new_operators_and_rejects_invalid_cuts(self):
|
||||||
|
for keep in (1, 2):
|
||||||
|
for action in (">", "x"):
|
||||||
|
item = parse_instruction_text(f"Copie16 Ex 4 : 1) c{{43.125}}{keep}{action} |Ex 4 : 2)")[0]
|
||||||
|
self.assertEqual(item.cut, (43.125, keep))
|
||||||
|
self.assertEqual(item.should_merge, action == ">")
|
||||||
|
self.assertTrue(item.pipe_first)
|
||||||
|
for operator in ("c{0}1>", "c{100}1>", "c{-1}1>", "c{101}1>", "c{43}3>",
|
||||||
|
"c{43}1s", "c{NaN}1>"):
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
parse_instruction_text(f"Copie16 A {operator} B")
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
parse_instruction_text("Copie16 A c{43}1> A")
|
||||||
|
|
||||||
|
def test_page_boundary_preserves_whole_pages_without_clipping(self):
|
||||||
|
source, first, second = [self.root / name for name in ("source.pdf", "first.pdf", "second.pdf")]
|
||||||
|
make_pdf(source, heights=(100, 200), rotation=180)
|
||||||
|
before = source.read_bytes()
|
||||||
|
with patch("pymupdf.Page.show_pdf_page", side_effect=AssertionError("must not clip whole pages")):
|
||||||
|
split_pdf(source, 33.333333, first, second)
|
||||||
|
for path, height in ((first, 100), (second, 200)):
|
||||||
|
with pymupdf.open(path) as pdf:
|
||||||
|
self.assertEqual(len(pdf), 1)
|
||||||
|
self.assertEqual(pdf[0].rect.height, height)
|
||||||
|
self.assertEqual(pdf[0].rotation, 180)
|
||||||
|
self.assertEqual(source.read_bytes(), before)
|
||||||
|
|
||||||
|
def test_in_page_cut_preserves_visible_pixels_for_cropped_rotated_pages(self):
|
||||||
|
source, first, second = [self.root / name for name in ("source.pdf", "first.pdf", "second.pdf")]
|
||||||
|
for rotation in (0, 90, 180, 270):
|
||||||
|
with self.subTest(rotation=rotation):
|
||||||
|
make_pdf(source, heights=(400,), rotation=rotation, cropped=True)
|
||||||
|
with pymupdf.open(source) as pdf:
|
||||||
|
pix = pdf[0].get_pixmap()
|
||||||
|
expected = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
|
||||||
|
split_pdf(source, 50, first, second)
|
||||||
|
for path, pixels in ((first, expected[:len(expected)//2]), (second, expected[len(expected)//2:])):
|
||||||
|
with pymupdf.open(path) as pdf:
|
||||||
|
pix = pdf[0].get_pixmap()
|
||||||
|
actual = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
|
||||||
|
self.assertEqual(actual.shape, pixels.shape)
|
||||||
|
self.assertLess(np.abs(actual.astype(float) - pixels).mean(), 0.1)
|
||||||
|
|
||||||
|
def test_cut_within_page_preserves_subsequent_pages(self):
|
||||||
|
source, first, second = [self.root / name for name in ("source.pdf", "first.pdf", "second.pdf")]
|
||||||
|
make_pdf(source)
|
||||||
|
split_pdf(source, 15, first, second)
|
||||||
|
with pymupdf.open(first) as pdf:
|
||||||
|
self.assertEqual([page.rect.height for page in pdf], [150])
|
||||||
|
with pymupdf.open(second) as pdf:
|
||||||
|
self.assertEqual([page.rect.height for page in pdf], [150, 700])
|
||||||
|
|
||||||
|
def test_resolver_archives_source_and_recorrrects_both_labels(self):
|
||||||
|
for keep in (1, 2):
|
||||||
|
for mode in (">", "x"):
|
||||||
|
for pipe_first in (False, True):
|
||||||
|
with self.subTest(keep=keep, mode=mode, pipe_first=pipe_first):
|
||||||
|
root = self.root / f"{keep}{mode == '>'}{pipe_first}"
|
||||||
|
copies = root / "Copies" / "Copie16"
|
||||||
|
copies.mkdir(parents=True)
|
||||||
|
source, target = copies / "A.pdf", copies / "B.pdf"
|
||||||
|
make_pdf(source)
|
||||||
|
make_pdf(target, heights=(80,))
|
||||||
|
original, destination = source.read_bytes(), target.read_bytes()
|
||||||
|
atomic_write_json(root / "correction.json", {
|
||||||
|
label: [[{"id": "16", "result": {}}]] for label in ("A", "B")
|
||||||
|
})
|
||||||
|
new_label = "|B" if pipe_first else "B|"
|
||||||
|
(root / "manual_resolutions.txt").write_text(f"Copie16 A c{{30}}{keep}{mode} {new_label}\n")
|
||||||
|
self.assertEqual(resolve_manual(EvaluationWorkspace(root)), 0)
|
||||||
|
self.assertEqual((copies / "A_old.pdf").read_bytes(), original)
|
||||||
|
self.assertEqual((copies / "B_old.pdf").read_bytes(), destination)
|
||||||
|
retained, moved = (300, 700) if keep == 1 else (700, 300)
|
||||||
|
with pymupdf.open(copies / "A_new.pdf") as pdf:
|
||||||
|
self.assertEqual([page.rect.height for page in pdf], [retained])
|
||||||
|
with pymupdf.open(copies / "B_new.pdf") as pdf:
|
||||||
|
expected = ([moved, 80] if pipe_first else [80, moved]) if mode == ">" else [moved]
|
||||||
|
self.assertEqual([page.rect.height for page in pdf], expected)
|
||||||
|
self.assertEqual(read_json(root / "refaire.json"), [["Copie16", ["A", "B"]]])
|
||||||
|
self.assertTrue(all(read_json(root / "correction.json")[label][0][0]["result"]["suffix"] == "_new" for label in ("A", "B")))
|
||||||
|
self.assertFalse(list(copies.glob("temp_*.pdf")))
|
||||||
|
|
||||||
|
def test_helper_snaps_to_gap_and_round_trips_percentage(self):
|
||||||
|
heights = [100, 200]
|
||||||
|
for y in (95, 100, 100 + PAGE_GAP / 2, 100 + PAGE_GAP + 5):
|
||||||
|
percent = percentage_at_y(y, heights, 1)
|
||||||
|
self.assertEqual(cut_position(heights, percent), (1, 0))
|
||||||
|
self.assertEqual(y_at_percentage(percent, heights, 1), 100 + PAGE_GAP / 2)
|
||||||
|
self.assertEqual(cut_operator(43, 1, ">"), "c{43}1>")
|
||||||
|
self.assertEqual(cut_operator(100/3, 2, "x"), "c{33.333333}2x")
|
||||||
|
self.assertEqual(cut_position(heights, 33.333333), (1, 0))
|
||||||
|
self.assertNotEqual(cut_position(heights, 33.3)[1], 0)
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
import copy
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator import EvaluationWorkspace, atomic_write_json, read_json
|
||||||
|
from copienator.commands import correction, resolve_manual
|
||||||
|
|
||||||
|
|
||||||
|
class ManualResolutionStateTests(unittest.TestCase):
|
||||||
|
def test_acknowledgement_clears_only_this_target_and_copy(self):
|
||||||
|
for error in ("wrg-lbl:B?", "wrg-lbl:B?delayed", "wrg-lbl:B?exists", "al:(->)B?(->)C?", "al:(delayed)B"):
|
||||||
|
with self.subTest(error=error):
|
||||||
|
source = {"id": "01", "result": {"error": error, "delayed": [
|
||||||
|
["wrong-label", "B"], ["add-label", "B"], ["add-label", "C"]]}}
|
||||||
|
other = {"id": "02", "result": {"error": error, "delayed": [["wrong-label", "B"]]}}
|
||||||
|
before_other = copy.deepcopy(other)
|
||||||
|
results = {"A": [[source, other]]}
|
||||||
|
resolve_manual.set_suffix_and_clean_error(results, "01", "A", None, "B")
|
||||||
|
self.assertEqual(source["result"]["delayed"], [["add-label", "C"]])
|
||||||
|
self.assertNotIn("B?", source["result"]["error"])
|
||||||
|
self.assertEqual(other, before_other)
|
||||||
|
resolve_manual.set_suffix_and_clean_error(results, "01", "A", None, "C")
|
||||||
|
self.assertNotIn("delayed", source["result"])
|
||||||
|
|
||||||
|
def fixture(self, root, operator):
|
||||||
|
copies = root / "Copies" / "Copie01"
|
||||||
|
copies.mkdir(parents=True)
|
||||||
|
for label in ("A", "B"):
|
||||||
|
with pymupdf.open() as doc:
|
||||||
|
page = doc.new_page(width=200, height=200)
|
||||||
|
page.insert_text((20, 40), label)
|
||||||
|
doc.save(copies / f"{label}.pdf")
|
||||||
|
atomic_write_json(root / "correction.json", {
|
||||||
|
"A": [[{"id": "01", "result": {"error": "wrg-lbl:B?", "delayed": [["wrong-label", "B"]]}}]],
|
||||||
|
"B": [[{"id": "01", "result": {"error": ""}}]],
|
||||||
|
})
|
||||||
|
(root / "manual_resolutions.txt").write_text(f"Copie01 A {operator} B|\n")
|
||||||
|
return EvaluationWorkspace(root)
|
||||||
|
|
||||||
|
def test_every_resolution_acknowledges_pending_conflict(self):
|
||||||
|
for operator in (*resolve_manual.OPERATORS, "c{50}1>", "c{50}2x"):
|
||||||
|
with self.subTest(operator=operator), tempfile.TemporaryDirectory() as temporary:
|
||||||
|
workspace = self.fixture(Path(temporary), operator)
|
||||||
|
resolve_manual.resolve_manual(workspace)
|
||||||
|
result = read_json(workspace.correction_file)["A"][0][0]["result"]
|
||||||
|
self.assertNotIn("delayed", result)
|
||||||
|
self.assertFalse(workspace.manual_resolutions_file.exists())
|
||||||
|
|
||||||
|
def test_refaire_does_not_recreate_a_resolved_source_conflict(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
workspace = self.fixture(Path(temporary), "x>")
|
||||||
|
resolve_manual.resolve_manual(workspace)
|
||||||
|
(workspace.groups_dir / "B").mkdir(parents=True)
|
||||||
|
args = correction.build_parser().parse_args([str(workspace.root), "--refaire"])
|
||||||
|
correction.configure_runtime(workspace, [], args, api_client=Mock())
|
||||||
|
with patch.object(correction.grouping, "get_pdf_height", return_value=400), patch.object(
|
||||||
|
correction.grouping, "create_jpg"
|
||||||
|
), patch.object(correction, "process_single_task", return_value=[]):
|
||||||
|
self.assertEqual(correction.run_configured(args), 0)
|
||||||
|
self.assertFalse(workspace.manual_resolutions_file.exists())
|
||||||
|
self.assertNotIn("delayed", correction.results["A"][0][0]["result"])
|
||||||
@@ -0,0 +1,223 @@
|
|||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from queue import Queue
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from copienator import CliError, EvaluationWorkspace, ExitCode
|
||||||
|
from copienator.copy_errors import copy_errors, clear_copy_error, mark_copy_error, marked_copy_paths
|
||||||
|
from copienator.commands import cutleft, page_splitter
|
||||||
|
from copienator_gui.app import CopienatorApp
|
||||||
|
|
||||||
|
|
||||||
|
class MarkedCopiesTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.workspace = EvaluationWorkspace(Path(self.temp.name))
|
||||||
|
self.workspace.copies_dir.mkdir()
|
||||||
|
self.workspace.original_copies_dir.mkdir()
|
||||||
|
self.files = [self.workspace.copies_dir / name for name in ("Copie01.pdf", "Copie02.pdf")]
|
||||||
|
for path in self.files:
|
||||||
|
path.write_bytes(b"processed")
|
||||||
|
(self.workspace.original_copies_dir / path.name).write_bytes(b"original")
|
||||||
|
|
||||||
|
def reviewer(self):
|
||||||
|
review = cutleft.ImageReviewer.__new__(cutleft.ImageReviewer)
|
||||||
|
review.workspace = self.workspace
|
||||||
|
review.files = self.files
|
||||||
|
review.output_dir = self.workspace.cutleft_dir
|
||||||
|
review.index = 0
|
||||||
|
review.is_processing = False
|
||||||
|
review.had_errors = False
|
||||||
|
review.completed = False
|
||||||
|
review.current_shift = 50
|
||||||
|
review.current_width_offset = 0
|
||||||
|
review.default_max_per_file = 5
|
||||||
|
review.current_max_per_file = 1
|
||||||
|
review.root = Mock()
|
||||||
|
review.stop_prefetch = threading.Event()
|
||||||
|
review.load_current_image = Mock()
|
||||||
|
review.update_display = Mock()
|
||||||
|
image = Image.new("RGB", (8, 8), "white")
|
||||||
|
review.current_result = (image, [image], {"total_pages": 1, "columns_per_file": [1]})
|
||||||
|
return review
|
||||||
|
|
||||||
|
def test_marks_survive_reload_and_resolve_originals_only_for_splitting(self):
|
||||||
|
mark_copy_error(self.workspace, self.files[1], "Wrong page order")
|
||||||
|
reloaded = EvaluationWorkspace(self.workspace.root)
|
||||||
|
self.assertEqual(marked_copy_paths(reloaded), [self.files[1]])
|
||||||
|
original = self.workspace.original_copies_dir / self.files[1].name
|
||||||
|
self.assertEqual(marked_copy_paths(reloaded, originals=True), [original])
|
||||||
|
self.files[1].unlink()
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
marked_copy_paths(reloaded)
|
||||||
|
self.assertEqual(marked_copy_paths(reloaded, originals=True), [original])
|
||||||
|
clear_copy_error(reloaded, original)
|
||||||
|
self.assertEqual(copy_errors(self.workspace), {})
|
||||||
|
|
||||||
|
def test_skip_flags_and_advances_without_replacing_existing_crop(self):
|
||||||
|
review = self.reviewer()
|
||||||
|
review.output_dir.mkdir()
|
||||||
|
previous = review.output_dir / "Copie01_01.jpg"
|
||||||
|
previous.write_bytes(b"previous crop")
|
||||||
|
review.handle_processing_result(review.current_result, self.files[0])
|
||||||
|
review.on_skip()
|
||||||
|
self.assertEqual(previous.read_bytes(), b"previous crop")
|
||||||
|
self.assertIn("Copie01.pdf", copy_errors(self.workspace))
|
||||||
|
self.assertEqual(review.index, 1)
|
||||||
|
self.assertEqual(review.current_max_per_file, 5)
|
||||||
|
self.assertEqual(review.current_width_offset, 0)
|
||||||
|
self.assertTrue(review.had_errors)
|
||||||
|
review.load_current_image.assert_called_once_with()
|
||||||
|
|
||||||
|
def test_accept_saves_and_clears_only_current_flag(self):
|
||||||
|
for path in self.files:
|
||||||
|
mark_copy_error(self.workspace, path, "Review needed")
|
||||||
|
review = self.reviewer()
|
||||||
|
review.on_next(None)
|
||||||
|
self.assertTrue((review.output_dir / "Copie01_01.jpg").is_file())
|
||||||
|
self.assertEqual(set(copy_errors(self.workspace)), {"Copie02.pdf"})
|
||||||
|
self.assertEqual(review.index, 1)
|
||||||
|
|
||||||
|
def test_failed_save_and_window_close_preserve_flag(self):
|
||||||
|
mark_copy_error(self.workspace, self.files[0], "Review needed")
|
||||||
|
review = self.reviewer()
|
||||||
|
with patch.object(cutleft, "save_results", side_effect=OSError("disk full")), patch.object(
|
||||||
|
cutleft.messagebox, "showerror"
|
||||||
|
):
|
||||||
|
review.on_next(None)
|
||||||
|
self.assertEqual(review.index, 0)
|
||||||
|
self.assertIn("Copie01.pdf", copy_errors(self.workspace))
|
||||||
|
review.on_close()
|
||||||
|
self.assertFalse(review.completed)
|
||||||
|
self.assertTrue(review.stop_prefetch.is_set())
|
||||||
|
self.assertIn("Copie01.pdf", copy_errors(self.workspace))
|
||||||
|
|
||||||
|
def test_enlarge_reprocesses_with_a_wider_selection(self):
|
||||||
|
review = self.reviewer()
|
||||||
|
review.trigger_processing = Mock()
|
||||||
|
|
||||||
|
review.on_enlarge(cutleft.CROP_WIDTH_STEP)
|
||||||
|
|
||||||
|
self.assertEqual(review.current_width_offset, 50)
|
||||||
|
review.trigger_processing.assert_called_once_with(self.files[0], 50)
|
||||||
|
|
||||||
|
def test_process_single_pdf_enlarges_crop_and_caps_it_at_page_edge(self):
|
||||||
|
page = Image.new("RGB", (900, 300), "white")
|
||||||
|
with patch.object(cutleft, "get_pdf_pages", return_value=[page]):
|
||||||
|
regular = cutleft.process_single_pdf(self.files[0])
|
||||||
|
enlarged = cutleft.process_single_pdf(self.files[0], width_offset=200)
|
||||||
|
capped = cutleft.process_single_pdf(self.files[0], width_offset=1000)
|
||||||
|
fullpage = cutleft.process_single_pdf(
|
||||||
|
self.files[0], max_per_file=1, width_offset=200
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(regular[1][0].size, (300, 300))
|
||||||
|
self.assertEqual(enlarged[1][0].size, (500, 300))
|
||||||
|
self.assertEqual(capped[1][0].size, (800, 300))
|
||||||
|
self.assertEqual(fullpage[1][0].size, (900, 300))
|
||||||
|
|
||||||
|
def test_processing_blocks_skip_and_failed_conversion_is_flagged(self):
|
||||||
|
review = self.reviewer()
|
||||||
|
review.is_processing = True
|
||||||
|
review.on_skip()
|
||||||
|
self.assertEqual(review.index, 0)
|
||||||
|
self.assertEqual(copy_errors(self.workspace), {})
|
||||||
|
review.manual_queue = Queue()
|
||||||
|
review.manual_queue.put(None)
|
||||||
|
review.load_current_image.side_effect = lambda: setattr(review, "is_processing", True)
|
||||||
|
review.check_manual_queue(self.files[0])
|
||||||
|
self.assertIn("Copie01.pdf", copy_errors(self.workspace))
|
||||||
|
self.assertEqual(review.index, 1)
|
||||||
|
self.assertTrue(review.is_processing) # Still loading the next copy.
|
||||||
|
|
||||||
|
def test_cli_splitting_preserves_flags_and_cropping_reports_partial_or_interrupted(self):
|
||||||
|
mark_copy_error(self.workspace, self.files[1], "Wrong order")
|
||||||
|
with patch.object(page_splitter.tk, "Tk"), patch.object(page_splitter, "PDFPreviewer") as preview:
|
||||||
|
preview.return_value.failed = False
|
||||||
|
self.assertEqual(page_splitter.main([str(self.workspace.root), "--marked"]), 0)
|
||||||
|
self.assertEqual(preview.call_args.args[2], [self.workspace.original_copies_dir / "Copie02.pdf"])
|
||||||
|
self.assertIn("Copie02.pdf", copy_errors(self.workspace))
|
||||||
|
with patch.object(cutleft, "ImageReviewer") as reviewer:
|
||||||
|
for completed, errors, expected in ((True, True, ExitCode.PARTIAL),
|
||||||
|
(False, False, ExitCode.INTERRUPTED),
|
||||||
|
(True, False, ExitCode.SUCCESS)):
|
||||||
|
reviewer.return_value.completed = completed
|
||||||
|
reviewer.return_value.had_errors = errors
|
||||||
|
self.assertEqual(cutleft.main([str(self.workspace.root), "--marked"]), expected)
|
||||||
|
self.assertEqual(reviewer.call_args.args[0], [self.files[1]])
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(os.environ.get("DISPLAY"), "Tk requires a display")
|
||||||
|
class MarkedCopiesGuiTests(unittest.TestCase):
|
||||||
|
setUp = MarkedCopiesTests.setUp
|
||||||
|
|
||||||
|
def test_keyboard_skip_then_accept_and_retry_clears_flag(self):
|
||||||
|
real_tk = cutleft.tk.Tk
|
||||||
|
|
||||||
|
def review_with_keys(files, keys):
|
||||||
|
sent = []
|
||||||
|
|
||||||
|
def make_root():
|
||||||
|
root = real_tk()
|
||||||
|
|
||||||
|
def send_when_ready():
|
||||||
|
info = root.winfo_children()[-1].cget("text")
|
||||||
|
index = len(sent)
|
||||||
|
if index < len(keys) and info.startswith(f"[{index + 1}/{len(files)}]"):
|
||||||
|
root.focus_force()
|
||||||
|
sent.append(keys[index])
|
||||||
|
root.event_generate(keys[index])
|
||||||
|
if len(sent) < len(keys):
|
||||||
|
root.after(10, send_when_ready)
|
||||||
|
|
||||||
|
root.after(20, send_when_ready)
|
||||||
|
root.after(3000, root.destroy) # Bound a failed keyboard test.
|
||||||
|
return root
|
||||||
|
|
||||||
|
with patch.object(cutleft.tk, "Tk", side_effect=make_root), patch.object(
|
||||||
|
cutleft, "get_pdf_pages", return_value=[Image.new("RGB", (600, 300), "white")]
|
||||||
|
), patch.object(cutleft, "OUTPUT_SIZE", (400, 200)):
|
||||||
|
reviewer = cutleft.ImageReviewer(files, self.workspace.cutleft_dir)
|
||||||
|
self.assertEqual(sent, keys)
|
||||||
|
self.assertTrue(reviewer.completed)
|
||||||
|
return reviewer
|
||||||
|
|
||||||
|
first = review_with_keys(self.files, ["<KeyPress-s>", "<Return>"])
|
||||||
|
self.assertTrue(first.had_errors)
|
||||||
|
self.assertEqual(set(copy_errors(self.workspace)), {"Copie01.pdf"})
|
||||||
|
self.assertFalse((self.workspace.cutleft_dir / "Copie01_01.jpg").exists())
|
||||||
|
self.assertTrue((self.workspace.cutleft_dir / "Copie02_01.jpg").exists())
|
||||||
|
second = review_with_keys(marked_copy_paths(self.workspace), ["<Return>"])
|
||||||
|
self.assertFalse(second.had_errors)
|
||||||
|
self.assertEqual(copy_errors(self.workspace), {})
|
||||||
|
self.assertTrue((self.workspace.cutleft_dir / "Copie01_01.jpg").exists())
|
||||||
|
|
||||||
|
def test_both_buttons_run_marked_and_single_copy_target_clears_filter(self):
|
||||||
|
mark_copy_error(self.workspace, self.files[1], "Review needed")
|
||||||
|
app = CopienatorApp(Path.cwd(), False, self.workspace.root)
|
||||||
|
try:
|
||||||
|
app.update()
|
||||||
|
for ident, command in (("page_splitter", "page-split"), ("cutleft", "crop-labels")):
|
||||||
|
app.tree.selection_set(ident)
|
||||||
|
app.update()
|
||||||
|
self.assertIn("(1)", app.marked_copies_button.cget("text"))
|
||||||
|
with patch.object(app, "_run_current_step") as run:
|
||||||
|
app.marked_copies_button.invoke()
|
||||||
|
run.assert_called_once_with()
|
||||||
|
self.assertIn(command, app._make_command())
|
||||||
|
self.assertIn("--marked", app._make_command())
|
||||||
|
self.assertEqual(app.arg_vars["target"].get(), app._evaluation_arg())
|
||||||
|
app.copy_var.set("Copie01.pdf")
|
||||||
|
app._target_selected_copy()
|
||||||
|
self.assertNotIn("--marked", app._make_command())
|
||||||
|
self.assertIn(str(self.files[0]), app._make_command())
|
||||||
|
finally:
|
||||||
|
for callback in app.tk.splitlist(app.tk.call("after", "info")):
|
||||||
|
app.after_cancel(callback)
|
||||||
|
app.destroy()
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
|
import pymupdf
|
||||||
|
|
||||||
|
from copienator.commands.page_splitter import PDFPreviewer, PAGE_SPLITTER_KB
|
||||||
|
|
||||||
|
|
||||||
|
class ReversePagesTests(unittest.TestCase):
|
||||||
|
def test_reverse_restarts_with_fresh_settings_and_exports_reversed_pages(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
source = Path(temporary) / "copy.pdf"
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
for index in range(3):
|
||||||
|
page = document.new_page(width=200 + 20 * index, height=300)
|
||||||
|
page.insert_text((30, 30), f"Page {index + 1}")
|
||||||
|
document.save(source)
|
||||||
|
original = source.read_bytes()
|
||||||
|
preview = PDFPreviewer.__new__(PDFPreviewer)
|
||||||
|
preview.doc = pymupdf.open(source)
|
||||||
|
self.addCleanup(lambda: None if preview.doc.is_closed else preview.doc.close())
|
||||||
|
preview.current_page_index = 2
|
||||||
|
preview.page_settings = [{"keep": "none"}, {"keep": "left"}]
|
||||||
|
preview.current_rotation = 180
|
||||||
|
preview.file_rotation = 180
|
||||||
|
preview.global_rotation = 180
|
||||||
|
preview.processing = False
|
||||||
|
preview.load_page = Mock()
|
||||||
|
|
||||||
|
preview.reverse_pages()
|
||||||
|
|
||||||
|
self.assertEqual([page.get_text().strip() for page in preview.doc],
|
||||||
|
["Page 3", "Page 2", "Page 1"])
|
||||||
|
self.assertEqual(preview.current_page_index, 0)
|
||||||
|
self.assertEqual(preview.page_settings, [])
|
||||||
|
self.assertEqual(preview.current_line_x, 120)
|
||||||
|
self.assertEqual(preview.current_rotation, 0)
|
||||||
|
self.assertEqual((preview.file_rotation, preview.global_rotation), (180, 180))
|
||||||
|
preview.load_page.assert_called_once_with()
|
||||||
|
self.assertEqual(source.read_bytes(), original)
|
||||||
|
|
||||||
|
preview.reverse_pages()
|
||||||
|
self.assertEqual([page.get_text().strip() for page in preview.doc],
|
||||||
|
["Page 1", "Page 2", "Page 3"])
|
||||||
|
preview.reverse_pages()
|
||||||
|
|
||||||
|
preview.base_name = "copy"
|
||||||
|
preview.split_dir = Path(temporary) / "split"
|
||||||
|
preview.reorder_dir = Path(temporary) / "reorder"
|
||||||
|
preview.final_file = Path(temporary) / "result.pdf"
|
||||||
|
preview.output_dir = None
|
||||||
|
preview.page_settings = [
|
||||||
|
{"keep": "as_is", "rotation": 0, "line_x": page.rect.width / 2}
|
||||||
|
for page in preview.doc
|
||||||
|
]
|
||||||
|
preview.split_pdf()
|
||||||
|
preview.reorder_pdfs()
|
||||||
|
preview.concate_files()
|
||||||
|
with pymupdf.open(preview.final_file) as result:
|
||||||
|
self.assertEqual([page.get_text().strip() for page in result],
|
||||||
|
["Page 3", "Page 2", "Page 1"])
|
||||||
|
|
||||||
|
def test_single_page_can_restart_and_processing_ignores_shortcut(self):
|
||||||
|
preview = PDFPreviewer.__new__(PDFPreviewer)
|
||||||
|
preview.doc = pymupdf.open()
|
||||||
|
self.addCleanup(preview.doc.close)
|
||||||
|
preview.doc.new_page()
|
||||||
|
preview.processing = True
|
||||||
|
preview.load_page = Mock()
|
||||||
|
preview.reverse_pages()
|
||||||
|
preview.load_page.assert_not_called()
|
||||||
|
preview.processing = False
|
||||||
|
preview.reverse_pages()
|
||||||
|
self.assertEqual(preview.current_page_index, 0)
|
||||||
|
self.assertEqual(preview.page_settings, [])
|
||||||
|
preview.load_page.assert_called_once_with()
|
||||||
|
|
||||||
|
def test_shortcut_available_with_personal_configuration(self):
|
||||||
|
self.assertEqual(PAGE_SPLITTER_KB["reverse_pages"], "i")
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from copienator import prompting
|
||||||
|
|
||||||
|
|
||||||
|
class PromptingTests(unittest.TestCase):
|
||||||
|
def test_perspective_is_inserted_with_alternative_method_guidance(self) -> None:
|
||||||
|
with patch.object(
|
||||||
|
prompting, "get_label_text_content", return_value="Question"
|
||||||
|
), patch.object(
|
||||||
|
prompting, "get_label_sol_content", return_value="Solution"
|
||||||
|
), patch.object(
|
||||||
|
prompting, "get_label_persp_content", return_value="Barème détaillé"
|
||||||
|
):
|
||||||
|
prompt = prompting.make_prompt("evaluation", "Ex 1")
|
||||||
|
|
||||||
|
self.assertIn(prompting.PERSPECTIVE_GUIDANCE, prompt)
|
||||||
|
self.assertIn("Barème détaillé", prompt)
|
||||||
|
|
||||||
|
def test_alternative_method_guidance_is_omitted_without_perspective(self) -> None:
|
||||||
|
with patch.object(
|
||||||
|
prompting, "get_label_text_content", return_value="Question"
|
||||||
|
), patch.object(
|
||||||
|
prompting, "get_label_sol_content", return_value="Solution"
|
||||||
|
), patch.object(prompting, "get_label_persp_content", return_value=None):
|
||||||
|
prompt = prompting.make_prompt("evaluation", "Ex 1")
|
||||||
|
|
||||||
|
self.assertNotIn(prompting.PERSPECTIVE_GUIDANCE, prompt)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import itertools
|
||||||
|
import runpy
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from copienator import EvaluationWorkspace, atomic_write_json, configuration
|
||||||
|
from copienator.commands import clean, giving_names
|
||||||
|
|
||||||
|
|
||||||
|
class ReturnOutputTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.workspace = EvaluationWorkspace(Path(self.temp.name))
|
||||||
|
self.workspace.copies_dir.mkdir()
|
||||||
|
atomic_write_json(self.workspace.copies_dir / "Copie01.json", {"name": "Student"})
|
||||||
|
(self.workspace.copies_dir / "Copie01.pdf").write_bytes(b"original")
|
||||||
|
atomic_write_json(self.workspace.correction_file, {})
|
||||||
|
(self.workspace.root / "names").write_text("Student\n")
|
||||||
|
self.source = self.workspace.root / "BGnot" / "Copie01"
|
||||||
|
self.source.mkdir(parents=True)
|
||||||
|
(self.source / "Concat.jpg").write_bytes(b"jpeg")
|
||||||
|
(self.source / "Concat_F.pdf").write_bytes(b"pdf")
|
||||||
|
atomic_write_json(self.source / "score.json", {"Ex 1": "4"})
|
||||||
|
atomic_write_json(self.source / "info.json", {
|
||||||
|
"Ex 1": {"present": True, "not_empty": True, "touched": True, "score": "4"}
|
||||||
|
})
|
||||||
|
self.answer_option = patch.object(configuration, "RETURN_ANSWERS_ENABLED", False)
|
||||||
|
self.answer_option.start()
|
||||||
|
self.addCleanup(self.answer_option.stop)
|
||||||
|
self.destination = self.workspace.return_dir / "Student (01)"
|
||||||
|
|
||||||
|
def prepare(self, jpeg=True, pdf=True):
|
||||||
|
with patch.object(configuration, "RETURN_JPEG_ENABLED", jpeg), patch.object(
|
||||||
|
configuration, "RETURN_PDF_ENABLED", pdf
|
||||||
|
), contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
self.assertEqual(giving_names.run(self.workspace, annotation_dir="BGnot"), 0)
|
||||||
|
|
||||||
|
def test_all_combinations_and_reenable_preserve_sources_and_scores(self):
|
||||||
|
for jpeg, pdf in itertools.product((True, False), repeat=2):
|
||||||
|
with self.subTest(jpeg=jpeg, pdf=pdf):
|
||||||
|
self.prepare()
|
||||||
|
self.prepare(jpeg, pdf)
|
||||||
|
expected = {"score.json", "info.json"}
|
||||||
|
if jpeg:
|
||||||
|
expected.add("Student.jpg")
|
||||||
|
if pdf:
|
||||||
|
expected.add("Student.pdf")
|
||||||
|
self.assertEqual({p.name for p in self.destination.iterdir()}, expected)
|
||||||
|
self.assertEqual((self.source / "Concat.jpg").read_bytes(), b"jpeg")
|
||||||
|
self.assertEqual((self.source / "Concat_F.pdf").read_bytes(), b"pdf")
|
||||||
|
self.assertEqual(
|
||||||
|
(self.destination / "score.json").read_bytes(),
|
||||||
|
(self.source / "score.json").read_bytes(),
|
||||||
|
)
|
||||||
|
self.prepare()
|
||||||
|
self.assertEqual((self.destination / "Student.jpg").read_bytes(), b"jpeg")
|
||||||
|
self.assertEqual((self.destination / "Student.pdf").read_bytes(), b"pdf")
|
||||||
|
|
||||||
|
def test_disabling_removes_regular_files_and_broken_links_only(self):
|
||||||
|
self.prepare()
|
||||||
|
jpg = self.destination / "Student.jpg"
|
||||||
|
jpg.unlink()
|
||||||
|
jpg.write_bytes(b"old fallback copy")
|
||||||
|
pdf = self.destination / "Student.pdf"
|
||||||
|
pdf.unlink()
|
||||||
|
pdf.symlink_to(self.source / "missing.pdf")
|
||||||
|
unrelated = self.destination / "notes.txt"
|
||||||
|
unrelated.write_text("keep")
|
||||||
|
self.prepare(False, False)
|
||||||
|
self.assertEqual({p.name for p in self.destination.iterdir()}, {"score.json", "info.json", "notes.txt"})
|
||||||
|
self.assertTrue((self.source / "Concat_F.pdf").is_file())
|
||||||
|
|
||||||
|
def test_fallback_source_respects_options(self):
|
||||||
|
self.source.parent.rename(self.workspace.root / "Anot")
|
||||||
|
(self.workspace.root / "BGnot").mkdir()
|
||||||
|
self.prepare(False, True)
|
||||||
|
self.assertFalse((self.destination / "Student.jpg").exists())
|
||||||
|
self.assertEqual((self.destination / "Student.pdf").read_bytes(), b"pdf")
|
||||||
|
self.assertTrue((self.destination / "score.json").is_file())
|
||||||
|
|
||||||
|
self.assertTrue((self.destination / "info.json").is_file())
|
||||||
|
|
||||||
|
def test_cleanup_preserves_enabled_returns_without_jpeg(self):
|
||||||
|
self.prepare(False, True)
|
||||||
|
with patch.object(configuration, "RETURN_JPEG_ENABLED", False):
|
||||||
|
plan = clean.build_cleanup_plan(self.workspace)
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
clean.apply_cleanup(self.workspace, plan)
|
||||||
|
self.assertFalse(self.source.exists())
|
||||||
|
self.assertEqual((self.destination / "Student.pdf").read_bytes(), b"pdf")
|
||||||
|
self.assertFalse((self.destination / "Student.pdf").is_symlink())
|
||||||
|
self.assertTrue((self.destination / "score.json").is_file())
|
||||||
|
self.assertTrue((self.destination / "info.json").is_file())
|
||||||
|
|
||||||
|
def test_cleanup_accepts_scores_only_but_still_requires_scores(self):
|
||||||
|
self.prepare(False, False)
|
||||||
|
with patch.object(configuration, "RETURN_JPEG_ENABLED", False):
|
||||||
|
plan = clean.build_cleanup_plan(self.workspace)
|
||||||
|
self.assertIn(self.destination / "score.json", plan.kept_files)
|
||||||
|
(self.destination / "score.json").unlink()
|
||||||
|
with self.assertRaisesRegex(clean.CliError, "score.json"):
|
||||||
|
clean.build_cleanup_plan(self.workspace)
|
||||||
|
|
||||||
|
def test_older_config_defaults_to_enabled(self):
|
||||||
|
old_config = self.workspace.root / "old_config.py"
|
||||||
|
old_config.write_text("ALWAYS_CROP = False\n")
|
||||||
|
with patch.dict("os.environ", {"COPIENATOR_CONFIG": str(old_config)}):
|
||||||
|
loaded = runpy.run_path(configuration.__file__)
|
||||||
|
self.assertIs(loaded["RETURN_JPEG_ENABLED"], True)
|
||||||
|
self.assertIs(loaded["RETURN_PDF_ENABLED"], True)
|
||||||
|
self.assertIs(loaded["RETURN_ANSWERS_ENABLED"], False)
|
||||||
|
self.assertIs(loaded["RETURN_ANSWERS_CONTEXT"], False)
|
||||||
|
self.assertIs(loaded["RETURN_ANSWERS_QUESTION"], True)
|
||||||
|
self.assertIs(loaded["RETURN_ANSWERS_SOLUTION"], False)
|
||||||
|
self.assertNotIn("RETURN_JSON_ENABLED", loaded)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
"""Compare extracted answers against the same Poppler view used for labels."""
|
||||||
|
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pymupdf
|
||||||
|
from pdf2image import convert_from_path
|
||||||
|
|
||||||
|
from copienator.commands.splitting_int import (
|
||||||
|
ANSWER_TOP_PADDING_POINTS,
|
||||||
|
_render_split_outputs,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SplittingGeometryTests(unittest.TestCase):
|
||||||
|
def make_source(self, path, rotation, cropped=True):
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
page = document.new_page(width=800, height=1000)
|
||||||
|
# Asymmetric colours exercise both translation and orientation.
|
||||||
|
for row in range(20):
|
||||||
|
for column in range(8):
|
||||||
|
colour = (row / 20, column / 8, (row + column) % 7 / 7)
|
||||||
|
page.draw_rect(
|
||||||
|
pymupdf.Rect(column * 100, row * 50,
|
||||||
|
(column + 1) * 100, (row + 1) * 50),
|
||||||
|
color=None, fill=colour,
|
||||||
|
)
|
||||||
|
page.insert_text((200, 500), "Middle of the answer")
|
||||||
|
if cropped:
|
||||||
|
page.set_cropbox(pymupdf.Rect(20, 80, 760, 920))
|
||||||
|
page.set_rotation(rotation)
|
||||||
|
document.save(path)
|
||||||
|
|
||||||
|
def assert_matches_preview(self, answer, preview, bounds):
|
||||||
|
actual = np.asarray(convert_from_path(answer, dpi=72)[0]).astype(float)
|
||||||
|
expected = np.asarray(preview.crop(bounds)).astype(float)
|
||||||
|
self.assertEqual(actual.shape, expected.shape)
|
||||||
|
self.assertLess(np.abs(actual - expected).mean(), 0.5)
|
||||||
|
|
||||||
|
def test_cropped_and_uncropped_pages_at_every_rotation(self):
|
||||||
|
visible = {
|
||||||
|
0: (20, 80, 760, 920),
|
||||||
|
90: (80, 20, 920, 760),
|
||||||
|
180: (40, 80, 780, 920),
|
||||||
|
270: (80, 40, 920, 780),
|
||||||
|
}
|
||||||
|
for rotation in (0, 90, 180, 270):
|
||||||
|
for cropped in (False, True):
|
||||||
|
for full_answer in (False, True):
|
||||||
|
with self.subTest(rotation=rotation, cropped=cropped, full=full_answer):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
source = root / "copy.pdf"
|
||||||
|
self.make_source(source, rotation, cropped)
|
||||||
|
before = source.read_bytes()
|
||||||
|
preview = convert_from_path(source, dpi=72, use_cropbox=False)[0]
|
||||||
|
width, height = preview.size
|
||||||
|
bounds = visible[rotation] if cropped else (0, 0, width, height)
|
||||||
|
if full_answer:
|
||||||
|
coordinates = [("A", 0, 0, 10, 0, 100)]
|
||||||
|
else:
|
||||||
|
coordinates = [("A", 0, 250, 260, 0, 100),
|
||||||
|
("_", 0, 711, 721, 0, 100)]
|
||||||
|
answer_top = int(height // 4 - ANSWER_TOP_PADDING_POINTS)
|
||||||
|
bounds = (bounds[0], max(bounds[1], answer_top),
|
||||||
|
bounds[2], min(bounds[3], height * 3 // 4))
|
||||||
|
_render_split_outputs(source, coordinates, root)
|
||||||
|
self.assert_matches_preview(root / "A.pdf", preview, bounds)
|
||||||
|
self.assertEqual(source.read_bytes(), before)
|
||||||
|
|
||||||
|
def test_answer_continues_to_bottom_then_next_page(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
source = root / "copy.pdf"
|
||||||
|
first = root / "first.pdf"
|
||||||
|
second = root / "second.pdf"
|
||||||
|
self.make_source(first, 180)
|
||||||
|
self.make_source(second, 0)
|
||||||
|
with pymupdf.open() as document:
|
||||||
|
for path in (first, second):
|
||||||
|
with pymupdf.open(path) as part:
|
||||||
|
document.insert_pdf(part)
|
||||||
|
document.save(source)
|
||||||
|
previews = convert_from_path(source, dpi=72, use_cropbox=False)
|
||||||
|
_render_split_outputs(source, [("A", 0, 700, 720, 0, 100),
|
||||||
|
("_", 1, 211, 230, 500, 600)], root)
|
||||||
|
rendered = convert_from_path(root / "A.pdf", dpi=72)
|
||||||
|
self.assertEqual(len(rendered), 2)
|
||||||
|
for actual, preview, bounds in zip(rendered, previews,
|
||||||
|
[(40, 688, 780, 920), (20, 80, 760, 250)]):
|
||||||
|
expected = np.asarray(preview.crop(bounds)).astype(float)
|
||||||
|
actual = np.asarray(actual).astype(float)
|
||||||
|
self.assertEqual(actual.shape, expected.shape)
|
||||||
|
self.assertLess(np.abs(actual - expected).mean(), 0.5)
|
||||||
@@ -0,0 +1,221 @@
|
|||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from tkinter import ttk
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from copienator import CliError, EvaluationWorkspace, ExitCode
|
||||||
|
from copienator.commands import enonce_info as personal
|
||||||
|
from copienator.commands import gemini_for_enonce as gemini
|
||||||
|
from copienator_gui.app import CopienatorApp, get_personal_interro_files
|
||||||
|
from copienator_gui.workflow import build_workflow
|
||||||
|
|
||||||
|
|
||||||
|
class PersonalStatementTests(unittest.TestCase):
|
||||||
|
def test_personal_generation_uses_statement_and_groups_by_exercise(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
root = Path(temporary)
|
||||||
|
(root / "correction.tex").write_text("Not SHEETINFO")
|
||||||
|
blocks = [{"id": 42, "indexes": [{"indices": [1]}, {"indices": [2]}]},
|
||||||
|
{"id": 99}]
|
||||||
|
(root / "enonce.tex").write_text("\n".join(
|
||||||
|
"%%SHEETINFO : " + json.dumps(block) for block in blocks))
|
||||||
|
(root / "label_groups").write_text("obsolete groups")
|
||||||
|
urls = []
|
||||||
|
|
||||||
|
def fetch(url):
|
||||||
|
urls.append(url)
|
||||||
|
if "/emacs/" in url:
|
||||||
|
return io.BytesIO(b"Statement\n 1) Question\n###Solution\n 1) Answer\n###Rubric\n 1) Points")
|
||||||
|
return io.BytesIO(b"Selected exercise content")
|
||||||
|
|
||||||
|
def compile_pdf(content, path):
|
||||||
|
Path(path).write_bytes(b"test PDF")
|
||||||
|
|
||||||
|
with patch.object(personal.urllib.request, "urlopen", side_effect=fetch), patch.object(
|
||||||
|
personal, "compile_to_pdf", side_effect=compile_pdf
|
||||||
|
):
|
||||||
|
self.assertEqual(personal.process_directory(EvaluationWorkspace(root)), ExitCode.SUCCESS)
|
||||||
|
self.assertEqual((root / "labels").read_text(), "Ex 1 : 1)\nEx 1 : 2)\nEx 2\n")
|
||||||
|
self.assertEqual((root / "label_groups").read_text(), "Ex 1 : 1), Ex 1 : 2)\nEx 2\n")
|
||||||
|
for folder in ("Text2", "Sol2"):
|
||||||
|
for label in ("Ex 1 : 1)", "Ex 1 : 2)", "Ex 2"):
|
||||||
|
self.assertTrue((root / folder / f"{label}.tex").is_file())
|
||||||
|
self.assertTrue((root / folder / f"{label}.pdf").is_file())
|
||||||
|
self.assertIn("Rubric", (root / "Persp" / "Ex 1").read_text())
|
||||||
|
self.assertTrue(any("/exo_q_text/42/1" in url for url in urls))
|
||||||
|
|
||||||
|
def test_personal_choice_and_optional_steps_only_in_personal_profile(self):
|
||||||
|
standard = {step.id: step for step in build_workflow(False)}
|
||||||
|
enabled = {step.id: step for step in build_workflow(True)}
|
||||||
|
self.assertEqual([variant.id for variant in standard["statement"].variants], ["gemini"])
|
||||||
|
self.assertEqual([variant.program for variant in enabled["statement"].variants],
|
||||||
|
["statement-personal", "statement"])
|
||||||
|
for ident in ("statement_groups", "statement_persp"):
|
||||||
|
self.assertNotIn(ident, standard)
|
||||||
|
self.assertTrue(enabled[ident].optional)
|
||||||
|
self.assertFalse(enabled[ident].auto_start_first_visit)
|
||||||
|
|
||||||
|
|
||||||
|
class SelectiveGeminiTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temp.cleanup)
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
self.workspace = EvaluationWorkspace(self.root)
|
||||||
|
self.labels = ["Ex 1 : 1)", "Ex 1 : 2)", "Ex 2"]
|
||||||
|
(self.root / "labels").write_text("\n".join(self.labels) + "\n")
|
||||||
|
(self.root / "label_groups").write_text(", ".join(self.labels[:2]) + "\nEx 2\n")
|
||||||
|
for folder in ("Text", "Sol", "Text2", "Sol2", "Persp"):
|
||||||
|
(self.root / folder).mkdir()
|
||||||
|
for label in self.labels:
|
||||||
|
suffix = ".tex" if folder.endswith("2") else ""
|
||||||
|
(self.root / folder / (label + suffix)).write_text(f"Personal {folder}: {label}")
|
||||||
|
(self.root / "Persp" / "Ex 1").write_text("Old aggregate rubric")
|
||||||
|
self.client = SimpleNamespace(models=SimpleNamespace(generate_content=Mock()))
|
||||||
|
|
||||||
|
def snapshot(self):
|
||||||
|
return {str(path.relative_to(self.root)): path.read_bytes()
|
||||||
|
for path in self.root.rglob("*") if path.is_file()}
|
||||||
|
|
||||||
|
def response(self, value):
|
||||||
|
return SimpleNamespace(text=json.dumps(value))
|
||||||
|
|
||||||
|
def test_grouping_changes_only_groups_and_keeps_exact_labels(self):
|
||||||
|
before = self.snapshot()
|
||||||
|
self.client.models.generate_content.return_value = self.response({"groups": [[label] for label in self.labels]})
|
||||||
|
self.assertEqual(gemini.refine_existing(self.workspace, "groups", api_client=self.client), ExitCode.SUCCESS)
|
||||||
|
after = self.snapshot()
|
||||||
|
self.assertEqual(after.pop("label_groups"), ("\n".join(self.labels) + "\n").encode())
|
||||||
|
before.pop("label_groups")
|
||||||
|
self.assertEqual(after, before)
|
||||||
|
|
||||||
|
def test_invalid_grouping_preserves_existing_groups(self):
|
||||||
|
before = self.snapshot()
|
||||||
|
for groups in ([[self.labels[0]]], [[*self.labels, self.labels[0]]], [[*self.labels, "invented"]]):
|
||||||
|
self.client.models.generate_content.return_value = self.response({"groups": groups})
|
||||||
|
with self.assertRaises(CliError):
|
||||||
|
gemini.refine_existing(self.workspace, "groups", api_client=self.client)
|
||||||
|
self.assertEqual(self.snapshot(), before)
|
||||||
|
|
||||||
|
def rubric_response(self, labels):
|
||||||
|
return self.response({"rubrics": [{"label": label, "rubric_content": "Barème Gemini sur 4 points"}
|
||||||
|
for label in labels]})
|
||||||
|
|
||||||
|
def test_rubrics_replace_only_persp_using_normal_prompt(self):
|
||||||
|
before = self.snapshot()
|
||||||
|
self.client.models.generate_content.side_effect = [self.rubric_response(self.labels[:2]),
|
||||||
|
self.rubric_response(self.labels[2:])]
|
||||||
|
self.assertEqual(gemini.refine_existing(self.workspace, "persp", api_client=self.client), ExitCode.SUCCESS)
|
||||||
|
after = self.snapshot()
|
||||||
|
self.assertEqual({key: value for key, value in before.items() if not key.startswith("Persp/")},
|
||||||
|
{key: value for key, value in after.items() if not key.startswith("Persp/")})
|
||||||
|
self.assertFalse((self.root / "Persp" / "Ex 1").exists())
|
||||||
|
for label in self.labels:
|
||||||
|
self.assertIn("Barème Gemini", (self.root / "Persp" / label).read_text())
|
||||||
|
for call in self.client.models.generate_content.call_args_list:
|
||||||
|
self.assertEqual(call.kwargs["contents"][0].parts[0].text, gemini.PROMPT_4)
|
||||||
|
self.assertTrue(call.kwargs["config"].automatic_function_calling.disable)
|
||||||
|
|
||||||
|
def test_rubric_prompt_omits_the_assumed_total_and_requires_latex(self):
|
||||||
|
self.assertIn("total est toujours implicite", gemini.PROMPT_4)
|
||||||
|
self.assertIn("caractères mathématiques Unicode", gemini.PROMPT_4)
|
||||||
|
self.assertIn(r"$\lfloor \sqrt{k} \rfloor$", gemini.PROMPT_4)
|
||||||
|
|
||||||
|
def test_incomplete_or_failed_rubrics_preserve_entire_persp(self):
|
||||||
|
before = self.snapshot()
|
||||||
|
for last in (self.response({"rubrics": []}), RuntimeError("API unavailable")):
|
||||||
|
self.client.models.generate_content.side_effect = [self.rubric_response(self.labels[:2]), last]
|
||||||
|
with self.assertRaises((CliError, RuntimeError)):
|
||||||
|
gemini.refine_existing(self.workspace, "persp", api_client=self.client)
|
||||||
|
self.assertEqual(self.snapshot(), before)
|
||||||
|
|
||||||
|
def test_cli_dispatches_selective_modes_without_full_extraction(self):
|
||||||
|
with patch.object(gemini, "refine_existing", return_value=ExitCode.SUCCESS) as refine, patch.object(
|
||||||
|
gemini, "process_exam"
|
||||||
|
) as full:
|
||||||
|
for flag, mode in (("--groups-only", "groups"), ("--persp-only", "persp")):
|
||||||
|
self.assertEqual(gemini.main([str(self.root), flag]), ExitCode.SUCCESS)
|
||||||
|
self.assertEqual(refine.call_args.args[1], mode)
|
||||||
|
full.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(os.environ.get("DISPLAY"), "Tk requires a display")
|
||||||
|
class PersonalStatementGuiTests(unittest.TestCase):
|
||||||
|
def test_get_file_button_completes_inputs_and_advances(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
repository = Path(temporary)
|
||||||
|
evaluation = repository / "Interro12"
|
||||||
|
source = repository / "source"
|
||||||
|
evaluation.mkdir()
|
||||||
|
source.mkdir()
|
||||||
|
(repository / "names").touch()
|
||||||
|
for name, content in (
|
||||||
|
("Interro12.pdf", b"pdf"),
|
||||||
|
("Interro12.tex", b"statement"),
|
||||||
|
("Interro12c.tex", b"correction"),
|
||||||
|
):
|
||||||
|
(source / name).write_bytes(content)
|
||||||
|
app = CopienatorApp(repository, True, evaluation)
|
||||||
|
try:
|
||||||
|
app.update()
|
||||||
|
buttons = [
|
||||||
|
child
|
||||||
|
for child in app.form.winfo_children()
|
||||||
|
if isinstance(child, ttk.Button)
|
||||||
|
]
|
||||||
|
get_button = next(
|
||||||
|
button for button in buttons if button.cget("text") == "Get the files"
|
||||||
|
)
|
||||||
|
with patch(
|
||||||
|
"copienator_gui.app.get_personal_interro_files",
|
||||||
|
side_effect=lambda target: get_personal_interro_files(target, source),
|
||||||
|
):
|
||||||
|
get_button.invoke()
|
||||||
|
app.update()
|
||||||
|
|
||||||
|
self.assertEqual(app.state_store.step("inputs")["status"], "success")
|
||||||
|
self.assertEqual(app.current_step.id, "statement")
|
||||||
|
self.assertEqual((evaluation / "enonce.pdf").read_bytes(), b"pdf")
|
||||||
|
self.assertEqual((evaluation / "enonce.tex").read_bytes(), b"statement")
|
||||||
|
self.assertEqual((evaluation / "correction.tex").read_bytes(), b"correction")
|
||||||
|
finally:
|
||||||
|
for callback in app.tk.splitlist(app.tk.call("after", "info")):
|
||||||
|
app.after_cancel(callback)
|
||||||
|
app.destroy()
|
||||||
|
|
||||||
|
def test_personal_requirements_and_optional_button_command_previews(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
evaluation = Path(temporary)
|
||||||
|
(evaluation / "enonce.tex").touch()
|
||||||
|
app = CopienatorApp(Path.cwd(), True, evaluation)
|
||||||
|
try:
|
||||||
|
app.update()
|
||||||
|
app.tree.selection_set("statement")
|
||||||
|
app.update()
|
||||||
|
self.assertEqual(app.variant_var.get(), "personal")
|
||||||
|
self.assertIn("statement-personal", app.command_var.get())
|
||||||
|
self.assertEqual(app._missing_requirements(app.current_step), [])
|
||||||
|
self.assertIn("SHEETINFO", app.description_label.cget("text"))
|
||||||
|
self.assertFalse(
|
||||||
|
any(
|
||||||
|
isinstance(child, ttk.LabelFrame)
|
||||||
|
and child.cget("text") == "Après génération — facultatif"
|
||||||
|
for child in app.form.winfo_children()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for ident, flag in (("statement_groups", "--groups-only"), ("statement_persp", "--persp-only")):
|
||||||
|
app.tree.selection_set(ident)
|
||||||
|
app.update()
|
||||||
|
self.assertEqual(app.current_step.id, ident)
|
||||||
|
self.assertIn(flag, app.command_var.get())
|
||||||
|
self.assertTrue(app.current_step.optional)
|
||||||
|
self.assertFalse(app.runner.running)
|
||||||
|
finally:
|
||||||
|
for callback in app.tk.splitlist(app.tk.call("after", "info")):
|
||||||
|
app.after_cancel(callback)
|
||||||
|
app.destroy()
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from copienator import utils
|
||||||
|
|
||||||
|
|
||||||
|
class CompileToPdfTests(unittest.TestCase):
|
||||||
|
def test_warns_when_latex_fails_even_if_partial_pdf_exists(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
output = Path(temporary) / "result.pdf"
|
||||||
|
|
||||||
|
def failed_run(*_args, **kwargs):
|
||||||
|
(Path(kwargs["cwd"]) / "text.pdf").write_bytes(b"partial")
|
||||||
|
return subprocess.CompletedProcess(
|
||||||
|
args=[], returncode=1,
|
||||||
|
stdout="! LaTeX Error: Something's wrong--perhaps a missing \\item.\n",
|
||||||
|
)
|
||||||
|
|
||||||
|
console = io.StringIO()
|
||||||
|
with patch.object(utils.subprocess, "run", side_effect=failed_run), \
|
||||||
|
contextlib.redirect_stdout(console):
|
||||||
|
utils.compile_to_pdf("broken", output)
|
||||||
|
|
||||||
|
self.assertEqual(output.read_bytes(), b"partial")
|
||||||
|
self.assertIn("Warning: LaTeX compilation failed", console.getvalue())
|
||||||
|
self.assertIn("missing \\item", console.getvalue())
|
||||||
|
|
||||||
|
def test_warns_when_no_pdf_is_produced(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
output = Path(temporary) / "result.pdf"
|
||||||
|
result = subprocess.CompletedProcess(args=[], returncode=0, stdout="")
|
||||||
|
|
||||||
|
console = io.StringIO()
|
||||||
|
with patch.object(utils.subprocess, "run", return_value=result), \
|
||||||
|
contextlib.redirect_stdout(console):
|
||||||
|
utils.compile_to_pdf("valid", output)
|
||||||
|
|
||||||
|
self.assertFalse(output.exists())
|
||||||
|
self.assertIn("produced no PDF", console.getvalue())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user