# Meeting minutes classifier repaired masking run

Run directory: `runs/multitask-rework-20260811T204433Z/model_bundle`

## Plain-English summary

- Evidence type: n/a before versus 34.9% on repaired test; similar by n/a.
- Action state: n/a before versus 49.7% on repaired test; similar by n/a.
- Signals: n/a before versus 66.1% on repaired test; similar by n/a.
- Quality gate: 74.4% test accuracy/micro-F1 and 74.4% macro-F1.
- Quality defects: 67.3% test micro-F1 and 52.8% macro-F1.

## Before and after metrics

| Head | Before | Repaired test | Change |
| --- | ---: | ---: | ---: |
| Evidence type macro-F1 | n/a | 34.9% | n/a |
| Action state macro-F1 | n/a | 49.7% | n/a |
| Signals macro-F1 | n/a | 66.1% | n/a |
| Quality gate macro-F1 | n/a | 74.4% | new |
| Quality gate micro-F1 | n/a | 74.4% | new |
| Quality defects macro-F1 | n/a | 52.8% | new |
| Quality defects micro-F1 | n/a | 67.3% | new |

## Quality-gate confusion matrix

| Actual \ predicted | intervention | ready |
| --- | ---: | ---: |
| intervention | 17 | 2 |
| ready | 9 | 15 |

## Weakest classes and labels

- evidence_type / low_value_noise: 0.0% F1 on 0 test examples.
- evidence_type / regulatory_obligation: 0.0% F1 on 4 test examples.
- quality_defects / fragment_or_false_start: 0.0% F1 on 1 test examples.
- evidence_type / process_overview: 14.3% F1 on 3 test examples.
- signals / regulatory_language: 16.0% F1 on 8 test examples.
- action_state / possible_action: 17.4% F1 on 15 test examples.
- evidence_type / document_control_task: 28.6% F1 on 13 test examples.
- evidence_type / action_commitment: 34.5% F1 on 16 test examples.
- signals / recipient_or_handover: 34.8% F1 on 18 test examples.
- evidence_type / macro avg: 34.9% F1 on 92 test examples.
- signals / process_sequence: 44.4% F1 on 6 test examples.
- evidence_type / risk_dependency: 45.5% F1 on 12 test examples.
- signals / deadline_or_sequence: 46.0% F1 on 20 test examples.
- evidence_type / weighted avg: 47.2% F1 on 92 test examples.
- action_state / macro avg: 49.7% F1 on 92 test examples.

## Suitability recommendation

| Head | Diagnostics | Rewrite warnings | Human-review routing | Automatic approval | Automatic deletion |
| --- | --- | --- | --- | --- | --- |
| Evidence type | yes | supporting only | yes | no | no |
| Action state | yes | supporting only | yes | no | no |
| Signals | yes | supporting only | yes | no | no |
| Quality gate | yes | yes | yes | no | no |
| Quality defects | yes | yes | yes | no | no |
| Faithfulness | provisional only | no | possible later | no | no |

## Highest-value next examples

Prioritise 50-100 examples that cover:

1. Real generated minutes rows that look fluent but are vague, especially `topic_only_or_missing_substance`.
2. Real generated minutes rows with reported speech: `said that`, `mentioned that`, `there was a discussion around`.
3. Short false starts and chairing phrases from different meeting types.
4. Good human-minute rows with first names in the body only, so the gate learns that names are not the defect.
5. Negative faithfulness pairs where the candidate is plausible but loses a deadline, owner, decision or constraint.
6. More decision-language examples; this remains a weak old-head signal.
7. More risk/process examples, because those dipped or remain fragile.

## Implementation issues and assumptions

- The repaired run uses MiniLM exactly, so this isolates schema repair rather than embedding-model changes.
- Faithfulness labels remain provisional; the pair test split currently lacks unsupported examples, so faithfulness metrics are not production evidence.
- No live minutes-generation behaviour has been changed.
