# AI-generated weak-spot training evaluation

Date: 2026-08-10

Generated source CSV:

```text
/root/meeting-minutes-evidence-model/source_uploads/meeting_minutes_evidence_training_ai_weak_spots_20260810.csv
```

Converted training file:

```text
/root/meeting-minutes-evidence-model/data/training_data_hard_negatives_plus_ai_weak_spots.csv
```

Model:

```text
/root/meeting-minutes-evidence-model/runs/ai-weak-spots-20260810T202748Z/model_bundle/classifier.joblib
```

## Generated data

130 targeted examples were generated:

- 35 true confirmed actions, including action-table fragments and audit/risk
  planning actions.
- 25 process/sequence-not-action negatives.
- 20 record-location/status-not-action negatives.
- 20 Teams/logistics/low-value noise negatives.
- 10 decision/agreement negatives.
- 10 hypothetical near-action negatives.
- 10 completed-history contrasts.

After duplicate filtering against the hard-negative set, 122 rows were added.
The merged set has 369 rows.

## Metrics

Compared with the hard-negative model:

- Evidence-type accuracy improved from 0.48 to 0.53.
- Evidence-type macro F1 improved from 0.46 to 0.53.
- Commitment-state accuracy improved from 0.53 to 0.56.
- Commitment-state macro F1 improved from 0.49 to 0.50.
- Signal macro F1 improved from 0.714 to 0.752.

Notable label movement:

- `low_value_noise`: F1 0.88.
- `decision_language`: F1 0.586.
- `process_sequence`: F1 0.635.
- `completed_tense`: F1 0.602.
- `confirmed_action`: F1 0.65.

## Probe results

Improved:

- `Complete the risk assessment to develop the audit plan.` now has
  `confirmed_action` as top commitment state.
- `Share the risk analysis with Niamh before she arrives.` is strongly
  `action_commitment` / `confirmed_action`.
- `I’ll just share my screen...` is now strongly `low_value_noise` /
  `not_action`.
- `Maybe we could look at the label...` is now `not_action` with high
  `hypothetical_language`.
- `David reviewed Risk Mgmt Plan.` is now clearly `completed_history`.
- goods-flow wording remains `process_overview` / `not_action`.
- `The team agreed that Excel was acceptable...` remains decision/history rather
  than action.
- record-location wording now has `not_action` as top commitment state.

Still needs deterministic gates:

- Sequence-only wording such as `risk assessment comes before audit plan` now
  has `process_overview` as top evidence, but commitment state is still close.
  Suppress sequence-only rows with `comes before`, `follows`, `depends on`,
  `once`, `after`, `before` unless there is a direct owner assignment.
- Action-table fragments are improved but still close. Keep deterministic
  imperative/action-table preservation.
- Transcript ranking should penalise `low_value_noise` hard. The classifier
  labels screen-share/logistics correctly in isolation, but the probe ranker can
  still surface logistics when grammar hints fire.

## Standing

This model is now the best evidence-ledger candidate so far. It is reasonable to
wire into the staged minutes workflow as a classifier-assisted evidence ledger,
provided final action extraction still applies deterministic keep/suppress
rules.

