# Augmented meeting minutes evidence classifier evaluation

Date: 2026-08-10

User-supplied CSV:

```text
/root/meeting-minutes-evidence-model/source_uploads/meeting_minutes_evidence_training_first_pass.csv
```

Converted training file:

```text
/root/meeting-minutes-evidence-model/data/training_data_augmented_first_pass.csv
```

Model:

```text
/root/meeting-minutes-evidence-model/runs/augmented-first-pass-20260810T201310Z/model_bundle/classifier.joblib
```

## What changed

The upload added 115 examples. The merged training set now has 185 examples:

- `confirmed_action`: 60
- `not_action`: 70
- `completed_history`: 38
- `possible_action`: 17

This is much better coverage for the failure modes we care about:

- confirmed action versus hypothetical;
- confirmed action versus completed history;
- audit/risk-assessment wording;
- DITA regulatory/document-control wording;
- T761 software/risk-management wording.

## Cross-validation

Evidence type became noisier:

```text
accuracy: 0.41
macro F1: 0.37
weighted F1: 0.41
```

Commitment state remains modest but improved on the key contrast classes:

```text
accuracy: 0.48
macro F1: 0.44
weighted F1: 0.49
```

Signal detection improved materially:

```text
micro F1: 0.821
macro F1: 0.692
```

Important signal improvements:

- `hypothetical_language`: F1 0.634
- `completed_tense`: F1 0.532
- `explicit_commitment_verb`: F1 0.887
- `document_or_record`: F1 0.918
- `risk_language`: F1 0.884
- `regulatory_language`: F1 0.800

`decision_language` still has no useful CV performance.

## Probe verdict

Exact probes:

- `Share the risk analysis with Niamh before she arrives.` now scores as
  `action_commitment` / `confirmed_action`.
- `Orla to follow up with Cody for the MedEnvoy project plan...` now scores as
  `document_control_task` / `confirmed_action`.
- `Share list of languages for DoCs...` now scores as document-control action
  evidence with regulatory/document signals.
- `John-Paul will review DoCs...MDR and PPE` now scores as
  document/regulatory confirmed-action evidence.
- warehouse goods-flow wording scores strongly as `process_overview` /
  `not_action`.
- `Could David review Risk Mgmt Plan?` scores as `not_action` with
  `hypothetical_language`.
- `David reviewed Risk Mgmt Plan.` scores as `completed_history`.

Remaining exact-probe weakness:

- `Complete the risk assessment to develop the audit plan.` still leans toward
  `risk_dependency` / `possible_action`, although it has strong risk/document
  signals. This should be kept by a deterministic imperative/document-control
  rule until more audit-plan examples are added.
- Short action-table fragments such as `Review Rebecca's updates...` need an
  action-table/imperative rule. The classifier alone can confuse them with
  completed-update history.

Transcript probes:

- T796 Abbott: now surfaces the risk-analysis sharing, risk-assessment/audit-plan
  sentence, code of conduct, training attestation, SharePoint access and pre-audit
  meeting evidence.
- T819 DITA: now surfaces MedEnvoy project-plan follow-up, QMS manual context,
  DoC update/risk rationale, label/HPRA fee evidence, and goods-flow process
  detail.
- T761 Eakin: surfaces testing readiness, benefit-risk/risk-file detail, debug
  review, Colm standards review and software/process evidence.

## Where we stand

This is now worth integrating as an evidence-ledger input, but not as a standalone
final action selector.

Use it to improve staged meeting minutes by:

1. adding classifier evidence to the staged context;
2. applying deterministic gates for imperative/action-table fragments;
3. applying deterministic suppressions for completed-history, hypothetical
   language, screen-share/logistics and question-only lines;
4. letting Trooper write from this evidence rather than raw transcript alone.

## Needed next samples

Highest value next:

- 20-30 more audit/risk-assessment/audit-plan/action-plan examples.
- 20 short action-table imperatives versus completed-history contrasts.
- 20 question-only / logistics / screen-share negatives.
- 10-15 decision/agreement examples, because `decision_language` is still weak.
- A small validation-only holdout CSV that is not used for training.

