# Meeting minutes evidence classifier first-pass evaluation

Date: 2026-08-10

Run:

```text
/root/meeting-minutes-evidence-model/runs/first-pass-20260810T195343Z/model_bundle/classifier.joblib
```

Training data:

```text
70 reviewed seed examples
```

## Cross-validation

Evidence type:

```text
accuracy: 0.51
macro F1: 0.48
weighted F1: 0.51
```

Commitment state:

```text
accuracy: 0.53
macro F1: 0.37
weighted F1: 0.53
```

Signals:

```text
micro F1: 0.731
macro F1: 0.590
```

Strongest signal labels:

- `regulatory_language`
- `process_sequence`
- `document_or_record`
- `deliverable_object`
- `risk_language`
- `named_owner`

Weakest signal labels:

- `completed_tense`
- `decision_language`
- `hypothetical_language`
- `deadline_or_sequence`

## Exact-sample probes

The classifier correctly treats these as useful evidence:

- `Share the risk analysis with Niamh before she arrives.`
- `Orla to follow up with Cody for the MedEnvoy project plan for manufacturer EO.`
- `Share list of languages for DoCs by 24 March.`
- `John-Paul will review DoCs for sunglasses for MDR and PPE.`
- `The goods arrive from the manufacturer...picked for customer orders.`
- `Maybe we could look at the label in a later session.`
- `The HPRA fee has already been paid...sent yesterday.`

The main weak exact sample is:

```text
Complete the risk assessment to develop the audit plan.
```

It finds the risk/document/deadline signals, but currently leans toward
`risk_dependency` / `possible_action` rather than confidently classifying it as
a document-control action. Add more risk-assessment/audit-plan contrast pairs.

## Transcript probes

Real transcripts probed:

- T761 Eakin software weekly check-in
- T819 DITA importer obligations client connect
- T796 Abbott audit kickoff

The transcript probe needed a deterministic grammar layer to be useful. Raw
MiniLM scoring alone over-selected short conversational fragments and weak
planning language. After adding speaker/timestamp parsing plus owner/verb/object
hints, the probe surfaced the relevant evidence much better, including:

- T796: sharing risk analysis with Niamh before arrival;
- T796: code of conduct / training attestation requirements;
- T819: MedEnvoy project-plan follow-up;
- T819: QMS manual review context;
- T819: DoC update / risk rationale;
- T819: warehouse/picking/packing/distribution process overview;
- T761: testing readiness, debug output review, standards review and risk-file
  context.

## Verdict

Good first pass as an evidence highlighter, not ready as a standalone action
extractor.

Use it as:

```text
MiniLM evidence type + deterministic grammar + staged writer prompt context
```

Do not use it as:

```text
MiniLM classifier alone decides final action rows
```

## Needed next samples

Highest priority:

- 30-50 real action/non-action contrast pairs from Trinzo transcripts.
- More audit/risk-assessment/action-plan examples.
- More completed-history examples: already sent/reviewed/paid/completed versus
  will send/review/pay/complete.
- More decision/agreement examples versus review/discussion wording.
- More hedged planning negatives: maybe/could/might/should we/worth thinking.
- More one-word/noisy Teams fragments so transcript probing can suppress them.

Useful target labels to strengthen:

- `completed_tense`
- `decision_language`
- `hypothetical_language`
- `deadline_or_sequence`
- `confirmed_action` versus `possible_action`

