# Hard-negative meeting minutes evidence classifier evaluation

Date: 2026-08-10

User-supplied CSV:

```text
/root/meeting-minutes-evidence-model/source_uploads/meeting_minutes_evidence_training_with_hard_negatives.csv
```

Converted training file:

```text
/root/meeting-minutes-evidence-model/data/training_data_with_hard_negatives.csv
```

Model:

```text
/root/meeting-minutes-evidence-model/runs/hard-negatives-20260810T201724Z/model_bundle/classifier.joblib
```

## What changed

The second upload contains 177 examples, including the previous action/contrast
examples plus 62 hard negatives. The merged training set now has 247 rows.

Commitment-state balance:

- `not_action`: 100
- `confirmed_action`: 60
- `completed_history`: 57
- `possible_action`: 30

## Metrics

Compared with the previous augmented model:

- Evidence-type accuracy improved from 0.41 to 0.48.
- Evidence-type macro F1 improved from 0.37 to 0.46.
- Commitment-state accuracy improved from 0.48 to 0.53.
- Commitment-state macro F1 improved from 0.44 to 0.49.
- Signal macro F1 improved from 0.692 to 0.714.

Important signal results:

- `hypothetical_language`: F1 0.708
- `completed_tense`: F1 0.551
- `decision_language`: F1 0.316, no longer zero
- `deadline_or_sequence`: F1 0.619
- `document_or_record`: F1 0.921
- `risk_language`: F1 0.833

## Probe verdict

Improved:

- `Share the risk analysis with Niamh before she arrives.` is now
  `action_commitment` / `confirmed_action`.
- `Orla to follow up with Cody for the MedEnvoy project plan...` remains
  `document_control_task` / `confirmed_action`.
- `Share list of languages for DoCs...` remains document-control action
  evidence.
- `John-Paul will review DoCs...MDR and PPE` remains document/regulatory action
  evidence.
- goods-flow wording remains strong `process_overview` / `not_action`.
- `David reviewed Risk Mgmt Plan.` is `completed_history`.
- `The team agreed that Excel was acceptable...` is now decision/history style,
  not confirmed action.

Still weak:

- `Complete the risk assessment to develop the audit plan.` still leans toward
  risk/process rather than clean document-control action. It should be preserved
  by a deterministic imperative/document-control rule.
- `The risk assessment comes before the audit plan...` still leans too much
  toward `possible_action`, despite process-sequence signal. Needs a deterministic
  sequence-vs-assignment suppressor.
- `The SharePoint contains the existing procedure documentation.` is partly
  improved but still too document-task-like. Needs a record-location suppressor.
- `Maybe we could look at the label...` has the right `hypothetical_language`
  signal but can still score as `possible_action`; final gates should suppress
  high-hypothetical/low-commitment cases.
- transcript probes still surface logistics such as screen sharing and holiday
  comments; the probe/ranking layer needs a logistics/noise suppressor.

## Where we stand

The hard-negative model is the best candidate so far for integration as a
meeting-minutes evidence ledger.

It should feed:

```text
evidence type + commitment state + signals + source text
```

into the staged workflow, but the final action inventory should still apply
deterministic gates:

- keep imperative/action-table rows when object and verb are strong;
- keep transcript actions with owner + commitment verb + deliverable;
- suppress completed-history;
- suppress hypothetical/conditional suggestions unless explicitly assigned;
- suppress sequence descriptions that only say X comes before Y;
- suppress record-location statements such as `SharePoint contains...`;
- suppress logistics/noise such as screen sharing, audio, travel, holiday and
  meeting mechanics.

## Next samples

Most useful further samples:

- 20 sequence-not-action negatives using `before`, `after`, `comes before`,
  `depends on`, `follows`, `once`.
- 20 record-location/status negatives using `contains`, `is stored in`,
  `already includes`, `sits in`, `available on`.
- 20 transcript logistics negatives from Teams calls.
- 10-15 more true imperative action-table fragments.
- 10-15 true audit-plan/risk-assessment action rows.

