# Malware detection: test samples vs held-out clean files

> The full engine detects 519/528 test samples at INFO+, 479/528 at SUSPICIOUS+ and 318/528 at MALICIOUS, with 0 SUSPICIOUS+ false alarms on clean files never used for tuning (0/2973, later 0/2812); a fix of two context heuristics cut held-out INFO+ findings from 119/2973 (4.00%) to 10/2973 (0.34%).

Area: Security (macOS malware scanning). Page: https://benchmarks.site/malware-detection-held-out/. Methodology and caveats: https://benchmarks.site/methodology/#malware

The full engine (all databases + heuristics) scans:

- 528 malicious test samples (expected: detected),
- 14 clean test samples (expected: no alarm),
- clean files from a real Mac that were used while tuning the heuristics,
- held-out clean files that were **never** used for tuning — the main false-alarm measure.

A false alarm is a clean file that reaches the given threshold. Thresholds are cumulative:
INFO+ = `INFO`, `SUSPICIOUS`, `PUA` or `MALICIOUS`; SUSPICIOUS+ = `SUSPICIOUS`, `PUA` or `MALICIOUS`;
MALICIOUS = `MALICIOUS` only. Only SUSPICIOUS+ is shown to the user as an alarm.

## Runs

| Run | Date (run end) | Files scanned | Malicious test samples | Clean test samples | Clean (tuning) | Clean (held-out) | Scan time (s) |
|---|---|---|---|---|---|---|---|
| `20261103` | 2026-11-03 13:43 | 5824 | 528 | 14 | 2309 | 2973 | 462 |
| `20261103-fix` | 2026-11-03 14:01 | 5824 | 528 | 14 | 2309 | 2973 | 827 |
| `20261109-speed` | 2026-11-09 13:38 | 5431 | 528 | 14 | 2077 | 2812 | 3558 |

Scan time is the elapsed time of the full scan as printed in the run log. The runs were not made
under controlled load, so the times are not comparable as a speed benchmark (see [Performance](/scanner-performance/)).
The `20261109-speed` run scanned a smaller clean set (393 fewer files: 232 fewer in the tuning set and
161 fewer in the held-out set).

## 1. Results by threshold

### Run `20261103`

| Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples |
|---|---|---|---|---|
| INFO+ | 519/528 (98.30%) | 119/2973 (4.00%) | 32/2309 (1.39%) | 1/14 (7.14%) |
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |

### Run `20261103-fix`

| Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples |
|---|---|---|---|---|
| INFO+ | 519/528 (98.30%) | 10/2973 (0.34%) | 4/2309 (0.17%) | 1/14 (7.14%) |
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |

### Run `20261109-speed`

| Threshold | Detection (test samples) | False alarms: held-out clean (never tuned) | False alarms: clean used for tuning | False alarms: clean test samples |
|---|---|---|---|---|
| INFO+ | 519/528 (98.30%) | 9/2812 (0.32%) | 4/2077 (0.19%) | 1/14 (7.14%) |
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) |

Verdict counts on the 528 test samples were the same in all three runs: 318 `MALICIOUS`, 141 `SUSPICIOUS`,
20 `PUA`, 40 `INFO`, 9 not detected.

## 2. Held-out clean files by category (INFO+ findings)

No held-out file reached SUSPICIOUS+ or MALICIOUS in any run, so only INFO+ counts are shown.

| Category | Description | `20261103` | `20261103-fix` | `20261109-speed` |
|---|---|---|---|---|
| `app_frameworks_macho` | Mach-O binaries inside application frameworks | 0/725 (0.00%) | 0/725 (0.00%) | 0/641 (0.00%) |
| `app_resources_scripts` | Scripts from application resource folders | 3/500 (0.60%) | 3/500 (0.60%) | 2/464 (0.43%) |
| `homebrew_cellar` | Files from installed Homebrew packages | 2/800 (0.25%) | 2/800 (0.25%) | 2/800 (0.25%) |
| `python_libs` | Installed Python library files | 0/400 (0.00%) | 0/400 (0.00%) | 0/400 (0.00%) |
| `user_documents` | User document files | 0/22 (0.00%) | 0/22 (0.00%) | 0/22 (0.00%) |
| `installers_archives` | Downloaded installers and archives | 5/30 (16.67%) | 2/30 (6.67%) | 2/30 (6.67%) |
| `app_support` | Application support data | 109/492 (22.15%) | 3/492 (0.61%) | 3/451 (0.67%) |
| `project_deps` | Dependencies inside software projects | 0/4 (0.00%) | 0/4 (0.00%) | 0/4 (0.00%) |
| All held-out | | 119/2973 (4.00%) | 10/2973 (0.34%) | 9/2812 (0.32%) |

## 3. What changed between runs

- `20261103` → `20261103-fix`: almost all held-out INFO+ findings came from `app_support` (109/492)
  and `installers_archives` (5/30). Across all 5824 scanned files, the number of `INFO` files with the
  context heuristic `Heur:Context.DownloadedUnsigned` fell from 138 to 2, and `Heur:Context.Name.Random`
  (32 `INFO` files) no longer appears. Total `INFO` verdicts fell from 192 to 55. Detection at every
  threshold stayed the same.
- `20261103-fix` → `20261109-speed`: same detection and the same per-threshold false-alarm results on
  a slightly smaller clean set; held-out INFO+ 9/2812 (0.32%).

## 4. Observations

- At the alarm threshold (SUSPICIOUS+), detection is 479/528 (90.72%) with 0 false alarms on held-out
  clean files in all three runs.
- `MALICIOUS` alone catches 318/528 (60.23%); about a third of detected samples are rated only
  `SUSPICIOUS` or `PUA`.
- 9 of the 528 test samples are not detected at any level.
- The single clean test sample with an INFO+ finding (1/14) is below the alarm threshold.
- Small categories (`project_deps` 4, `user_documents` 22, `installers_archives` 30) give rates with
  large uncertainty; one file in `installers_archives` is 3.33 percentage points.
