benchmarks.site

Security (macOS malware scanning)

Malware detection: test samples vs held-out clean files

A fair test: how much malware does the scanner catch, and how often does it wrongly flag normal files it was never tuned on?

Key finding

The full engine detects 519/528 test samples at INFO+, 479/528 at SUSPICIOUS+ and 318/528 at MALICIOUS, with 0 SUSPICIOUS+ false alarms on clean files never used for tuning (0/2973, later 0/2812); a fix of two context heuristics cut held-out INFO+ findings from 119/2973 (4.00%) to 10/2973 (0.34%).

How to read the numbers

Detection rate — higher is better. False alarms — lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

The full engine (all databases + heuristics) scans:

  • 528 malicious test samples (expected: detected),
  • 14 clean test samples (expected: no alarm),
  • clean files from a real Mac that were used while tuning the heuristics,
  • held-out clean files that were never used for tuning — the main false-alarm measure.

A false alarm is a clean file that reaches the given threshold. Thresholds are cumulative: INFO+ = INFO, SUSPICIOUS, PUA or MALICIOUS; SUSPICIOUS+ = SUSPICIOUS, PUA or MALICIOUS; MALICIOUS = MALICIOUS only. Only SUSPICIOUS+ is shown to the user as an alarm.

Runs

3 rows · 8 columnsCSV
202611032026-11-03 13:4358245281423092973462
20261103-fix2026-11-03 14:0158245281423092973827
20261109-speed2026-11-09 13:38543152814207728123558

Scan time is the elapsed time of the full scan as printed in the run log. The runs were not made under controlled load, so the times are not comparable as a speed benchmark (see Performance). The 20261109-speed run scanned a smaller clean set (393 fewer files: 232 fewer in the tuning set and 161 fewer in the held-out set).

1. Results by threshold

Run 20261103

3 rows · 5 columnsCSV
INFO+519/528 (98.30%)119/2973 (4.00%)32/2309 (1.39%)1/14 (7.14%)
SUSPICIOUS+479/528 (90.72%)0/2973 (0.00%)0/2309 (0.00%)0/14 (0.00%)
MALICIOUS318/528 (60.23%)0/2973 (0.00%)0/2309 (0.00%)0/14 (0.00%)

Run 20261103-fix

3 rows · 5 columnsCSV
INFO+519/528 (98.30%)10/2973 (0.34%)4/2309 (0.17%)1/14 (7.14%)
SUSPICIOUS+479/528 (90.72%)0/2973 (0.00%)0/2309 (0.00%)0/14 (0.00%)
MALICIOUS318/528 (60.23%)0/2973 (0.00%)0/2309 (0.00%)0/14 (0.00%)

Run 20261109-speed

3 rows · 5 columnsCSV
INFO+519/528 (98.30%)9/2812 (0.32%)4/2077 (0.19%)1/14 (7.14%)
SUSPICIOUS+479/528 (90.72%)0/2812 (0.00%)0/2077 (0.00%)0/14 (0.00%)
MALICIOUS318/528 (60.23%)0/2812 (0.00%)0/2077 (0.00%)0/14 (0.00%)

Verdict counts on the 528 test samples were the same in all three runs: 318 MALICIOUS, 141 SUSPICIOUS, 20 PUA, 40 INFO, 9 not detected.

2. Held-out clean files by category (INFO+ findings)

No held-out file reached SUSPICIOUS+ or MALICIOUS in any run, so only INFO+ counts are shown.

9 rows · 5 columnsCSV
app_frameworks_machoMach-O binaries inside application frameworks0/725 (0.00%)0/725 (0.00%)0/641 (0.00%)
app_resources_scriptsScripts from application resource folders3/500 (0.60%)3/500 (0.60%)2/464 (0.43%)
homebrew_cellarFiles from installed Homebrew packages2/800 (0.25%)2/800 (0.25%)2/800 (0.25%)
python_libsInstalled Python library files0/400 (0.00%)0/400 (0.00%)0/400 (0.00%)
user_documentsUser document files0/22 (0.00%)0/22 (0.00%)0/22 (0.00%)
installers_archivesDownloaded installers and archives5/30 (16.67%)2/30 (6.67%)2/30 (6.67%)
app_supportApplication support data109/492 (22.15%)3/492 (0.61%)3/451 (0.67%)
project_depsDependencies inside software projects0/4 (0.00%)0/4 (0.00%)0/4 (0.00%)
All held-out119/2973 (4.00%)10/2973 (0.34%)9/2812 (0.32%)

3. What changed between runs

  • 20261103 → 20261103-fix: almost all held-out INFO+ findings came from app_support (109/492) and installers_archives (5/30). Across all 5824 scanned files, the number of INFO files with the context heuristic Heur:Context.DownloadedUnsigned fell from 138 to 2, and Heur:Context.Name.Random (32 INFO files) no longer appears. Total INFO verdicts fell from 192 to 55. Detection at every threshold stayed the same.
  • 20261103-fix → 20261109-speed: same detection and the same per-threshold false-alarm results on a slightly smaller clean set; held-out INFO+ 9/2812 (0.32%).

4. Observations

  • At the alarm threshold (SUSPICIOUS+), detection is 479/528 (90.72%) with 0 false alarms on held-out clean files in all three runs.
  • MALICIOUS alone catches 318/528 (60.23%); about a third of detected samples are rated only SUSPICIOUS or PUA.
  • 9 of the 528 test samples are not detected at any level.
  • The single clean test sample with an INFO+ finding (1/14) is below the alarm threshold.
  • Small categories (project_deps 4, user_documents 22, installers_archives 30) give rates with large uncertainty; one file in installers_archives is 3.33 percentage points.