The full engine (all databases + heuristics) scans:
- 528 malicious test samples (expected: detected),
- 14 clean test samples (expected: no alarm),
- clean files from a real Mac that were used while tuning the heuristics,
- held-out clean files that were never used for tuning — the main false-alarm measure.
A false alarm is a clean file that reaches the given threshold. Thresholds are cumulative:
INFO+ = INFO, SUSPICIOUS, PUA or MALICIOUS; SUSPICIOUS+ = SUSPICIOUS, PUA or MALICIOUS;
MALICIOUS = MALICIOUS only. Only SUSPICIOUS+ is shown to the user as an alarm.
Runs
20261103 | 2026-11-03 13:43 | 5824 | 528 | 14 | 2309 | 2973 | 462 |
|---|---|---|---|---|---|---|---|
20261103-fix | 2026-11-03 14:01 | 5824 | 528 | 14 | 2309 | 2973 | 827 |
20261109-speed | 2026-11-09 13:38 | 5431 | 528 | 14 | 2077 | 2812 | 3558 |
Scan time is the elapsed time of the full scan as printed in the run log. The runs were not made
under controlled load, so the times are not comparable as a speed benchmark (see Performance).
The 20261109-speed run scanned a smaller clean set (393 fewer files: 232 fewer in the tuning set and
161 fewer in the held-out set).
1. Results by threshold
Run 20261103
| INFO+ | 519/528 (98.30%) | 119/2973 (4.00%) | 32/2309 (1.39%) | 1/14 (7.14%) |
|---|---|---|---|---|
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
Run 20261103-fix
| INFO+ | 519/528 (98.30%) | 10/2973 (0.34%) | 4/2309 (0.17%) | 1/14 (7.14%) |
|---|---|---|---|---|
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2973 (0.00%) | 0/2309 (0.00%) | 0/14 (0.00%) |
Run 20261109-speed
| INFO+ | 519/528 (98.30%) | 9/2812 (0.32%) | 4/2077 (0.19%) | 1/14 (7.14%) |
|---|---|---|---|---|
| SUSPICIOUS+ | 479/528 (90.72%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) |
| MALICIOUS | 318/528 (60.23%) | 0/2812 (0.00%) | 0/2077 (0.00%) | 0/14 (0.00%) |
Verdict counts on the 528 test samples were the same in all three runs: 318 MALICIOUS, 141 SUSPICIOUS,
20 PUA, 40 INFO, 9 not detected.
2. Held-out clean files by category (INFO+ findings)
No held-out file reached SUSPICIOUS+ or MALICIOUS in any run, so only INFO+ counts are shown.
app_frameworks_macho | Mach-O binaries inside application frameworks | 0/725 (0.00%) | 0/725 (0.00%) | 0/641 (0.00%) |
|---|---|---|---|---|
app_resources_scripts | Scripts from application resource folders | 3/500 (0.60%) | 3/500 (0.60%) | 2/464 (0.43%) |
homebrew_cellar | Files from installed Homebrew packages | 2/800 (0.25%) | 2/800 (0.25%) | 2/800 (0.25%) |
python_libs | Installed Python library files | 0/400 (0.00%) | 0/400 (0.00%) | 0/400 (0.00%) |
user_documents | User document files | 0/22 (0.00%) | 0/22 (0.00%) | 0/22 (0.00%) |
installers_archives | Downloaded installers and archives | 5/30 (16.67%) | 2/30 (6.67%) | 2/30 (6.67%) |
app_support | Application support data | 109/492 (22.15%) | 3/492 (0.61%) | 3/451 (0.67%) |
project_deps | Dependencies inside software projects | 0/4 (0.00%) | 0/4 (0.00%) | 0/4 (0.00%) |
| All held-out | 119/2973 (4.00%) | 10/2973 (0.34%) | 9/2812 (0.32%) |
3. What changed between runs
20261103→20261103-fix: almost all held-out INFO+ findings came fromapp_support(109/492) andinstallers_archives(5/30). Across all 5824 scanned files, the number ofINFOfiles with the context heuristicHeur:Context.DownloadedUnsignedfell from 138 to 2, andHeur:Context.Name.Random(32INFOfiles) no longer appears. TotalINFOverdicts fell from 192 to 55. Detection at every threshold stayed the same.20261103-fix→20261109-speed: same detection and the same per-threshold false-alarm results on a slightly smaller clean set; held-out INFO+ 9/2812 (0.32%).
4. Observations
- At the alarm threshold (SUSPICIOUS+), detection is 479/528 (90.72%) with 0 false alarms on held-out clean files in all three runs.
MALICIOUSalone catches 318/528 (60.23%); about a third of detected samples are rated onlySUSPICIOUSorPUA.- 9 of the 528 test samples are not detected at any level.
- The single clean test sample with an INFO+ finding (1/14) is below the alarm threshold.
- Small categories (
project_deps4,user_documents22,installers_archives30) give rates with large uncertainty; one file ininstallers_archivesis 3.33 percentage points.