benchmarks.site

Security (macOS malware scanning)

Malware scanning: static-heuristic false alarms before and after fixes

How often the scanner wrongly flagged normal Mac files, before and after fixes.

Key finding

Fixes to the static heuristics removed all 6 SUSPICIOUS false alarms on 2945 clean Mac files (0.20% → 0/2944) while raising heuristic-only detection on the 528 test samples from 107 to 135 at SUSPICIOUS+; an intermediate change that briefly added 2 new false alarms was reverted.

How to read the numbers

False alarms — lower is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Dates: 2026-11-02 and 2026-11-03. Two measurements are taken for each version of the engine:

  • Clean files: about 2900 clean files from a real Mac, grouped into 12 categories. Any SUSPICIOUS, PUA or MALICIOUS verdict is a false alarm ("flagged"); INFO findings are counted separately.
  • Fixtures: 528 malicious test samples, 14 clean test samples and 48 neutral archive files (no expected verdict), scanned in two modes: heur (static heuristics only) and yara (heuristics together with YARA rules).

Runs:

4 rows · 3 columnsCSV
before2026-11-02Heuristics before the fixes.
after2026-11-02After the fixes, including an extraction of embedded scripts from Mach-O binaries.
after-final2026-11-02The after run with the records it flagged re-scanned after reverting the Mach-O embedded-script extraction. Removing an analyzer path can only remove detections; the two later script rules were checked separately: 0 matches on all 2380 clean text files.
bench-202611032026-11-03Re-measurement the next day, on the day of the first fair benchmark run.

1. Clean files: summary

4 rows · 6 columnsCSV
before29456 (0.20%)061 (2.07%)4149.2
after29442 (0.07%)161 (2.07%)4954.6
after-final29440 (0.00%)062 (2.11%)4954.6 (from after)
bench-2026110329100 (0.00%)046 (1.58%)1044.6

Scan time is the sum of per-file scan times.

2. Clean files by category

Each cell: false alarms (SUSPICIOUS+) / INFO findings / files.

13 rows · 6 columnsCSV
apps_plistInfo.plist files of installed apps0 / 0 / 360 / 0 / 360 / 0 / 360 / 0 / 36
apps_helperHelper executables inside app bundles (Mach-O)4 / 0 / 900 / 0 / 900 / 0 / 900 / 0 / 90
apps_mainMain executables of installed apps1 / 2 / 411 / 1 / 410 / 2 / 410 / 2 / 41
apps_scriptScripts inside app bundles1 / 0 / 840 / 1 / 840 / 1 / 840 / 1 / 84
homebrew_binExecutables installed by Homebrew0 / 0 / 4951 / 0 / 4950 / 0 / 4950 / 1 / 495
extra_scriptAdditional script files (JavaScript, Python, shell, text, PowerShell, Perl)0 / 46 / 4770 / 46 / 4770 / 46 / 4770 / 29 / 459
launchd_plistlaunchd job definitions (plist)0 / 0 / 1310 / 0 / 1310 / 0 / 1310 / 0 / 131
sysapps_plistInfo.plist files of macOS system apps0 / 0 / 650 / 0 / 650 / 0 / 650 / 0 / 65
sysapps_mainMain executables of macOS system apps0 / 0 / 660 / 0 / 660 / 0 / 660 / 0 / 66
docs_ownSource and config files of the author's own projects0 / 7 / 6600 / 7 / 6590 / 7 / 6590 / 7 / 643
docs_depsDependency files inside those projects0 / 6 / 5490 / 6 / 5490 / 6 / 5490 / 6 / 549
usr_bin_scriptmacOS system command-line scripts (Perl, shell, Ruby)0 / 0 / 2510 / 0 / 2510 / 0 / 2510 / 0 / 251
All6 / 61 / 29452 / 61 / 29440 / 62 / 29440 / 46 / 2910

3. Which detections caused the false alarms

before (6 false alarms, all SUSPICIOUS)

2 rows · 4 columnsCSV
Heur:MachO.CertificateRevokedHIGH5apps_helper 4, apps_main 1 (together with Heur:Context.DownloadedUnsigned)
Heur:SHELL.RemoveQuarantine + Heur:SHELL.KeychainAccessMEDIUM + MEDIUM1apps_script 1

after (2 false alarms)

Both were caused by the new extraction of embedded scripts from Mach-O binaries: shell-script rules fired on strings inside two clean executables.

2 rows · 4 columnsCSV
apps_mainMALICIOUS231Own:OWN_Script_Bash_CurlPipeShell_RawIpv4, Heur:SHELL.PipeToShell, Heur:SHELL.DecodePipe, Heur:SHELL.KeychainExfil, Heur:SHELL.DownloadExecute, Heur:SHELL.Exfil, Heur:MachO.Downloader, Heur:MachO.AdhocSigned
homebrew_binSUSPICIOUS53Heur:SHELL.ReverseShellDevTcp, Heur:MachO.Downloader, Heur:MachO.AdhocSigned

after-final

The extraction was reverted and the two records re-scanned: the apps_main file became INFO (score 13) and the homebrew_bin file CLEAN (score 8). No false alarms remain.

INFO findings on clean files

INFO is below the alarm threshold. In before, 58 of the 61 INFO files carry Heur:Context.DownloadedUnsigned (45 in extra_script, 7 in docs_own, 6 in docs_deps) and 5 carry Heur:Context.Name.Random (all extra_script). In bench-20261103, Heur:Context.DownloadedUnsigned appears on 41 files and Heur:Context.Name.Random on 4.

4. Fixtures: detection on test samples

The 14 clean test samples were never flagged (0/14) in any run and mode.

6 rows · 5 columnsCSV
beforeheur168/528 (31.82%)107/528 (20.27%)31/528 (5.87%)
afterheur197/528 (37.31%)135/528 (25.57%)43/528 (8.14%)
bench-20261103heur197/528 (37.31%)135/528 (25.57%)43/528 (8.14%)
beforeyara519/528 (98.30%)466/528 (88.26%)296/528 (56.06%)
afteryara519/528 (98.30%)479/528 (90.72%)319/528 (60.42%)
bench-20261103yara519/528 (98.30%)479/528 (90.72%)319/528 (60.42%)

By sample group, heur mode (INFO+ / SUSPICIOUS+ / MALICIOUS)

7 rows · 5 columnsCSV
documents1515 / 15 / 815 / 15 / 815 / 15 / 8
eicar20 / 0 / 00 / 0 / 00 / 0 / 0
scripts1616 / 16 / 516 / 16 / 616 / 16 / 6
yara/gen10737 / 28 / 755 / 45 / 1455 / 45 / 14
yara/macos6033 / 17 / 541 / 26 / 841 / 26 / 8
yara/native164 / 3 / 04 / 3 / 04 / 3 / 0
yara/scripts_web31263 / 28 / 666 / 30 / 766 / 30 / 7

By sample group, yara mode (INFO+ / SUSPICIOUS+ / MALICIOUS)

7 rows · 5 columnsCSV
documents1515 / 15 / 1015 / 15 / 1015 / 15 / 10
eicar22 / 2 / 22 / 2 / 22 / 2 / 2
scripts1616 / 16 / 916 / 16 / 916 / 16 / 9
yara/gen107107 / 55 / 21107 / 68 / 35107 / 68 / 35
yara/macos6060 / 60 / 5460 / 60 / 6060 / 60 / 60
yara/native167 / 6 / 47 / 6 / 47 / 6 / 4
yara/scripts_web312312 / 312 / 196312 / 312 / 199312 / 312 / 199

Neutral archives (48 files, no expected verdict)

6 rows · 5 columnsCSV
beforeheur30108
beforeyara30810
afterheur30108
afteryara30810
bench-20261103heur30108
bench-20261103yara30810

5. Observations

  • The fixes removed the Heur:MachO.CertificateRevoked false alarms on clean app helpers and main executables (5 files) and the script false alarm in apps_script; none of these files is flagged afterwards.
  • At the same time, heuristic-only detection on the test samples rose (SUSPICIOUS+ 107 → 135, MALICIOUS 31 → 43), and with YARA rules SUSPICIOUS+ rose from 466 to 479 and MALICIOUS from 296 to 319.
  • Extracting embedded scripts from Mach-O binaries produced 2 new false alarms on clean files (one rated MALICIOUS) and was reverted.
  • The bench-20261103 clean set is 34 files smaller (2910 vs 2944), entirely in extra_script (459 vs 477) and docs_own (643 vs 659).