Dates: 2026-11-02 and 2026-11-03. Two measurements are taken for each version of the engine:
- Clean files: about 2900 clean files from a real Mac, grouped into 12 categories. Any
SUSPICIOUS,PUAorMALICIOUSverdict is a false alarm ("flagged");INFOfindings are counted separately. - Fixtures: 528 malicious test samples, 14 clean test samples and 48 neutral archive files (no expected
verdict), scanned in two modes:
heur(static heuristics only) andyara(heuristics together with YARA rules).
Runs:
before | 2026-11-02 | Heuristics before the fixes. |
|---|---|---|
after | 2026-11-02 | After the fixes, including an extraction of embedded scripts from Mach-O binaries. |
after-final | 2026-11-02 | The after run with the records it flagged re-scanned after reverting the Mach-O embedded-script extraction. Removing an analyzer path can only remove detections; the two later script rules were checked separately: 0 matches on all 2380 clean text files. |
bench-20261103 | 2026-11-03 | Re-measurement the next day, on the day of the first fair benchmark run. |
1. Clean files: summary
before | 2945 | 6 (0.20%) | 0 | 61 (2.07%) | 4149.2 |
|---|---|---|---|---|---|
after | 2944 | 2 (0.07%) | 1 | 61 (2.07%) | 4954.6 |
after-final | 2944 | 0 (0.00%) | 0 | 62 (2.11%) | 4954.6 (from after) |
bench-20261103 | 2910 | 0 (0.00%) | 0 | 46 (1.58%) | 1044.6 |
Scan time is the sum of per-file scan times.
2. Clean files by category
Each cell: false alarms (SUSPICIOUS+) / INFO findings / files.
apps_plist | Info.plist files of installed apps | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 |
|---|---|---|---|---|---|
apps_helper | Helper executables inside app bundles (Mach-O) | 4 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 |
apps_main | Main executables of installed apps | 1 / 2 / 41 | 1 / 1 / 41 | 0 / 2 / 41 | 0 / 2 / 41 |
apps_script | Scripts inside app bundles | 1 / 0 / 84 | 0 / 1 / 84 | 0 / 1 / 84 | 0 / 1 / 84 |
homebrew_bin | Executables installed by Homebrew | 0 / 0 / 495 | 1 / 0 / 495 | 0 / 0 / 495 | 0 / 1 / 495 |
extra_script | Additional script files (JavaScript, Python, shell, text, PowerShell, Perl) | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 29 / 459 |
launchd_plist | launchd job definitions (plist) | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 |
sysapps_plist | Info.plist files of macOS system apps | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 |
sysapps_main | Main executables of macOS system apps | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 |
docs_own | Source and config files of the author's own projects | 0 / 7 / 660 | 0 / 7 / 659 | 0 / 7 / 659 | 0 / 7 / 643 |
docs_deps | Dependency files inside those projects | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 |
usr_bin_script | macOS system command-line scripts (Perl, shell, Ruby) | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 |
| All | 6 / 61 / 2945 | 2 / 61 / 2944 | 0 / 62 / 2944 | 0 / 46 / 2910 |
3. Which detections caused the false alarms
before (6 false alarms, all SUSPICIOUS)
Heur:MachO.CertificateRevoked | HIGH | 5 | apps_helper 4, apps_main 1 (together with Heur:Context.DownloadedUnsigned) |
|---|---|---|---|
Heur:SHELL.RemoveQuarantine + Heur:SHELL.KeychainAccess | MEDIUM + MEDIUM | 1 | apps_script 1 |
after (2 false alarms)
Both were caused by the new extraction of embedded scripts from Mach-O binaries: shell-script rules fired on strings inside two clean executables.
apps_main | MALICIOUS | 231 | Own:OWN_Script_Bash_CurlPipeShell_RawIpv4, Heur:SHELL.PipeToShell, Heur:SHELL.DecodePipe, Heur:SHELL.KeychainExfil, Heur:SHELL.DownloadExecute, Heur:SHELL.Exfil, Heur:MachO.Downloader, Heur:MachO.AdhocSigned |
|---|---|---|---|
homebrew_bin | SUSPICIOUS | 53 | Heur:SHELL.ReverseShellDevTcp, Heur:MachO.Downloader, Heur:MachO.AdhocSigned |
after-final
The extraction was reverted and the two records re-scanned: the apps_main file became INFO (score 13)
and the homebrew_bin file CLEAN (score 8). No false alarms remain.
INFO findings on clean files
INFO is below the alarm threshold. In before, 58 of the 61 INFO files carry
Heur:Context.DownloadedUnsigned (45 in extra_script, 7 in docs_own, 6 in docs_deps) and 5 carry
Heur:Context.Name.Random (all extra_script). In bench-20261103, Heur:Context.DownloadedUnsigned
appears on 41 files and Heur:Context.Name.Random on 4.
4. Fixtures: detection on test samples
The 14 clean test samples were never flagged (0/14) in any run and mode.
before | heur | 168/528 (31.82%) | 107/528 (20.27%) | 31/528 (5.87%) |
|---|---|---|---|---|
after | heur | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) |
bench-20261103 | heur | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) |
before | yara | 519/528 (98.30%) | 466/528 (88.26%) | 296/528 (56.06%) |
after | yara | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) |
bench-20261103 | yara | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) |
By sample group, heur mode (INFO+ / SUSPICIOUS+ / MALICIOUS)
documents | 15 | 15 / 15 / 8 | 15 / 15 / 8 | 15 / 15 / 8 |
|---|---|---|---|---|
eicar | 2 | 0 / 0 / 0 | 0 / 0 / 0 | 0 / 0 / 0 |
scripts | 16 | 16 / 16 / 5 | 16 / 16 / 6 | 16 / 16 / 6 |
yara/gen | 107 | 37 / 28 / 7 | 55 / 45 / 14 | 55 / 45 / 14 |
yara/macos | 60 | 33 / 17 / 5 | 41 / 26 / 8 | 41 / 26 / 8 |
yara/native | 16 | 4 / 3 / 0 | 4 / 3 / 0 | 4 / 3 / 0 |
yara/scripts_web | 312 | 63 / 28 / 6 | 66 / 30 / 7 | 66 / 30 / 7 |
By sample group, yara mode (INFO+ / SUSPICIOUS+ / MALICIOUS)
documents | 15 | 15 / 15 / 10 | 15 / 15 / 10 | 15 / 15 / 10 |
|---|---|---|---|---|
eicar | 2 | 2 / 2 / 2 | 2 / 2 / 2 | 2 / 2 / 2 |
scripts | 16 | 16 / 16 / 9 | 16 / 16 / 9 | 16 / 16 / 9 |
yara/gen | 107 | 107 / 55 / 21 | 107 / 68 / 35 | 107 / 68 / 35 |
yara/macos | 60 | 60 / 60 / 54 | 60 / 60 / 60 | 60 / 60 / 60 |
yara/native | 16 | 7 / 6 / 4 | 7 / 6 / 4 | 7 / 6 / 4 |
yara/scripts_web | 312 | 312 / 312 / 196 | 312 / 312 / 199 | 312 / 312 / 199 |
Neutral archives (48 files, no expected verdict)
before | heur | 30 | 10 | 8 |
|---|---|---|---|---|
before | yara | 30 | 8 | 10 |
after | heur | 30 | 10 | 8 |
after | yara | 30 | 8 | 10 |
bench-20261103 | heur | 30 | 10 | 8 |
bench-20261103 | yara | 30 | 8 | 10 |
5. Observations
- The fixes removed the
Heur:MachO.CertificateRevokedfalse alarms on clean app helpers and main executables (5 files) and the script false alarm inapps_script; none of these files is flagged afterwards. - At the same time, heuristic-only detection on the test samples rose (SUSPICIOUS+ 107 → 135, MALICIOUS 31 → 43), and with YARA rules SUSPICIOUS+ rose from 466 to 479 and MALICIOUS from 296 to 319.
- Extracting embedded scripts from Mach-O binaries produced 2 new false alarms on clean files (one rated
MALICIOUS) and was reverted. - The
bench-20261103clean set is 34 files smaller (2910 vs 2944), entirely inextra_script(459 vs 477) anddocs_own(643 vs 659).