# Malware scanning: static-heuristic false alarms before and after fixes

> Fixes to the static heuristics removed all 6 SUSPICIOUS false alarms on 2945 clean Mac files (0.20% → 0/2944) while raising heuristic-only detection on the 528 test samples from 107 to 135 at SUSPICIOUS+; an intermediate change that briefly added 2 new false alarms was reverted.

Area: Security (macOS malware scanning). Page: https://benchmarks.site/malware-static-false-positives/. Methodology and caveats: https://benchmarks.site/methodology/#malware

Dates: 2026-11-02 and 2026-11-03. Two measurements are taken for each
version of the engine:

- **Clean files:** about 2900 clean files from a real Mac, grouped into 12 categories. Any `SUSPICIOUS`,
  `PUA` or `MALICIOUS` verdict is a false alarm ("flagged"); `INFO` findings are counted separately.
- **Fixtures:** 528 malicious test samples, 14 clean test samples and 48 neutral archive files (no expected
  verdict), scanned in two modes: `heur` (static heuristics only) and `yara` (heuristics together with YARA
  rules).

Runs:

| Run | Date | Description |
|---|---|---|
| `before` | 2026-11-02 | Heuristics before the fixes. |
| `after` | 2026-11-02 | After the fixes, including an extraction of embedded scripts from Mach-O binaries. |
| `after-final` | 2026-11-02 | The `after` run with the records it flagged re-scanned after reverting the Mach-O embedded-script extraction. Removing an analyzer path can only remove detections; the two later script rules were checked separately: 0 matches on all 2380 clean text files. |
| `bench-20261103` | 2026-11-03 | Re-measurement the next day, on the day of the first [fair benchmark](/malware-detection-held-out/) run. |

## 1. Clean files: summary

| Run | Files | False alarms (SUSPICIOUS+) | Of which `MALICIOUS` | `INFO` findings | Summed scan time (s) |
|---|---|---|---|---|---|
| `before` | 2945 | 6 (0.20%) | 0 | 61 (2.07%) | 4149.2 |
| `after` | 2944 | 2 (0.07%) | 1 | 61 (2.07%) | 4954.6 |
| `after-final` | 2944 | 0 (0.00%) | 0 | 62 (2.11%) | 4954.6 (from `after`) |
| `bench-20261103` | 2910 | 0 (0.00%) | 0 | 46 (1.58%) | 1044.6 |

Scan time is the sum of per-file scan times.

## 2. Clean files by category

Each cell: false alarms (SUSPICIOUS+) / `INFO` findings / files.

| Category | Description | `before` | `after` | `after-final` | `bench-20261103` |
|---|---|---|---|---|---|
| `apps_plist` | `Info.plist` files of installed apps | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 | 0 / 0 / 36 |
| `apps_helper` | Helper executables inside app bundles (Mach-O) | 4 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 | 0 / 0 / 90 |
| `apps_main` | Main executables of installed apps | 1 / 2 / 41 | 1 / 1 / 41 | 0 / 2 / 41 | 0 / 2 / 41 |
| `apps_script` | Scripts inside app bundles | 1 / 0 / 84 | 0 / 1 / 84 | 0 / 1 / 84 | 0 / 1 / 84 |
| `homebrew_bin` | Executables installed by Homebrew | 0 / 0 / 495 | 1 / 0 / 495 | 0 / 0 / 495 | 0 / 1 / 495 |
| `extra_script` | Additional script files (JavaScript, Python, shell, text, PowerShell, Perl) | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 46 / 477 | 0 / 29 / 459 |
| `launchd_plist` | launchd job definitions (plist) | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 | 0 / 0 / 131 |
| `sysapps_plist` | `Info.plist` files of macOS system apps | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 | 0 / 0 / 65 |
| `sysapps_main` | Main executables of macOS system apps | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 | 0 / 0 / 66 |
| `docs_own` | Source and config files of the author's own projects | 0 / 7 / 660 | 0 / 7 / 659 | 0 / 7 / 659 | 0 / 7 / 643 |
| `docs_deps` | Dependency files inside those projects | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 | 0 / 6 / 549 |
| `usr_bin_script` | macOS system command-line scripts (Perl, shell, Ruby) | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 | 0 / 0 / 251 |
| All | | 6 / 61 / 2945 | 2 / 61 / 2944 | 0 / 62 / 2944 | 0 / 46 / 2910 |

## 3. Which detections caused the false alarms

### `before` (6 false alarms, all `SUSPICIOUS`)

| Detection | Severity | Files | Categories |
|---|---|---|---|
| `Heur:MachO.CertificateRevoked` | HIGH | 5 | `apps_helper` 4, `apps_main` 1 (together with `Heur:Context.DownloadedUnsigned`) |
| `Heur:SHELL.RemoveQuarantine` + `Heur:SHELL.KeychainAccess` | MEDIUM + MEDIUM | 1 | `apps_script` 1 |

### `after` (2 false alarms)

Both were caused by the new extraction of embedded scripts from Mach-O binaries: shell-script rules
fired on strings inside two clean executables.

| Category | Verdict | Score | Detections |
|---|---|---|---|
| `apps_main` | `MALICIOUS` | 231 | `Own:OWN_Script_Bash_CurlPipeShell_RawIpv4`, `Heur:SHELL.PipeToShell`, `Heur:SHELL.DecodePipe`, `Heur:SHELL.KeychainExfil`, `Heur:SHELL.DownloadExecute`, `Heur:SHELL.Exfil`, `Heur:MachO.Downloader`, `Heur:MachO.AdhocSigned` |
| `homebrew_bin` | `SUSPICIOUS` | 53 | `Heur:SHELL.ReverseShellDevTcp`, `Heur:MachO.Downloader`, `Heur:MachO.AdhocSigned` |

### `after-final`

The extraction was reverted and the two records re-scanned: the `apps_main` file became `INFO` (score 13)
and the `homebrew_bin` file `CLEAN` (score 8). No false alarms remain.

### `INFO` findings on clean files

`INFO` is below the alarm threshold. In `before`, 58 of the 61 `INFO` files carry
`Heur:Context.DownloadedUnsigned` (45 in `extra_script`, 7 in `docs_own`, 6 in `docs_deps`) and 5 carry
`Heur:Context.Name.Random` (all `extra_script`). In `bench-20261103`, `Heur:Context.DownloadedUnsigned`
appears on 41 files and `Heur:Context.Name.Random` on 4.

## 4. Fixtures: detection on test samples

The 14 clean test samples were never flagged (0/14) in any run and mode.

| Run | Mode | INFO+ | SUSPICIOUS+ | MALICIOUS |
|---|---|---|---|---|
| `before` | `heur` | 168/528 (31.82%) | 107/528 (20.27%) | 31/528 (5.87%) |
| `after` | `heur` | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) |
| `bench-20261103` | `heur` | 197/528 (37.31%) | 135/528 (25.57%) | 43/528 (8.14%) |
| `before` | `yara` | 519/528 (98.30%) | 466/528 (88.26%) | 296/528 (56.06%) |
| `after` | `yara` | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) |
| `bench-20261103` | `yara` | 519/528 (98.30%) | 479/528 (90.72%) | 319/528 (60.42%) |

### By sample group, `heur` mode (INFO+ / SUSPICIOUS+ / MALICIOUS)

| Group | Samples | `before` | `after` | `bench-20261103` |
|---|---|---|---|---|
| `documents` | 15 | 15 / 15 / 8 | 15 / 15 / 8 | 15 / 15 / 8 |
| `eicar` | 2 | 0 / 0 / 0 | 0 / 0 / 0 | 0 / 0 / 0 |
| `scripts` | 16 | 16 / 16 / 5 | 16 / 16 / 6 | 16 / 16 / 6 |
| `yara/gen` | 107 | 37 / 28 / 7 | 55 / 45 / 14 | 55 / 45 / 14 |
| `yara/macos` | 60 | 33 / 17 / 5 | 41 / 26 / 8 | 41 / 26 / 8 |
| `yara/native` | 16 | 4 / 3 / 0 | 4 / 3 / 0 | 4 / 3 / 0 |
| `yara/scripts_web` | 312 | 63 / 28 / 6 | 66 / 30 / 7 | 66 / 30 / 7 |

### By sample group, `yara` mode (INFO+ / SUSPICIOUS+ / MALICIOUS)

| Group | Samples | `before` | `after` | `bench-20261103` |
|---|---|---|---|---|
| `documents` | 15 | 15 / 15 / 10 | 15 / 15 / 10 | 15 / 15 / 10 |
| `eicar` | 2 | 2 / 2 / 2 | 2 / 2 / 2 | 2 / 2 / 2 |
| `scripts` | 16 | 16 / 16 / 9 | 16 / 16 / 9 | 16 / 16 / 9 |
| `yara/gen` | 107 | 107 / 55 / 21 | 107 / 68 / 35 | 107 / 68 / 35 |
| `yara/macos` | 60 | 60 / 60 / 54 | 60 / 60 / 60 | 60 / 60 / 60 |
| `yara/native` | 16 | 7 / 6 / 4 | 7 / 6 / 4 | 7 / 6 / 4 |
| `yara/scripts_web` | 312 | 312 / 312 / 196 | 312 / 312 / 199 | 312 / 312 / 199 |

### Neutral archives (48 files, no expected verdict)

| Run | Mode | `CLEAN` | `INFO` | `SUSPICIOUS` |
|---|---|---|---|---|
| `before` | `heur` | 30 | 10 | 8 |
| `before` | `yara` | 30 | 8 | 10 |
| `after` | `heur` | 30 | 10 | 8 |
| `after` | `yara` | 30 | 8 | 10 |
| `bench-20261103` | `heur` | 30 | 10 | 8 |
| `bench-20261103` | `yara` | 30 | 8 | 10 |

## 5. Observations

- The fixes removed the `Heur:MachO.CertificateRevoked` false alarms on clean app helpers and main
  executables (5 files) and the script false alarm in `apps_script`; none of these files is flagged afterwards.
- At the same time, heuristic-only detection on the test samples rose (SUSPICIOUS+ 107 → 135, MALICIOUS
  31 → 43), and with YARA rules SUSPICIOUS+ rose from 466 to 479 and MALICIOUS from 296 to 319.
- Extracting embedded scripts from Mach-O binaries produced 2 new false alarms on clean files (one rated
  `MALICIOUS`) and was reverted.
- The `bench-20261103` clean set is 34 files smaller (2910 vs 2944), entirely in `extra_script` (459 vs 477)
  and `docs_own` (643 vs 659).
