benchmarks.site

Security (macOS malware scanning)

ML classifiers on TLSH fingerprints of Mach-O files

Can a machine-learning model spot malicious Mac programs from a file 'fingerprint' alone?

Key finding

Gradient-boosted trees are the best model on TLSH features — GBDT 600×d6 reaches out-of-fold AUC 0.966 and 39.3% recall at a 0.1% false-positive rate, ahead of MLPs (18–24%) and logistic regression (9.5%); nearest-neighbour TLSH distance rules alone are far weaker (AUC 0.582).

How to read the numbers

AUC: 1.0 is perfect, 0.5 is guessing. Recall at 0.1% FP = share of malware caught while flagging at most 1 in 1000 clean files. Higher is better.

Hover a column name for its meaning, or see the glossary.

How we tested & caveats · Markdown · CSV

Date: 2026-11-03. (file types macho and macho_fat).

The question: can a classifier trained on TLSH fuzzy-hash fingerprints of Mach-O binaries recognise malware it has not seen, better than the plain rule "TLSH distance to a known sample ≤ threshold"?

Dataset

4 rows · 2 columnsCSV
Malicious Mach-O files (positives)1436
Clean Mach-O files (negatives)4353
Benchmark samples (bench)547
Malware families23

Validation: grouped 5-fold cross-validation, where groups keep whole malware families and whole applications in one fold, so the test fold always contains families and apps not seen in training. Per fold this gives on average 287.2 positives and 870.6 negatives in the test part.

Metrics

  • AUC test — ROC AUC on the test folds, averaged over the 5 folds (auc_te).
  • OOF AUC — ROC AUC on the pooled out-of-fold predictions of all 5 folds (each file scored by the model that did not see it in training) (oof_auc).
  • Recall at 0.1% FP (OOF) — share of the 1436 malicious files detected when the decision threshold on the pooled out-of-fold scores is set so that at most 0.1% of the 4353 clean files are flagged (oof_rec_at_0.1). This is the main number: a scanner must keep false alarms very low.
  • FP at that threshold — clean files flagged at that threshold (oof_fp_at_0.1); 4/4353 = 0.092% for every model.
  • Precision (OOF) — share of flagged files that are malicious at that threshold (oof_prec_at_0.1).
  • Params — number of trainable parameters; for GBDT the node count of full trees, trees × (2^(depth+1) − 1), e.g. 600 × 127 = 76200.
  • Training seconds — training time as recorded by the script (seconds).

1. Main results (out-of-fold, threshold at 0.1% FP)

5 rows · 8 columnsCSV
GBDT 600× d6 (600 trees, depth 6)0.9670.96639.3% (564/1436)4/43530.99376200568.2
GBDT 300× d4 (300 trees, depth 4)0.9630.96236.5% (524/1436)4/43530.9929300159.4
MLP 1M (561-1024-512-1)0.9430.92323.8% (342/1436)4/43530.9881100801307.0
MLP 0.2M (561-256-256-1)0.9420.92518.0% (259/1436)4/43530.98520992167.6
Logistic regression (linear)0.8960.8949.5% (137/1436)4/43530.9725621.9

The input layer of the MLPs (561) is the TLSH feature vector length; logistic regression has 561 weights + 1 bias = 562 parameters.

2. Per-fold thresholds (summed over the 5 test folds)

Here each test fold is evaluated with its own threshold (as set by the script) instead of one threshold on the pooled out-of-fold scores; counts are summed over the 5 folds.

5 rows · 7 columnsCSV
GBDT 600× d636.0%498/14362/43530.9961.000100.0%
GBDT 300× d432.8%479/14362/43530.9961.000100.0%
MLP 1M (561-1024-512-1)21.4%378/14363/43530.9921.000100.0%
MLP 0.2M (561-256-256-1)21.7%296/14362/43530.9931.000100.0%
Logistic regression (linear)4.7%72/14365/43530.9350.99250.7%

Mean false positives per fold: GBDT 0.4, MLP 1M 0.6, MLP 0.2M 0.4, logistic regression 1.0 (out of 870.6 clean files per fold).

3. Baseline: TLSH nearest-neighbour rule (no model)

A file is flagged if its TLSH distance to the nearest known malicious sample is ≤ the threshold.

2 rows · 6 columnsCSV
TLSH-NN ≤ 304.7% (68/1436)1/4353 (0.023%)0.9860.5820
TLSH-NN ≤ 5013.7% (197/1436)7/4353 (0.161%)0.9660.5820

The ≤ 50 rule already exceeds the 0.1% FP budget (7/4353 = 0.161%); the project summary lists the TLSH ≤ 50 rule alone as giving ~0% detection at 0.1% FP.

4. Additional recall figures recorded by the script

The results file also contains two further recall values at the 0.1% FP threshold. Their exact definitions are in the training script, not in the results file; the field names are given verbatim.

5 rows · 3 columnsCSV
GBDT 600× d630.9% (383/1239)40.4% (580/1436)
GBDT 300× d428.7% (355/1239)38.4% (552/1436)
MLP 1M (561-1024-512-1)19.8% (245/1239)30.8% (442/1436)
MLP 0.2M (561-256-256-1)12.0% (149/1239)24.1% (346/1436)
Logistic regression (linear)9.2% (114/1239)21.7% (311/1436)

oof_far_recall is computed over a subset of 1239 of the 1436 malicious files; oof_union_recall is computed over all 1436.

5. Observations

  • Trees win clearly: GBDT 600×d6 detects 39.3% of unseen-family malware at 0.1% FP, about 4× logistic regression (9.5%) and 1.65× the larger MLP (23.8%).
  • The larger GBDT gains 2.8 percentage points of recall over GBDT 300×d4 at 8× the parameters and 3.6× the training time.
  • All neural and tree models fit the training folds perfectly (train AUC 1.000, train recall 100%), so the test numbers measure generalisation to new families and apps.
  • The MLP 1M has a higher per-fold test AUC than the MLP 0.2M (0.943 vs 0.942) but a lower OOF AUC (0.923 vs 0.925); its recall at 0.1% FP is higher (23.8% vs 18.0%).
  • A TLSH distance rule alone is weak on unseen families (AUC 0.582); the learned models are needed for generalisation.