# ML classifiers on TLSH fingerprints of Mach-O files

> Gradient-boosted trees are the best model on TLSH features — GBDT 600×d6 reaches out-of-fold AUC 0.966 and 39.3% recall at a 0.1% false-positive rate, ahead of MLPs (18–24%) and logistic regression (9.5%); nearest-neighbour TLSH distance rules alone are far weaker (AUC 0.582).

Area: Security (macOS malware scanning). Page: https://benchmarks.site/ml-tlsh-macho-classifiers/. Methodology and caveats: https://benchmarks.site/methodology/#malware

Date: 2026-11-03.
(file types `macho` and `macho_fat`).

The question: can a classifier trained on TLSH fuzzy-hash fingerprints of Mach-O binaries recognise
malware it has not seen, better than the plain rule "TLSH distance to a known sample ≤ threshold"?

## Dataset

| Set | Count |
|---|---|
| Malicious Mach-O files (positives) | 1436 |
| Clean Mach-O files (negatives) | 4353 |
| Benchmark samples (`bench`) | 547 |
| Malware families | 23 |

Validation: grouped 5-fold cross-validation, where groups keep whole malware families and whole
applications in one fold, so the test fold always contains families and apps not seen in training.
Per fold this gives on average 287.2 positives and 870.6 negatives in the test part.

## Metrics

- **AUC test** — ROC AUC on the test folds, averaged over the 5 folds (`auc_te`).
- **OOF AUC** — ROC AUC on the pooled out-of-fold predictions of all 5 folds (each file scored by the model
  that did not see it in training) (`oof_auc`).
- **Recall at 0.1% FP (OOF)** — share of the 1436 malicious files detected when the decision threshold
  on the pooled out-of-fold scores is set so that at most 0.1% of the 4353 clean files are flagged
  (`oof_rec_at_0.1`). This is the main number: a scanner must keep false alarms very low.
- **FP at that threshold** — clean files flagged at that threshold (`oof_fp_at_0.1`); 4/4353 = 0.092% for every model.
- **Precision (OOF)** — share of flagged files that are malicious at that threshold (`oof_prec_at_0.1`).
- **Params** — number of trainable parameters; for GBDT the node count of full trees, trees × (2^(depth+1) − 1),
  e.g. 600 × 127 = 76200.
- **Training seconds** — training time as recorded by the script (`seconds`).

## 1. Main results (out-of-fold, threshold at 0.1% FP)

| Model | AUC test | OOF AUC | Recall at 0.1% FP | FP at threshold | Precision | Params | Training (s) |
|---|---|---|---|---|---|---|---|
| GBDT 600× d6 (600 trees, depth 6) | 0.967 | 0.966 | 39.3% (564/1436) | 4/4353 | 0.993 | 76200 | 568.2 |
| GBDT 300× d4 (300 trees, depth 4) | 0.963 | 0.962 | 36.5% (524/1436) | 4/4353 | 0.992 | 9300 | 159.4 |
| MLP 1M (561-1024-512-1) | 0.943 | 0.923 | 23.8% (342/1436) | 4/4353 | 0.988 | 1100801 | 307.0 |
| MLP 0.2M (561-256-256-1) | 0.942 | 0.925 | 18.0% (259/1436) | 4/4353 | 0.985 | 209921 | 67.6 |
| Logistic regression (linear) | 0.896 | 0.894 | 9.5% (137/1436) | 4/4353 | 0.972 | 562 | 1.9 |

The input layer of the MLPs (561) is the TLSH feature vector length; logistic regression has 561 weights + 1 bias = 562 parameters.

## 2. Per-fold thresholds (summed over the 5 test folds)

Here each test fold is evaluated with its own threshold (as set by the script) instead of one threshold
on the pooled out-of-fold scores; counts are summed over the 5 folds.

| Model | Recall test (mean over folds) | Malicious detected (sum) | False positives (sum) | Precision | AUC train | Recall train |
|---|---|---|---|---|---|---|
| GBDT 600× d6 | 36.0% | 498/1436 | 2/4353 | 0.996 | 1.000 | 100.0% |
| GBDT 300× d4 | 32.8% | 479/1436 | 2/4353 | 0.996 | 1.000 | 100.0% |
| MLP 1M (561-1024-512-1) | 21.4% | 378/1436 | 3/4353 | 0.992 | 1.000 | 100.0% |
| MLP 0.2M (561-256-256-1) | 21.7% | 296/1436 | 2/4353 | 0.993 | 1.000 | 100.0% |
| Logistic regression (linear) | 4.7% | 72/1436 | 5/4353 | 0.935 | 0.992 | 50.7% |

Mean false positives per fold: GBDT 0.4, MLP 1M 0.6, MLP 0.2M 0.4, logistic regression 1.0 (out of 870.6 clean files per fold).

## 3. Baseline: TLSH nearest-neighbour rule (no model)

A file is flagged if its TLSH distance to the nearest known malicious sample is ≤ the threshold.

| Rule | Recall | False positives | Precision | AUC test | Params |
|---|---|---|---|---|---|
| TLSH-NN ≤ 30 | 4.7% (68/1436) | 1/4353 (0.023%) | 0.986 | 0.582 | 0 |
| TLSH-NN ≤ 50 | 13.7% (197/1436) | 7/4353 (0.161%) | 0.966 | 0.582 | 0 |

The ≤ 50 rule already exceeds the 0.1% FP budget (7/4353 = 0.161%); the project summary lists the TLSH
≤ 50 rule alone as giving ~0% detection at 0.1% FP.

## 4. Additional recall figures recorded by the script

The results file also contains two further recall values at the 0.1% FP threshold. Their exact
definitions are in the training script, not in the results file; the field names are given verbatim.

| Model | `oof_far_recall` | `oof_union_recall` |
|---|---|---|
| GBDT 600× d6 | 30.9% (383/1239) | 40.4% (580/1436) |
| GBDT 300× d4 | 28.7% (355/1239) | 38.4% (552/1436) |
| MLP 1M (561-1024-512-1) | 19.8% (245/1239) | 30.8% (442/1436) |
| MLP 0.2M (561-256-256-1) | 12.0% (149/1239) | 24.1% (346/1436) |
| Logistic regression (linear) | 9.2% (114/1239) | 21.7% (311/1436) |

`oof_far_recall` is computed over a subset of 1239 of the 1436 malicious files; `oof_union_recall` is
computed over all 1436.

## 5. Observations

- Trees win clearly: GBDT 600×d6 detects 39.3% of unseen-family malware at 0.1% FP, about 4× logistic
  regression (9.5%) and 1.65× the larger MLP (23.8%).
- The larger GBDT gains 2.8 percentage points of recall over GBDT 300×d4 at 8× the parameters and 3.6× the
  training time.
- All neural and tree models fit the training folds perfectly (train AUC 1.000, train recall 100%), so the
  test numbers measure generalisation to new families and apps.
- The MLP 1M has a higher per-fold test AUC than the MLP 0.2M (0.943 vs 0.942) but a lower OOF AUC (0.923 vs
  0.925); its recall at 0.1% FP is higher (23.8% vs 18.0%).
- A TLSH distance rule alone is weak on unseen families (AUC 0.582); the learned models are needed for
  generalisation.
