In applied machine learning, the model is the tiebreaker

This is the supervised counterpart to our fuzzy-hashing research, and it exists to answer a question that work left open: on the same class of problem, with labels available, what does supervised learning actually buy you?

The system is seven decoupled stages. Collect labelled captures, extract packet and flow features, clean and balance, select features by ensemble vote, train four classical models plus a convolutional network, evaluate with real confusion matrices, and deploy a detector that reads a live interface and raises an alert above a probability threshold. Roughly fifty hours of work, accounted for honestly. Most of it was not modeling.

def run_pipeline(self):    self.collect_data()        # 1. PCAPs -> data/raw_pcaps/{normal,malicious}    self.extract_features()    # 2. PCAPs -> packet/flow CSVs (scapy)    self.clean_data()          # 3. dedupe, IQR clip, engineer, scale    self.select_features()     # 4. ensemble vote across 4 methods    self.train_models()        # 5. SVM/RF/XGB/MLP + 1D CNN    self.evaluate_models()     # 6. metrics, ROC, confusion matrices    self.deploy_model()        # 7. live scapy sniff -> predict_proba -> alert

The choice that decided everything¶

The most consequential decision came at stage two, and it was not a modeling decision. A naive extractor labels every packet and trains on packets. That gives you millions of training samples at the cost of making most labels false, because a malicious session contains plenty of ordinary handshakes and keepalives.

Aggregating packets into flows, keyed on source and destination address, both ports and the protocol, and computing statistics over each flow, is what makes the labels true. Packet counts, byte totals, duration, packet size distribution, inter-arrival time distribution, and derived rates. The model got dramatically easier the moment the unit of prediction matched the unit the label described.

flow_key = (    packet[IP].src,    packet[IP].dst,    src_port,    dst_port,    protocol,)flows[flow_key].append(packet)

Two of the features computed over those flows carry most of the separation, and it is worth seeing how much of the result is decided before any model exists. Below, the flows are laid out on those two axes with a threshold rule over them. Move the cuts and the confusion matrix moves with them.

Where the accuracy actually came from¶

Switching the outlier treatment from dropping at one and a half times the interquartile range to clipping at three times it moved validation F1 by roughly three and a half points. No grid search we ran produced a delta anywhere near that size. That single comparison is the most useful thing this project taught us, and it is the opposite of where attention usually goes.

numeric_columns = df.select_dtypes(include=[np.number]).columnsfor col in numeric_columns:    if col != 'label':        Q1 = df[col].quantile(0.25)        Q3 = df[col].quantile(0.75)        IQR = Q3 - Q1        lower_bound = Q1 - 3 * IQR        upper_bound = Q3 + 3 * IQR        df[col] = df[col].clip(lower_bound, upper_bound)

Deduplication was the other surprise. Dropping duplicate flows removed about twenty-two percent of rows. We assumed the counter was broken; it was not. Public capture datasets overlap heavily, because everyone samples from the same well-known sources. A model trained without that step is being evaluated partly on data it memorized.

Feature selection by disagreement¶

We selected features by ensemble vote across four methods rather than trusting one: an F-test, mutual information, recursive feature elimination and random-forest importance. The value is in the disagreement, not the agreement.

We expected mutual information to be strictly more permissive than the F-test, on the reasoning that a non-linear test should accept everything a linear one accepts and more. It is not, because the two score on different axes, variance against entropy. Engineered binary features sat in mutual information’s sweet spot while looking irrelevant to the F-statistic. A single-method selection would have dropped features that carried real signal, and the ensemble vote exists precisely to cover that gap.

top_f    = set(f_scores['feature'][:20].tolist())top_mi   = set(mi_scores['feature'][:20].tolist())top_rfe  = set(rfe_features)top_tree = set(tree_features) for feature_set in [top_f, top_mi, top_rfe, top_tree]:    for feature in feature_set:        feature_counts[feature] = feature_counts.get(feature, 0) + 1 # Keep anything at least two of the four methods asked for.self.selected_features = [f for f, c in feature_counts.items() if c >= 2]

The deep model that did not earn its keep¶

We trained a one-dimensional convolutional network alongside the classical models, with the full regularization stack: batch normalization, dropout, weight decay and early stopping. It lost to gradient-boosted trees, and not narrowly.

The reason is structural rather than a tuning failure. Convolutions exploit local ordering. Engineered flow statistics have no meaningful ordering between columns, so there is nothing for the architecture to exploit and its inductive bias is simply wrong for the data. If we wanted a deep model to win, the input would have to be raw packet sequences rather than flow scalars, which is a different project with a different data pipeline.

We report this because the incentive in most engagements runs the other way. A convolutional network sounds like more value delivered than a boosted tree. It is not, and a firm that lets a client pay for the more impressive-sounding model when the boring one wins is not doing the client a service.

feature               MI   chi2   RFE    L1   votesflow_iat_std           *      *     *     *      4fwd_pkt_len_max        *      *     *     .      3init_win_bytes_fwd     *      .     *     *      3bwd_pkt_len_mean       .      *     .     *      2   dropped--stage 6/7  gradient-boosted trees   F1 0.981stage 6/7  1D convolutional net     F1 0.974

The number the client actually argues about¶

A trained classifier does not raise alerts. A threshold on its output does, and choosing that number is where the project stops being a modelling exercise. Every position on the slider trades a missed detection against a false one, and there is no position that is optimal in any sense the mathematics can settle.

It is a business decision wearing a technical costume: what does an investigation cost, what does a miss cost, and how many alerts will the people on the receiving end actually work before they start closing them unread. We present it to clients exactly like this, as a curve with a handle on it, because a threshold chosen by the engineers is a threshold nobody in operations agreed to.

Engineering that made the rest survivable¶

• Every stage writes its output to disk and reads its input from disk, so a crash at stage five does not force a re-collection at stage one. Unglamorous, and the single most valuable structural decision in the project.
• Explicit pipeline state, recording which stages have completed, so any run is replayable from any point.
• The selected feature column list is written to disk at training time and read back at inference. Never regenerated. Column order is part of the contract between training and serving, and regenerating it is a class of bug that produces confident nonsense rather than an error.
• One configuration file as the single source of truth for paths, thresholds and hyperparameters, so a run is reproducible by someone who was not there.

Profile the thing, not the story about the thing¶

We expected model inference to be the bottleneck in the live detector, because inference is the part that feels expensive. It is not close. Scoring a flow is sub-millisecond. The packet parsing and field extraction loop in front of it is at least an order of magnitude slower. Optimizing for throughput means replacing the parser, and touching the model would accomplish nothing.

This generalizes badly for anyone selling AI and well for anyone buying it. The expensive part of a machine learning system is rarely the machine learning. It is acquisition, cleaning, labelling, serving and the plumbing between them, which is exactly the work that gets under-scoped because it is not the part in the demo.

What we would do differently¶

• Model the pipeline as a dependency graph rather than a script, so stages can be re-run individually, cached by content hash and parallelized without rewriting the orchestrator.
• Move oversampling inside the cross-validation loop from day one. Oversampling before the split looks fine until a reviewer asks how leakage was avoided, and the honest answer is that it was not.
• Version the intermediate datasets, not just the code. "Why did the score drop on Monday" should be a two-minute question.
• Plot the per-class distribution of every feature before training anything. Half the bugs and most of the wins are visible in those histograms.

The most valuable artifact the project produced was not the trained classifier. It was seven decoupled stages, an explicit column contract and a confusion matrix on every model. Every component inside that skeleton is replaceable. The skeleton is what we carry forward, and it is what we build for clients before we build them a model.