Feature

1Choosing what to look at shapes performance

The same material can produce very different results depending on which values get pulled out of it. To estimate a used car's price, the model year and the mileage are essential, while the license plate number is completely useless. Leave out a value you actually need, and no method, however good, will get you there.

Values don't always get used as-is; they're often reshaped first. Age is more useful than a birth date; the number of years since a car left the factory is more useful than the release date and today's date side by side. Building useful values like these out of raw material is called feature engineering.

Back when machine learning was first catching on, this work was most of the job. Someone who knew the subject well, picking out what to look at, mattered more than swapping in a different model.

2Values with no connection can get in the way

It seems like adding more values should help, but it doesn't work that way. Mix in a value that has nothing to do with the judgment, and the model can pick up on whatever coincidental pattern showed up in that value. If, purely by chance, the cars with higher plate numbers in the training data happened to be pricier, the model learns that relationship as if it were real.

A value like that holds up only in the training material and tends to collapse once new material comes in. That's why, when choosing features, the bigger concern is usually trimming values that don't help rather than adding more.

Overlapping values cause trouble too. Put height in centimeters into the mix and then add it again in meters, and it tells the model nothing new while adding to the computation.

3Features and labels sit in different places

Picture one record as a row in a table: the earlier columns are features, and the last column is the label. A feature is the question the model has to work out an answer to; a label is the answer known ahead of time. Sometimes a person attaches it, and sometimes it comes from records, such as sales figures.

Both are needed for training — the answer produced from the features gets checked against the label, and the numbers get adjusted by however much they missed. Once the model is actually in use, though, there's no label. New material arrives with features only, and figuring out the answer is the model's job.

Mix this up and it causes a real mistake. Slip a value into the features that can only be filled in once you already know the answer, and test scores look great while the model turns out useless in practice, since that value can't be had in the real world.

4Everything has to become a number before it can be computed

A model can't work with letters or pictures as they are. Whatever the feature, it eventually has to turn into a number. A value like refrigerated-or-not gets spread across present-or-absent slots; a value with no natural order, like region or category, gets the same treatment, split across separate slots.

In many models, values that differ wildly in scale need their scale matched too. If one value moves between 0 and 1 while another swings in the hundreds of thousands, the larger one ends up dragging the whole judgment along with it. So values are often rescaled to a similar range beforehand. Some methods, like tree-shaped models that split on the order of values alone, aren't affected by scale.

5These days the model finds its own ingredients

This picture has shifted a lot as deep learning has spread. Nobody has to tell the model what to look for in a photo — a model built from stacked layers builds useful ingredients directly out of the raw pixels, layer by layer. Early layers pick out something simple, like lines and edges; later layers catch chunkier shapes, like a wheel or the outline of a face.

That hasn't erased the human role, though. With table-style data in particular, features a person has shaped still often carry a lot of weight. What material to feed in, and what values to leave out entirely, are still calls a person makes. Values like age or region, in particular, which can steer the model toward bias, need careful judgment about whether to include them at all.