Big Data

"Big" isn't measured by a number

There's no agreed line like "starting at so many gigabytes" for big data. The bar is relative: the moment your current tools won't open it, and your current methods won't finish computing it, that data is big data right there.

One day's boarding records for a single stop open fine in a spreadsheet program. A month's worth for one line gets a bit heavy. A year's worth for the entire country won't open at all. Same kind of record, but as the amount grows, at some point it crosses the line where you have to switch tools.

So big data describes a relationship between data and tools, more than a property of the data itself. What was too much to handle ten years ago runs on a laptop today. The line itself keeps moving.

Split it up, and split the computing up too

If it doesn't fit on one computer, there's one option: split it across many. Divide national records by region, spread them across several machines, have each one compute only its own share, then gather the results and combine them.

Do this and capacity grows with the number of machines, but new worries come too. If one machine breaks, its share disappears, so the same piece gets duplicated across more than one, and work has to be divided so that moving data between machines doesn't cost more time than the computing itself. Splitting something that would run fine on one machine can actually make it slower, so in practice the first question is whether this data is really large enough to be worth splitting at all.

Speed and shape matter too

Transit card taps are being logged this very second. While yesterday's records are still being organized, today's share piles up just as fast. Waiting until everything's collected before computing means you never catch up, so a separate method is needed: process it continuously, as it streams in.

The shapes don't match either. Card records are a tidy table with clear columns, but the bus location signals and complaint messages used alongside them come in all different formats. Recording methods change over the years too, so old records and recent ones often don't even share the same columns.

That's why talk about big data usually covers volume, speed, and variety together — any one of the three growing on its own is enough to break the tools you already had.

One record is trivial, but together they show texture

A single record carries almost nothing — just the fact that someone boarded at some station on a Tuesday morning. But once tens of millions overlap, things appear that no individual record had: which stretch saturates at which hour, which route people take to transfer on a rainy day.

AI owes a lot of its current strength to exactly this. Showing a machine plenty of examples instead of writing rules by hand only works once the examples are actually plentiful. Once text and images piled up on the internet, and machines could read them in, there were finally enough examples to work with — and that had to arrive alongside enough computing power to actually chew through it all.

A lot piled up isn't the same as well chosen

National transit-card records leave out anyone who doesn't use a card. People who walk, bike, or drive — none of their movement shows up in a single line, no matter how many records pile up. Piling up more doesn't fill in the gap on its own. Stack tens of millions of records that already lean one way, and that lean starts to look like fact — a big number makes people trust it more, which is exactly the danger.

Privacy comes along for the ride too. Once one person's movements are logged long enough, where they live and work shows through even with the name stripped out, which is why this kind of data gets released only in bundled, anonymized form.