Kenneth Cukier is data editor of The Economist and co-author with Viktor Mayer-Schonberger of “Big Data: A Revolution That Will Transform How We Live, Work and Think.”
It used to be that knowledge was power. To put a finer point on it, today, data is power.
That’s the premise behind the new book “Big Data: A Revolution That Will Transform How We Live, Work and Think.” In it, Kenneth Cukier, data editor with The Economist, and co-author Viktor Mayer-SchÖnberger examine how our lives are changing as a result of our growing capacity to rapidly process and analyze data.
Cukier crunched the numbers and decided it would be worth the time to talk with REIT magazine about his new book.
REIT: You serve as data editor for The Economist, which strikes me as a unique title. To what degree would you say that economic journalism has become an exercise in data analysis?
Kenneth Cukier: I think economic journalism is improved when we use data wisely, but I would be aghast if journalism ever genuflected before the data and gave up doing what it does well (when it is done well), which is leave the spreadsheets and datasets behind, and examine what’s happening on the ground and the impact this has on people’s everyday lives. Economic journalism can’t be about the data but about people. We need to use data to help us understand the impact on people. It’s not just about the data—though the data can inform us and enrich our understanding in a myriad of ways.
The good news is that newsrooms are getting more comfortable with handling massive data sets and presenting new forms of information to readers, as well as harnessing new visualization techniques. The Economist has done that for more than a century. And in the past 25 years, we’ve become notorious for things like the “Big Mac Index” that examines purchasing power parity in many countries through the lens of the local price of a McDonald’s hamburger. But other papers are brilliant in their uses of economic data, notably the New York Times and the Wall Street Journal. The Times in particular is inspiring for its extraordinary design and computer programming skills—it is redefining data-visualization in the 21st century and serving readers’ needs very well.
REIT: It’s intriguing to me that “big data” has grown into its own concept separate from plain, old data. What’s the difference?
Cukier: The term is used loosely since it emerged over time from numerous fields, and there is no single definition. But there is a “there” there.
In short, the idea is we are able to collect and process more information than ever before, and we’ve found with new techniques that we can do amazing things with a big body of data that we couldn’t when it was in smaller amounts.
One example of this is a branch of computer science and artificial intelligence called machine learning. Take translation. When we tried to program a computer with the rules of translation, the results were atrocious. It’s a hard program to code into rules. So we changed the approach to statistical machine translation, and we fed in millions of pages of documents in French and English and told the computer to score the probability that a word in one language is the best alternative for the word in another. It worked much better, but it wasn’t great. Then, we fed in many orders of magnitude more documents—and the results improved tremendously.
More data, not a smarter algorithm or faster processor, was the basis of the success. That’s new. We couldn’t do those sorts of things before, because we didn’t have enough data. Now, we’re applying the same techniques to create self-driving cars having computers read cell biopsies to determine if there is cancer.
REIT: Can you give us some concrete examples of some of the ways that big data now interacts with our daily lives?
Cukier: We’re just at the beginning of the big data age, but it’s already all around us. For example, spam filters “learn” over time by statistically calculating whether an email is likely spam. The U.S. postal service doesn’t read addresses to sort the mail—a computer does, and extracts the relevant details, but the system needed to be “trained” with lots of data. When Google identifies that we’ve misspelled a search query, it’s not because it has a dictionary loaded into its system: It statistically knows common misspelled variants and also where users eventually click, so it can still offer us up relevant pages results. Even traffic-navigation systems and self-driving cars are big data success stories.
REIT: Do you see any pitfalls in the applications of big data in business or social policy?
Cukier: Absolutely. In the book I recently published with Viktor Mayer-Schönberger, we talk about numerous “dark sides of big data.” The most troubling isn’t “privacy” but “propensity”—the idea that algorithms may predict our likelihood to do things and we’ll be penalized for things that we haven’t actually done, but are only predicted to have a likelihood to do. If that sounds like the science fiction movie “Minority Report,” that’s because it should, since it’s close in nature.
Another big worry is that we succumb to data’s false charms—that we endow in the data more meaning that it merits, and made terrible decisions from that. One need only look at the financial crisis and how investment banks quantified risk in their portfolios and counter-parties to see the degree of disaster that a misunderstanding of data can beget.
REIT: Fundamentally, aren’t all data-driven applications mediated by users and their own personal interpretations and agendas? Ultimately, does that mean our intuition and instincts are really still in charge?
Cukier: Well, yes and no. Yes, if the information is seen and used by people, then they need to apply their judgment to it. But big data is different—it’s often the machines that are crunching the data. And there too, yes, people are still in the driver’s seat. But only in the sense that we devise the technology that we use to produce the outcomes that we want.
Our reasoning is in charge in terms of the data we bother to collect and how we use it—these are a function of our values and decisions. But this also points to why the answer is also “no.” In big data, we are applying our “intuition and instincts” at a higher level of abstraction—not at the issue at hand, but at the computer science approaches that we adopt.
But the bigger point is that big data is allowing us to do things that we couldn’t do before and it can surpass our “personal interpretations and agendas” that you describe in your question. Just think of if credit scoring were to work like statistical language translation. We may strip out some of our biases, intuition and instincts, even if these still exist at the level of computer science, not the level of whether a loan is likely to be repaid.
REIT: If you were advising executives in an industry like commercial real estate, what kind of advice would you offer them about effective use of big data in driving their decision making?
Cukier: It’s clear that in order to do your job well, you need to tap data, not just rely on hunches and experience. No single person can keep in mind the totality of information to make the best decisions all the time. So we need to outsource to the computer and big data what computers and big-data techniques do well.
At the same time, however, it will be vital that executives hone their skills so that they can not only tap the data, but they can excel in applying their wisdom, judgment, imagination and creativity to problems. It is letting the data do what it does best and the human mind do what it does best.
But considering that we’re emerging from a primitive state of affairs in which we did not use data very effectively, the emphasis for the moment should be less on applying our skills and more on having the humility to harness these extraordinary new big-data techniques to learn from data to improve what we do.