Reality Mining

A while back, your scribe wrote about the inexorable march of progress, with all of its unexpected, yet rewarding, effects on our daily lives.

This quote appeared in that post:

The days of standalone GPS devices may be numbered. At least, that’s what cell phone vendors and service providers would have you believe.It turns out they may be right. In the span of about a year, we went from almost no GPS-enabled handsets to close to half of all available models including the navigation features. And that number will only increase, with GPS radios becoming standard in 2008 much like cell phone cameras became the norm back in 2004.

Today, the New York Times reports on the development of applications that are taking advantage of the ubiquity of GPS devices.

Via this story, we see that it hasn’t taken long for the engines of business to find ways to capitalize on the technology. A few tastes:

We’re in the midst of a boom in devices that show where people are at any point in time…

Such data could redefine what we know about consumer behavior, giving businesses early insight into economic trends, better ways to determine sites for offices and retail stores, and more effective ways to advertise.

Just this month, the journal Nature published a paper that looked at cellphone data from 100,000 people in an unnamed European country over six months and found that most follow very predictable routines. Knowing those routines means that you can set probabilities for them, and track how they change…

It’s hard to make sense of such data, but Sense Networks, a software analytics company in New York, earlier this month released Macrosense, a tool that aims to do just that. Macrosense applies complex statistical algorithms to sift through the growing heaps of data about location and to make predictions or recommendations on various questions — where a company should put its next store, for example…

Mr. Jebara, who is also an associate professor of computer science at Columbia University, says the key to drawing such conclusions starts with having very large sets of data that go back several years. Sense’s models were developed initially from sources like taxicab companies that let it look at location data over such a period. Sense also uses publicly available data, like weather information, and other nonpublic sources that it would not disclose. “We had three-quarters of a billion data points from just one city,” Mr. Skibiski says.

Mr. Jebara’s statistical models interpret those patterns and look at whether they correlate with things in the real world, like tourism levels or retail sales. The algorithms are complex. Even so, the model doesn’t work for everything Sense tries it on, often because more data is needed. But Mr. Jebara says that when it has the data, the model works well. Several hedge funds made an investment in Sense earlier this year.

The Macrosense tool lets companies engage in “reality mining,” a phrase coined by Sandy Pentland, an M.I.T. researcher who was also a co-founder of Sense and now advises it on privacy issues…

“The reality is that location data is new, and we don’t have 10 years of history to work from,” says Ted Morgan, the chief executive and founder of Skyhook Wireless, which sells a service that lets people use WiFi network access points to get information about their location.

“But if their algorithms can do the things they say, we’d probably do a lot with them,” Mr. Morgan says.

Read the whole thing.

Friends, things are about to get very, very interesting.

Cross-posted at Gates of Academe.