How Big Data, cloud computing, Amazon and poll quants won the U.S. election

16 November 2012– In the U.S., for more than a year, a group of what the reporters described as “math geeks and data wizards” worked long days creating elaborate statistical models to determine which voters were likely to vote for the U.S. president. And they were not just working in broad terms — young people, single women, Asians, et al. They were identifying individuals and providing data to field workers who went to the doors of those people and persuaded them to vote.

The man of the hour is statistician and New York Times blogger Nate Silver, who repeated his 2008 triumph. In 2008 he made forecasting elections “nerd-cool” by predicting in the presidential election the winner of 49 of 50 states … and all 35 senate races. And he hit it perfectly again this year. Only this year, he had a lot of company.

The Obama campaign’s technology team built, deployed, ran, and scaled up their applications on Amazon Web Services. Databases were the key, along with very complex applications comparable in scope and complexity to those seen in the largest enterprises and data-rich startups. For example: they had massive data modeling using Vertica and Elastic MapReduce; multi-channel media management via TV, print, web, mobile, radio and email using dynamic production, targeting, retargeting, and multi-variant testing like you’d find in a sophisticated digital media agency; social coordination and collaboration of volunteers, donors, and supporters; massive transaction processing; voter abuse prevention and protection, including capture of incoming incidents and dispatch of volunteers; a rich information delivery system for campaign news, polls, information on the issues, voter registration, and more.

So the quants and their statistical models were right, while the political pundits and their guts were wrong. The victory of mathematics over bloviation has been resounding. For the full post of our analysis please click here.