DataSift - Democratizing Data Via the Twitter Fire Hose

Written by Bonnie Boglioli...November 17, 2011
DataSift Logo

Intent on democratizing data, DataSift crossed the Atlantic November 16 to much anticipated applause for its US launch. The platform offers real-time social media data filtering in a disruptive and digestible way thanks to its unique relationship with Twitter. DataSift, sprung from Nick Halstead’s genius (the same guy responsible for TweetMeme) was in fact founded four years ago in the UK and has been busy perfecting and growing its social data arsenal so that the rest of us don’t have to.  

Here’s what it does: DataSift’s relevant and nuanced filtering options enable clients to zero in on data points of interest across the major social networks in a user-friendly interface with a real time stream. From monitoring brand perceptions to political campaign engagement and financial service trends, DataSift is a comprehensive platform that gives each client, no matter their vertical, exactly what they seek- no less and no more. No fancy contractors, in-house developers or expensive data collectors.  DataSift’s massive and scalable infrastructure give you the pieces of data that you seek, whenever you want.

“DataSift was born out of the desire to build a data platform where anyone could come along and build out a curated stream and use that in their own business,” Halstead explains to me during a comprehensive demo of the dazzling platform.

Under his company’s initial incarnation (TweetMeme), he was turning away a wide variety of organizations requesting data from his team. “We could only do large scale projects, but there was clearly a huge market for data curation in Twitter. We just needed a better interface to scale beyond what we could do consulting,” he says.

Halstead got just what he needed. With close Twitter ties previously fostered, Halstead and team successfully courted the San Francisco-based micro-blogging giant and gained access to its closely guarded fire hose. It is one of only two such companies with exclusive access and it allows DataSift to search for posts by utilizing the metadata within tweets.

“They don’t give out their fire hose easily,” Halstead says with a pause. “When you’re filtering 250 million tweets, the scale required is pretty big. They knew that we had what it took to deal with that kind of volume of data.”

“Customers really want a cross-section of the data to get the sort of insight they are looking for,” he says. Better yet? “We do the heavy lifting and data processing so clients don’t need hundreds of servers to obtain the same data.”

And oh what data! Clients can layer gender, location and other parameters to gain a better understanding of otherwise overwhelming data. “A lot of this is about showing people that Twitter is more than they know. It has a quarter billion tweets so research on anything can be done out there,” Halstead says with excitement.

The sentiment filter, for instance, shows every positive and negative comment associated within the search parameters defined by the client. “It’s a very processor-intensive task,” says Halstead, geeking out in appreciation of this filtering gem as he continues to walk me through the demo. By overlaying an additional filter to the search, clients can further dissect the data. 

 “One of my favorites is filtering by interests,” he says. “We can look for dog owners, for example, by narrowing the search to users with the term ‘dog owner’ in their profile… now we see what dog owners are discussing in real time.” But that doesn’t have to be dogs. It could be veterans, Baby Boomers or people who ride bicycles to work. If you can imagine a group of people you’d like to gauge, you can do it on DataSift.

 “The Guardian came to us and wanted to track the News of the World Story when it was breaking last summer. It took four of their own developers four weeks to build something for tracking that we could do in 30 seconds,” he says with a chuckle. “And with us, they only had to pay for the one week that they utilized the data.” I inquire what they thought of it. “They absolutely loved it,” he said bluntly.

“As a cloud platform, we’re on-demand and we’re changing how businesses buy data,” says Halstead. That’s because the platform offers pay-as-you-go and monthly packages, melding nicely with the whole ‘democratization of data’ mantra.  

“We give customers access to the whole host but only charge them for the data they actually use,” CEO Rob Bailey explains. “If you only want a little, the pay-as-you-go option is great. But we also offer multiple subscription levels that allow companies of all sizes to choose the best suited fit for them.”

Is DataSift indeed democratizing data, splicing it this way and that to accommodate the needs of varied interests in the cloud? Halstead and team think so.

“I love the term ‘democratization’,” Halstead says. “It gets me excited about our platform. It excites me that we are opening up the floodgates to a whole new round of product development on twitter that hasn’t happened before.”

Want an inside peak for yourself? Head over to the DataSift Test Streams for a sampling of insightful, tailored data that is now at everyone’s fingertips.

Social login powered by Gigya