
The Big/Small Data Conundrum
I’ve spent most of my time in I.T working in cyber security, SQL, web, and analytics. I first got to know about social media monitoring after learning about a south African based company called brands eye.That’s what sparked my interest in data science about a year ago.I see data science, machine learning and data mining in particular as just the next step in value over traditional business intelligence of data warehousing scenarios. You can think of Business Intelligence as really being the backward facing, looking back in time analysis kind of activity whereas data science is quite frankly a lot about taking that information and actually projecting forward in time which is actually really interesting. So you’re moving from looking backwards and trying to understand what happened to using data maybe to predict or shape what will happen. I think what’s important to me about twitter is that, it’s the closest thing to a global consciousness where you can actually see the world in real time and not only can you see what’s happening from a news and event level but you can actually see how people think and feel about it. And ultimately if we have more of that sort of activity we actually see how people live and what they desire and how they want to see, change and live in the world. And I believe that creates a potential for empathy. And if we have more empathy in the world then we have more opportunity to remove a lot of conflict. So I think broadly we can use these tools to build and measure more empathy around the world. We’re busy exploring space and the physical world but what really interests me are the folks exploring the depths of consciousness and how they interconnect. I think technology provides a model that might teach us using different ways. It’s a fact that products or brands with the highest sales get an equally large amount of mention on Social media platforms. And consequently those brands that have less sales have less mentions. We have lots of places where there are #bigdata. X-box, google, bing and so on. But where I work we don’t have any #bigdata problems. However, that does not mean that you don’t get lots of business value from doing data science on #small data. Big data problems tend to be quite difficult, and so that leaves us to work with what we have. And in doing so we’ll be coming up with ways to make data driven decisions.