I believe that right now there is nobody in IT anymore who does not talk about BigData, at least in a buzzword drop. There is a lot of hype around this subject and as usual some of the points are really valid and the others are more ideas or fantasies.
My intention is not to enter into the details and talk about the value or non-value of using BigData but some of the surroundings that I find at least very curious.
BigData is typically defined by the V’s:
Good to see that more and more are adding not the 4th V for VALUE to it. Actually to me this is the most important of all of them. It must be clear to anybody thinking to implement a BigData Scenario of what he wants to achieve with it = VALUE! Probably another great thought behind BigData is to me that we are providing and Answer Platform, without knowing yet the questions. This gives a lot more power to the user. Empowered users typically are happier and thus also become more productive.
Hadoop is probably the leading platform for BigData implementations, it is there, it is proven (Google, Yahoo and many others are probably quite a good reference). It allows companies to create the Answer Platform to questions which have not been asked.
But BigData drives another very important change today one of the foundations of Analytics was that Data hat to come from a trusted source and even more there is just one source of truth. With BigData scenario we need to take into consideration that the data sources can be from anywhere (normally people will now list all the SocialMedia etc…) this is somewhat contrary to what we have been used to, or is it just that we might need to review our view on the definitions.
Is BigData an Super-Amplifier of Rumours?
This seemingly obvious answer is in reality not so simple. One example to me has been the the 1% dip of the DowJones Index within 2 minutes of trading due to a fake Tweet. I might be simplifying and appologize for that but in short what happened was:
- a Tweet under the name of AP was published stating 2 explosions at the Whitehouse and that Obama has been injured
- A panic has been caused on the stock market
- 5 minutes later the market has recovered.
To illustrate the impact during the drop there was a los of approx $200bn in the wink of an eye. BigData has had something todo with it since most trading today is based on computer platforms which take inputs from a variety of sources and based their decisions on very complex formulas and algorithms.
In this case has enough data been used ?
Maybe another paradox is that almost all IT vendors are focusing on Speed, RealTime etc as a key driver to value. This is very true but can be also very complex as the above case has shown. On the stock markets banks pay very premium rates to have their Computers as close to the stock market as possible to gain on milliseconds in latency. So the conjunction of Speed without enough trusted sources can lead to lack of control. Just to make sure that there is no misunderstanding, by no means I am referring to the prestige of the agency which appeared on the tweet!
What about the Human Factor?
The typical reaction when something that should never go wrong finally goes wrong is, “that was a human” error. Actually quite obvious since humans make all the tech we are using today. The human needs of course facts to take a decision but also uses a lot of soft factors. Yet here is another factor which is not a fact nor a soft factor but has very often a strong influence on the decisions taken: Rumours
The above example is nothing else than a Rumour. There has been no fact that there has been really an attack but yet people and then machines believed in the Rumour and took decisions based on it.
The paradox is that we have more and more information available, this makes it even more difficult to determine what is relevant and what not. Saying “I took the wrong decision because I did not have the data available to make a better and informed decision” is getting less and less accepted as an excuse.
Probably it is exactly the human factor that makes our life and world still fun and interesting to live in. Things are not just programmed and black and white. No matter how many sources of trusted information we have, the human decision factor will always consider, experience, feeling and probably also rumours (in all its extends) into consideration.