The Big Data IT Challenge: Giving Business Users the Data They Want

Last December, International Data Corporation presented a top ten list of technology predictions for 2014. Among those predictions, IDC claims that enterprise spending on Big Data analytics services in 2014 will increase by 21 percent over 2013, to exceed $4.5 Billion. Obviously companies are starting to recognize that their vast pools of data hold immense potential.

But spending money on Big Data and extracting real value from it are two different things. According to research by Gartner published in a 2011 article on, between 70 and 80 percent of corporate business intelligence projects flat out fail.

At Gartner’s 2013 Business Intelligence and Analytics Summit, a panel of vendors agreed that more than 70 percent of analytics projects failed to meet expectations.

What is the main reason for that kind of failure? According to Gartner, it’s mainly due to a lack of communication between IT and C-level execs. Clearly IT has some challenges to overcome in order to give business users the data they want and need to improve and optimize business decisions and gain competitive advantage. Among the bigger challenges IT faces are:

Departmental dissonance and data silos

Among the major obstacles that IT must overcome in giving business users the data they want is the sequestering of data among various silos and systems in the enterprise. This problem is compounded when groups within departments in the organization become protective of their own data, power, and position and are unwilling to work with others for the overall good of the organization. Since IT already has a reputation for not sharing their toys in the corporate sandbox, it is incumbent upon IT execs to play a more active role in breaking down departmental data barriers. After all, the real value of data in any organization can only be fully realized by pooling all data and viewing it as a whole.

Shortage of Data Scientists

As the sheer volume of both structured and unstructured data grows at explosive rates, IT departments are feeling the crunch, due to the very real shortage of data scientists. Collecting and storing data is one thing, but only data scientists have the advanced backgrounds in computer science, statistics and the functional expertise to effectively analyze data to gain actionable insights. According to Google Trends, the interest and demand for data scientists has never been greater than it is now. And yet, according to a recent study by UCLA, “Students who are pursuing a degree in science-related fields are dropping out or switching their major at increasing numbers.” These disturbing statistics call for new solutions, in the form of new technologies that can increase the influence and impact of existing data scientists while providing a pathway for more people to become data scientists in order to meet the demand.

That being said, in order for IT to be successful in giving business execs the data they want, IT needs to have individuals with the leadership skills that will allow them to work flexibly and effectively with business users.

Poor collaboration and communication

In attributing a lack of communication between IT and business as the primary reason that business intelligence projects fail, Gartner analysts explain that until IT teams start thinking about what the business really needs, they will continue to ask the wrong questions, and by doing so, fail to deliver. In order to find out what the business needs, IT needs to better communicate and collaborate with business users. But as Patrick Meehan, president and research director in Gartner’s CIO Research group told ComputerWeekly, “Part of the problem is that CEOs and CIOs approach BI from different directions. They have not sat down and created a common lingua franca for BI. How can you meet expectations when you have one side speaking Latin and the other speaking Greek?" In answer to this statement, a number of data scientists would likely argue that the reason they haven’t sat down with business execs is that they have not been invited to the table. If that is the case, it’s incumbent upon IT leaders to invite themselves.

By increasing efforts to align with all departments in the organization and to communicate and collaborate more effectively, IT can build a level of mutual understanding and trust. Understanding and trust that needs to exist in order to provide business users the data they want and need to drive corporate success with big data technology projects. If you are needing assistance with such projects then know that Qubole is there to assist you.