Big Data Challenge: The Right People

4/25/2012

By Somaditya Sinha

According to IBM, we create 2.5 quintillion bytes of data every day. Ninety percent of the data we have has been created in the past two years and the amount of data is expected to increase exponentially. The data we create is expanding rapidly as enterprises capture more data in greater detail, as multimedia becomes more common, as social media conversations explode and as we use the Internet to get things done. This is “big data,” and it’s getting even bigger.

Source Image: My Melting Brain/Flickr

Big data is complex. It’s complex because of the variety of data that it encompasses – from structured data, such as transactions we make or measurements we calculate and store, to unstructured data such as text conversations, multimedia presentations and video streams. Big data is complex because of the speed at which it’s delivered and used, such as in “real-time.” The right people need to see the data instantaneously so that they can make informed decisions. And obviously, big data is complex because of the volume of information we are creating. We used to speak in terms of megabytes and gigabytes of home storage – now we speak in terms of terabytes. Enterprises speak in terms of petabytes.

Big data presents a number of challenges relating to its complexity. One challenge is how we can understand and use big data when it comes in an unstructured format, such as text or video. Another challenge is how we can capture the most important data as it happens and deliver that to the right people in real-time. A third challenge is how we can store the data, and how we can analyze and understand it given its size and our computational capacity. And there are numerous other challenges, from privacy and security to access and deployment.

Loraine Lawson of ITBusinessEdge recently noted that discussions about Big Data are shifting away from the technical issues involved toward the “strategic value” of Big Data and “how organizations can really put it to work.” Lawson states that Big Data is more or less useless without quality people” to translate Big Data stores into usable, relevant information.”

Lawson writes, “Maybe it’s my past as a PR person, but before processes, before business cases, I think it’s important to choose the right people for any initiative, and Big Data is no exception. In some ways, Big Data will be easy to sell — it’s hip, it’s hyped, it’s happening now. But the realities of Big Data — the data discipline it will require, the shift in how we think about data — these things won’t come so easy.”

She continues,” That’s why the CIO is one of the key people who needs to be on board with Big Data. And to do that, CIOs really need to rethink their role. I know CIOs have been told this before, but insights from Big Data tend to come differently than other data. In the past, you formulated a question, built a query and checked the data. Big Data requires a bit more of an open-minded process than that. You’re not going to look for specific answers, so much as you’re going to hunt for insights.”

Companies are trying to turn Big Data into big dollars with the help of data scientists. As Mashable reported earlier this year, the largest-ever global survey of the data science community yielded some telling results:

Most data science professionals (65 percent) believe demand for data science talent will outpace the supply over the next five years.

Only 1/3 of respondents said they are very confident in their company’s ability to make business decisions based on new data.

A mere 12 percent of business intelligence professionals and 22 percent of data scientists strongly believe employees have the access to run experiments on data – undermining a company’s ability to innovate by rapidly testing and validating ideas.

Less than four in ten of business intelligence analysts and data scientists (38 percent) strongly agree that their company uses data to learn more about customers.

While it's clear there is gold in the hills, the hard part is taking it out. McKinsey Global Institute estimates the business world would need other 190,000 data analysts to extract value from the growing mountain of data. Wealthy companies like Facebook appear intent on hiring as many analysts as they can find -- at top salaries. Even Wall Street investment houses find it hard to recruit the best analysts. At Syracuse University, the GET undergraduate minor program is designed to introduce those who do not have existing experience in the field of information technology about large-scale information systems across various industries while providing those students who do have a technology background with the opportunity to leverage their existing skills and develop new ones in a real-world context. This minor gives students enrolled in the program a leg up over other job aspirants out there.

Big data, which refers to a collection of several sets of data ranging from customer data to competitive data, and online data to offline data, presents an opportunity for businesses who can use that information to better understand their customers and determine how to target new customers. The key to taking advantage of this data comes from people who are able to effectively analyze and process the information.

Do you feel you have the right skills to become a data scientist? Would you want to improve your data analytical and retrieval skills? Tell us your thoughts in the comments section!

Somaditya Sinha is a 2nd year Masters student in the Information Management program at Syracuse University. He is interested in management of large data sets and the design of data management systems. Contact him at sosinha@syr.edu or follow him on twitter @somsinha86.