According to a recent study, by data integration company Informatica, more than 67% of respondents viewed “big data” as a business opportunity rather than a risk. The survey asked almost 600 business and IT professionals their views on a range of big data issues and over a third were either testing/piloting, or had in production already, big data related projects with the aims of achieving greater operational efficiencies and increasing business agilities.

One of the big challenges identified by the respondents were the lack of tools, or the lack of maturity of tools, to handle the task of big data analysis. Reflecting on this one problem with the existing tools for big data is that they are often quite complicated to operate and with data now falling outside the realms of the traditional information technology department and directly into the hands of the user base these tools need to become simpler and more intuitive.

Business intelligence tools are already playing catch up but dealing with big data requires a different perspective as often big data can sit outside of databases and can include additional layers of meta-data. Hadoop is one such open tool that utilises the power of distributed processing architectures for handling big data sets.

At a recent conference IBM outlined how they are tackling big data and in doing so it looks as if the big blue is set to migrate from being a builder of things to an information services company. Microsoft are aware of the challenge of big data and are already incorporating tools such as Apache Hadoop and Windows Azure into its “Big Data Solutions”. Big data plays a key role in the latest revisions of SQL Server 2012 which also ties into the various business intelligence tools to allow analysis, as Microsoft say, “on all data, including those in Hadoop.”

One company that is well versed in crunching large amounts of information, and doing it quickly, is Google who are working on an online analytical processing system dubbed BigQuery that allows users to run secure SQL-like queries against massive sets of data (terabytes of data and trillions of records) and process queries that return, as Google puts it, “up to billions of rows”. Access to the service is through a subscribe web service.