Big Data FAQ

What Are The Biggest Challenges With Doing Big Data Analytics?

Before I begin, I am a current employee of Looker and my industry knowledge comes largely from my experience at Looker. That said, I have done my best to answer this question impartially, although having more knowledge of Looker than any other BI platform sways this.

I think the single greatest challenge facing data analytics in the 21st century is the so-called “utilization gap.” Every major company has vast stores of information in increasingly complex databases. However, despite having more data than ever before, most data analytics still fail to provide actionable insights.

This common problem has been well documented with hard facts describing both the extent and causes of the problem (The Utilization Gap:  Big Data’s Biggest Challenge). And a few companies have actually done a lot to address it. In terms of databases themselves, companies like Teradata Aster have continued the trend towards fully in-database analytics encompassed by fully unified database architectures. This allows companies to gather data from disparate sources into a single database. Then, with in-database analytics your data never moves, removing the tradition ETL layer at query time. Looker is an excellent example of this. Looker connects directly to any relational database, querying data at the source. With this capability users across a business have all their data in one place, available at all times, providing a single source of truth for everyone involved. This innovation also eliminates the need for proprietary databases where subsets of a company’s data are stored temporarily.

Efforts to bridge the utilization gap extend to BI platforms as well with business intelligence becoming more user friendly with each iteration, allowing the typical business user to query data themselves. For example, every popular BI platform provides reasonably intuitive UIs that allow normal users to find basic visualizations and charts (Tableau, Birst, etc.). However, visualizations are often not enough. Modern users want to explore databases themselves. This is possible on a few platforms today in large part due to the advent of modeling languages that query relational databases automatically, filling in SQL code with the press of a button. However, modeling languages are still brand new. As far as I can tell only Looker has a fully functional language based modeling layer (LookML). As this comes with huge practical advantages it will become industry standard sooner than later.

Further addressing the utilization gap is a growing trend towards more analytical, explorative BI. By this I mean BI that focuses on exploring data to reveal insights rather than BI that focuses on visualizations. With modern BI the user does not need to know exactly what data they are looking for. Instead they can take advantage of more powerful analytics to find patterns, relationships, and outliers that have traditionally remained hidden. Here again older systems that focus on the destination rather than the journey (think legacy style systems and platforms whose focus is visualizations) will, in my view, phase out in favor of these more agile platforms.

Of course, security and privacy are perennial concerns, but ones that any self-respecting BI platform addresses thoroughly.

In choosing the right solution you need to discern your needs. If you think simple visualizations are all you need, there are a lot of great options on the market. If you are looking for something that allows you to explore your data thoroughly and make connections, consider a platform that is a bit more complex to learn, but far more powerful. If you are at all technical (i.e. have any sort of familiarity with SQL) I would recommend a more agile solution with a language based modeling layer. If you have no analytics team and no interest in learning, go with a simpler visualization platform.

If you’d like more info on Looker, please don’t hesitate to reach out.

Read Kyle Coleman‘s answer to What are the biggest challenges with doing big data analytics? on Quora

You may also like

Read More