Big data projects are complex to plan and execute. The complexity stems from your need to perform data discovery before you can document a single user requirement. If you lack clear business requirements, you cannot plan the remaining project logistics, including team, skills, execution steps, rollout, and training.
Another issue with big data projects is the lack of availability of business subject matter experts (SMEs) who have both data and institutional knowledge as well as a command of math and logic. Organizations often shy away from big data due to this critical resource — the data scientist.
Here are some of the challenges we face while implementing Big Data analytics?
Getting the technology right
As mentioned in the introduction, this article isn’t about Technology. This being said, there are a few points that need special attention. The Big Data technology space is still in its infancy which means the number of actors in the market is still quite high and evolving pretty fast. The choice of solutions is very large and can therefore fit many different needs. Obviously, it makes use of all technology options such as SAAS, cloud, virtualisation, mobile, etc.
As a consequence, it can be very easy to get lost in this jungle of possibilities. Best advice? Start small, Think big. Investing massively from the start would be detrimental to your business case.
One note for later: visual analytics. Mark my word. It’s not only a nice graphical trend. It’s the future of analytics.
Business analytics and Big Data vendors are eager to knock on your door with turnkey reports and easy ways to get started with Big Data — but all too often, the tendency of end business users is to ask that the top 10 to 20 reports they’ve been using for the past 15 years get converted to the new solution first. This isn’t a good way to use Big Data — or to help the company get closer to answering tough business questions that have eluded it in the past. Knowing how to query Big Data to answer the big questions is also where present skills fall short in businesses. One way to grow this skills area is to contract with the vendor (which usually has Big Data trainers and specialists on staff) to provide Big Data/business analytics training to end users as part of the solution implementation process.
Using Agile Big Data Analytics
As previously stated, the role of business stakeholders is key to the success of Big Data Analytics within an organisation. They are key to the development of the initiative in their ability to identify the right use cases and also in delivering successful outcomes. Delivering a BDA use case can be compared to delivering a project. But in this case, the standard waterfall approach to project management should be avoided.
Coming back to the maturity on analytics, you will notice that this maturity also evolves significantly during the lifecycle of a BDA use case. Business users have a general idea of the scope when the analytics process is launched, but they need the BDA team to guide them in defining a detailed scope. Once the data is extracted, the exploration process provides more insights to the users and helps them to further refine their understanding of the desired outcome. Then, in the core analytics phase, the various stages of results allow users to engage with the upcoming outcome.
A traditional project management approach wouldn’t allow for a sufficient level of interactions between the users and the BDA team. Doing so would result in an analytics deliverable that wouldn’t fit business needs. For this reason, applying an Agile project management methodology, SCRUM-like, addresses this weakness.
The aim here is to work in a series of iterations involving business users and the BDA team along the various phases of the analytics process: from scope definition, requirements gathering, data extraction, data exploration, analytics phases, to delivery. Working in small iterations and in close collaboration with the users ensures the delivery of a meaningful business outcome (3). One to which the business users can relate because they will have been involved in the full lifecycle.
Time to time cleaning up of data
Big Data and business analytics are only as good as the data itself. This is why cleaning up data to ensure that incomplete, inaccurate, and duplicate data is removed should be the first step of any Big Data project. The CIO must explain this and secure top management’s support for a Big Data cleanup, which will seem to those on the outside as a lot of effort expended for no tangible results. The best approach to selling the process is to present the facts upfront so there are no surprises.
Finding the right resources and skills
Everyone is scrambling to find people who can run Big Data and business analytics. Who are these people? Some are statistical engineers who are trained in logic, mathematics, computer science, and complex problem-solving involving huge amounts of data. These data engineers and analysts are not the people currently performing analysis and programming on your IT staff. That’s why many companies initiating Big Data and business analytics projects either hire consultants to train their people or look to hire data engineers. Data engineers are in high demand and they are very expensive. For CIOs, fostering an aggressive training program for internal staff members whom you believe you can develop into Big Data specialists might be the best bet.
What best practices can you recommend to avoid these problems?
In order to implement a big data project, here are a few tips.
First, create a powerful team that can set up a platform to explore big data. This team will work with business and data analysts to create the road map for further execution. Critical success factors include:
- Availability of IT resources to build and configure the selected big data platform
- Availability of business SMEs with data and domain expertise
- Availability of resources with BI expertise and deep statistical knowledge
- Implement a technology center of excellence to provide big data infrastructure support
- Extend other BI best practices, including data governance, MDM, metadata management, analytics definition, and visualization, to include big data
- Ensure adequate training for users to understand the new data and its integration into the analytical and reporting platforms
When it comes to people, having a combination of individuals mentioned above will create a team that can leverage each other’s skills and create a unified vision for exploring big data.