A guide through the stages of a data & analytics platform

Bilde av Alexander Khorunzhiy
by Alexander Khorunzhiy

24. Mar 2021, 13 minutes reading time

A guide through the stages of a data & analytics platform

Does your organization have a data and analytics platform, or do you consider establishing one? Do your data platform capabilities cover the organization's needs, or you need to boost it to the next level?

This guide will break down the growth and demands of the data & analytics platform into stages. It provides a list of helpful questions. I will use tools and technologies from the Microsoft platform and toolbox as examples, but you can use other platforms as well. The platform could be in the cloud, on-premises, or hybrid. 

 

Stage 0: Establishing the need for a data and analytics platform

Signs that indicate you need a data and analytics platform include:

  • Standard analytics and reporting provided with existing enterprise tools and line-of-business applications is not enough.
  • Manual reporting and analytics get tedious and time-consuming.
  • Need for sharing and collaboration across your data and analytics initiatives. 
  • Number of report users and business analysts outgrow capabilities of simple solutions.
  • Need for integration of data from multiple sources.

If you have answered Yes for one or several questions above, you have established a need for a data and analytics platform. At this point, it is normal to have uncertainty in how much effort your organization can put into a data platform. Here you can consider the first stage. It is easy and quick to implement and might be good enough for many organizations. 

 

Stage 1: Standardized reporting and analysis

Usually, the journey starts with dashboards, reports, and analytics. It is the visual part, which users, stakeholders, and clients get to see first. The implementation is about choosing and purchasing an analytics and visualization tool. A simple Microsoft Power BI Pro license can currently be obtained for a monthly cost of 9,99$ per user.

Capabilities and benefits:

It gets easy to create pretty visualizations, dashboards, and reports. Business users can do that with some training. Collaboration, publishing, and sharing of analytics are easy now.

Other capabilities of visualization tools that can be perceived as benefits at this stage, including the possibility to incorporate business rules and logic into the reports. Many visualization tools can connect to different data sources and do data integration or blending for you.

Screenshot 2021-03-10 at 10.29.30

Signs that you want to boost your platform to the next stage or jump over stage 1.

  • As the number of reports and dashboards grow, you have to start duplicating all your data connections and business logic for each of them, it is tedious work;
  • The previous item also creates a new problem: no single version of the truth; different reports would eventually get deviations in the business logic or small errors here and there, which would result in the illogical inconsistencies in numbers for the "same" data; 
  • Data quality assessment is a manual job;
  • With a certain scale, it becomes unmanageable. 

Guide on assessing needs for creating or upgrading your data analytics platform 2

When manageability, scale, quality, "single version of the truth," and time spent on development become an issue, the next step would be to think about automation.

 

Stage 2: Data ingestion and preparation automation

At this stage, the point is to increase the efficiency of efforts spent on development. Key improvements are increased productivity, reduced time to insights, increased robustness, and scalability. The data flows should become manageable and traceable. The data delivered to consumers should be reliable and represent a "single version of the truth."

Automation of data and analytics platform is a fuzzy concept and depends on the tool you choose. Some visualization tools provide some degree of automation. However, specialized vendors and products give more capabilities without having to build the automation framework yourself. 

I will use as example Xpert BI, which is a Microsoft complimentary data platform and data warehousing automation tool. 

Capabilities and benefits of using automation with Xpert BI:

Automated connectivity and managed connections to the data sources reduce the source systems' load and enforce naming standards. It minimizes the number of data loads, also improves the time to deliver data to consumers. Automated data processing and standardized business logic reduce time spent on development and enforce previously defined business rules and code.

Guide on assessing needs for creating or upgrading your data analytics platform 3

The tool provides data traceability on an object (table) and column (number or KPI) level from the data sources to the consumer level. It gives control over where these numbers are coming from and which business logic was in use. It enforces a "single version of the truth" for your data platform and all the reports and dashboards you have. As the sum of all the previous, the data quality goes up. The data platform becomes manageable even at large scales.

Signs that you want to boost your platform to the next stage: 

Now your data sources connectivity, information management, and data engineering are automated and efficient. However, you may notice that some tasks are still taking a lot of time. Usually, it is preparing data for user consumption by developing analytics, reports, dashboards, etc. The larger the scale of the organization, the more noticeable the issue is. 

Also, having a large organization with a lot of data fed to reports might cause "data digestion" issues. Your users might complain about their reports being very slow. 

The next step would be to think about analytics automation and optimization.

 

Stage 3: Analytics automation and optimization

At this stage, the point is to increase the efficiency of efforts spent on the development of analytics. In our example, it would be the automation of development in Microsoft Power BI. It is hard to automate the visualization process in Power BI. However, it is possible to optimize data modeling, creating measures, attributes, analytical functions, and formulas. It is done by moving data models and analytics (also called BI semantic models) out of Power BI reports and using Power BI reports for visualizations and sharing. Here both on-premises or Azure SQL Server Analysis Services or cloud-based Power BI Premium capacity datasets could be used as the destination. They use the same underlying technology (Analysis Services VertiPaq), so the choice between them would be based on your on-premises vs. cloud strategy and price calculation.

 In some cases, stage 2 and stage 3 could be taken in one leap. It is recommended if you have this experience before and want to get to the maximum efficiency level faster. 

Capabilities and benefits:

Usually, a few domain specific BI semantic models are created, e.g. HR, Finance, Operations etc. And then many Power BI reports and dashboards could reuse analytics from these models. It reduces the time spent on Power BI development and simplifies the lifecycle management of analytics and visualizations.

Several tools exist for semantic model development, including native tools from Microsoft. Using the Xpert BI tool, you could boost efficiency even more by automating the process of model creation. Xpert BI can generate semantic models in one click based on metadata from your data platform.

The poor performance for reports and dashboards which use lots of data will also be fixed. The latest version of SSAS Tabular models and Power BI Premium datasets use in-memory technology that stores all the numbers in quick memory, precalculated for immediate access by users.

Using a combination of separate model and Power BI visualization versus pure Power BI all-in-one provides more efficient access control management to your data on column, row, and table-level defined for specific users or groups.

Guide on assessing needs for creating or upgrading your data analytics platform 4

Signs that you want to boost your platform to the next stage: 

The effect of a successful implementation and growth of your data and analytics platform is an increasing business value to where it even becomes business-critical. Then the need for efficient and stable operationalization is crucial. When time-to-delivery of data and analytics to users becomes critical, you should start looking at the introduction of automated testing and DataOps principles.

 

Stage 4: Automated operationalization

At this stage, the point is to automate the daily operations and maintenance of the data and analytics platform. The goal is to ensure stability, data quality, performance, and quick detection of errors. Ideally, we want to spend as little time as possible on platform operations tasks having as high as possible user satisfaction.

Effort spent on operationalization is dependent on which tools and technologies you have chosen. If tools do not provide capabilities, they will have to be created manually.

Xpert BI tool provides out-of-the-box capabilities to operationalize your data platform.

Capabilities and benefits:

Improving Data Quality using automated tests: Automated testing is well known and used in software development. However, it struggles to settle in a data world (there are reasons for that, but it is a topic for another discussion). An emerging DataOps concept is created to fix that. There are tools to implement automated testing for data platforms. Few do it both well and time-efficient. 

DataOps module in Xpert BI has a library of pre-defined standard data quality checks, which could be set up in a few clicks, for the whole data platform. This module also has a framework that helps developers define their specific tests. It supports DataOps principles such as test-driven development and inclusion of automated testing in the solution's lifecycle.

Operational reporting and alerting: The bigger the data platform is, the more you will appreciate if all the operational KPIs and errors are collected in one place and presented conveniently. Items worth monitoring are data refresh frequency and scheduling, error reporting and alerting, data platform performance overview, automated tests run results. Some tools like Xpert BI come with the Operational dashboard out-of-the-box. However, it is possible to create it on your own in Power BI (development and connecting all the sources would take time).

Live documentation for the data platform and reports: The classical documentation issue that it becomes outdated immediately after the Save button is pressed in the document. In reality, the documentation is rarely reliable and usually very outdated. The issue would never be there if the documentation is dynamically built based on the data platform's metadata and always updated by the data platform itself. Here you need to have a tool that can access data platform metadata and make use of it. Most probably, the tool should be provided by the vendor of the data platform. Xpert BI comes with Solution Catalog module, which generates live documentation for solutions within the data platform. It also allows tracing data from the data sources level to the data consumption level (Power BI reports and dashboards).

Use data tagging to improve Information management: Some data elements could be tagged for various reasons (e.g., GDPR compliance). It would save time if the data platform could recognize the tagged data element's usage and propagate the tag from the bottom to the top-level. Xpert BI can do that. The ability to search data by tags and trace it through the data platform from the data source level to the consumption level is powerful.

 

Overview of data platform tools and their suitability:

Guide on assessing needs for creating or upgrading your data analytics platform 5

Good luck with your data and analytics platform journey. If you have any questions, do not hesitate to contact Alexander Khorunzhiy.

 

Alexander Khorunzhiy

Alexander Khorunzhiy

Alexander Khorunzhiy is Senior Solution Architect at BI Builders. He has experience with the development, architecture, testing of BI solutions and underlying IT infrastructure for BI platforms from Microsoft, Oracle and SAS Institute. He works in the Operational Excellence group which helps our clients to run daily operations of the enterprise data warehouses and analysis services, ensuring standards, data quality, performance and uptime. In his free time, Alexander design family board games and helps his wife with running their board game publishing company.

Follow our blog


Our dedicated employees write professional blogs worth reading.
Follow the blog for a sneak peek at the future!

Others would also read


How a data strategy can make you a better leader

How a data strategy can make you a better leader

Picture of Kurt S. Helland

by Kurt S. Helland

21. Apr 2021, 4 min reading time

How to empower your team through data

How to empower your team through data

Picture of Alf Inge Johansen

by Alf Inge Johansen

14. Apr 2021, 4 min reading time

BI Builders expands into the Benelux with strategic DataToko partnership

BI Builders expands into the Benelux with strategic DataToko partnership

Picture of BI Builders

by BI Builders

31. Mar 2021, 4 min reading time