<img alt="" src="https://secure.dump4barn.com/213374.png" style="display:none;">

Innovation Norway project: Data warehouse automation in Azure

Bilde av Sverre Helland
by Sverre Helland

12. Feb 2021, 4 minutes reading time

Innovation Norway project: Data warehouse automation in Azure

In November 2020 BI Builders were very happy to announce that we had received a grant from Innovation Norway. The project is now well on its way, milestones have been reached, software (Xpert BI) has been released, and customers are using it.

We have become a certified Azure Synapse Analytics data integration partner and an Azure Synapse Analytics data management partner. 

azure synapse analytics

For this first release our focus has been on Azure Synapse Analytics. 

It is now possible to use Xpert BI for automating the building of your data warehouse (DWH) in Synapse Analytics. As always, we are leveraging the use of metadata to be able to do all the automation.

There are a lot of best practices which are automated, such as the building of folder structures, handling of data load optimization, Synapse Analytics design decisions and so on. It is easy to change these default best practices if for any reason the specific solution has other needs. It makes it easy to adapt your company's best practices and enforce them across the whole development team.

With a few clicks data is ingested into Synapse Analytics. It is possible to load the data directly or via Azure Data Lake Storage (ADLS). The final data destination can also be set to ADLS instead of Synapse Analytics.

We have built in full flexibility so that the DWH may be built on different technologies. An example could be using Azure dB as DWH for low data volumes from a CRM system, while IoT data is loaded into ADLS, and parts of the IoT data is further loaded into Synapse Analytics.

20190705_1127LR_SVJWith a click of a button the underlying technology can be switched. A solution can start out running on Azure dB, switch to Azure SQL Server Managed Instance when it grows, and then to Azure Synapse Analytics when the need arises. This is a concept we refer to as agile growth paths. It future proofs the architecture and the flexibility mitigates some of the fear of making the wrong architectural decision at the start of a project.

Some of the many Synapse Analytics automation features supported are:

  • DataOps for making sure the data and metadata quality of the solution is always as good as possible
  • Data load optimization, based on intelligently derived dependencies from source metadata
  • Integration with Microsoft Master Data Services (MDS)
  • Easy migration between DEV / QA / PROD environments
  • Export data to files / databases / ADLS / REST APIs (push data)
  • Make data automatically available via REST APIs as a service (3rd party will pull data)
  • Autogenerate SQL Server Analysis Services Tabular model based on schema
  • Incremental load
  • Source filtering
  • Notification and single click update of destination-schema when source-schema changes
  • Materialization of views (with incremental load possibilities)

The next stage in our development is to automate the process of generating a DWH built in Azure Data Factory. This is expected to significantly accelerate the process and enable each developer to deliver more business value, faster. Through standardization and best practices the DWH solution will be more robust leading to less resources needed for maintenance as well as minimizing personal dependencies.

Stay tuned.

 

Sverre Helland

Sverre Helland

Sverre Helland is our Head of R&D and one of the founders of BI Builders. Sverre has an MSc in Computer Science and has 15 years of software development experience.

Follow our blog


Our dedicated employees write professional blogs worth reading.
Follow the blog for a sneak peek at the future!

Others would also read


Why a data platform adds value to your ERP project

Why a data platform adds value to your ERP project

Picture of Alf Inge Johansen

by Alf Inge Johansen

15. Nov 2021, 5 min reading time

BI Builders enters partnership with Elevondata Labs.

BI Builders enters partnership with Elevondata Labs.

Picture of BI Builders

by BI Builders

11. Nov 2021, 3 min reading time

Modernizing your analytics architecture part 1: Key objectives

Modernizing your analytics architecture part 1: Key objectives

Picture of Terje Vatle

by Terje Vatle

12. Oct 2021, 8 min reading time