<img alt="" src="https://secure.dump4barn.com/213374.png" style="display:none;">

Multicloud Azure & AWS and Databricks

Bilde av Terje Vatle
by Terje Vatle

17. Mar 2023, 9 minutes reading time

Multicloud Azure & AWS and Databricks

With Xpert BI version 4 we are excited to announce two major additions making it even more flexible and scalable: 
1) Xpert BI running on both Azure and AWS 
2) We now support loading data into Databricks 

Our opinion is that you should benefit from automation regardless of your choice of infrastructure. By giving you freedom of choice and the ability to change without rewriting code you can keep optimizing price performance in the cloud and leverage new, exciting technologies in the future when they fit your needs and budget.   

Multicloud Azure and AWS
Xpert BI can now fully operate on Amazon Web Services (AWS) using Amazon RDS for SQL Server. It means all Xpert BI data automation and data governance functionality is available in AWS. We believe the future is hybrid and multicloud, and now our customers can choose between Azure and AWS or have both depending on their needs.

We now support: 
•    Azure SQL db (full stack)
•    Azure Synapse DSP (full stack)
•    Azure Data Lake Storage v2 (ingest)
•    Snowflake (ingest)
•    Databricks (ingest) [NEW]
•    Amazon S3 (ingest)
•    Amazon RDS for SQL Server (full stack) [NEW]

Full stack means the complete Xpert BI automation and governance experience, whereas ingest means using automation to extract data from APIs, databases, and files, and load it into tables or files in the respective destination system. Complete with loading mechanisms, surrogate keys, hash keys, folder structures, flattening of complex hierarchies from APIs, etc. As a data engineer, you don’t have to think about if you are loading data into a distributed file system of a datalake such as ADLS gen2, a relational database such as Azure SQL db, or an MPP database such as Azure Synapse Dedicated SQL Pool. We are working on expanding our automation and governance capabilities for the technologies above which we do ingest today. 

Secondly, we have added support for loading data into Databricks using their SQL interface. Databricks has gained a lot of momentum from its close integration and collaboration with Microsoft Azure and the focus on the data lakehouse.  The idea behind the data lakehouse as described in the Databricks paper is to have a single one-tier architecture with a data lake replace the typical two-tier architecture with a data lake for staging and a relational database for data warehouse on top. By adding SQL interfaces and ACID compliance through open-source Delta Lake format they can make a data lake work much like a relational database. It is a competitor to Azure Synapse and Snowflake, which offer three different approaches to how a data warehouse can be implemented. 

Key reasons for a data lakehouse according to Databricks: 
•    Big data (larger variety, volume, and velocity of data), and more AI workloads
•    Fixing the shortcomings of data lakes by adding e.g. support for transactions, consistency/isolation as well as enforcing data quality
•    Combining the best architectural concepts from both the data lake and data warehouse into a single system, reducing the need for multiple analytics systems down to potentially one

We believe: 
1) Big potential: Databricks is a promising technology with a lot a potential and rapid growth, especially for large-scale analytics and companies with big data ambitions and typically correspondingly big budgets. 
2) Tough competition: Databricks competes with Snowflake and Azure Synapse. The three technologies have different approaches, but all aim at achieving a complete and scalable data platform:  
a. Databricks: Making a data lake work like an MPP relational database.
b. Snowflake: Making an MPP relational database also have data lake capabilities.
c. Synapse: Bridging across data lake and MPP relational database with virtualization.
3) The relational database isn’t dead: For a data warehouse with normal-sized structured data, a relational database still has a set of strengths such as security, consistent performance, high availability of skilled workforce, and making self-service and automation easier as a database combines metadata and data (often separated in a data lake) 
4) Freedom of choice: Since there are several tradeoffs, our customers should have the freedom of choice. They should be able to try out any of Databricks, Snowflake, and Synapse, and choose the one best fitting their data and analytics needs. Also, they should be able to swap technologies later for continuous optimization of price–performance. 
5) Start small, scale up/out when you need: We believe automation should enable your organization to start small e.g. with a relational database, and at any time be able to upscale to Databricks without having to rewrite any code. This way, you save costs, minimize risks, and have full flexibility to get more performance when your business case and budget allow for it.

In summary, you now have the choice and flexibility to bring more automation and governance to both Azure and AWS, and to Databricks. 

The latest release of Xpert BI extends support for enterprise data platforms. To learn more about the release make sure to watch our release webinar

Watch our release webinar


Terje Vatle

Terje Vatle

Terje Vatle is Chief Technology Officer at BI Builders following global market trends within data & analytics. Terje has a technology and advisory background, and focuses on how to make organizations achieve their goals by becoming more data driven.

Follow our blog

Our dedicated employees write professional blogs worth reading.
Follow the blog for a sneak peek at the future!

Others would also read

Multicloud Azure & AWS and Databricks

Multicloud Azure & AWS and Databricks

Picture of Terje Vatle

by Terje Vatle

17. Mar 2023, 9 min reading time

DataOps: Source control and data quality

DataOps: Source control and data quality

Picture of Anja Loug Helland

by Anja Loug Helland

24. Feb 2023, 10 min reading time

DNO Norge selects Xpert BI to accelerate digital transformation.

DNO Norge selects Xpert BI to accelerate digital transformation.

Picture of BI Builders

by BI Builders

17. Feb 2023, 3 min reading time