Multicloud Azure & AWS and Databricks

With Xpert BI version 4 we are excited to announce two major additions making it even more flexible and scalable: 
1) Xpert BI running on both Azure and AWS 
2) We now support loading data into Databricks 

Why?
Our opinion is that you should benefit from automation regardless of your choice of infrastructure. By giving you freedom of choice and the ability to change without rewriting code you can keep optimizing price performance in the cloud and leverage new, exciting technologies in the future when they fit your needs and budget.   

Multicloud Azure and AWS
Xpert BI can now fully operate on Amazon Web Services (AWS) using Amazon RDS for SQL Server. It means all Xpert BI data automation and data governance functionality is available in AWS. We believe the future is hybrid and multicloud, and now our customers can choose between Azure and AWS or have both depending on their needs.

We now support: 
•    Azure SQL db (full stack)
•    Azure Synapse DSP (full stack)
•    Azure Data Lake Storage v2 (ingest)
•    Snowflake (ingest)
•    Databricks (ingest) [NEW]
•    Amazon S3 (ingest)
•    Amazon RDS for SQL Server (full stack) [NEW]

Full stack means the complete Xpert BI automation and governance experience, whereas ingest means using automation to extract data from APIs, databases, and files, and load it into tables or files in the respective destination system. Complete with loading mechanisms, surrogate keys, hash keys, folder structures, flattening of complex hierarchies from APIs, etc. As a data engineer, you don’t have to think about if you are loading data into a distributed file system of a datalake such as ADLS gen2, a relational database such as Azure SQL db, or an MPP database such as Azure Synapse Dedicated SQL Pool. We are working on expanding our automation and governance capabilities for the technologies above which we do ingest today. 

Databricks
Secondly, we have added support for loading data into Databricks using their SQL interface. Databricks has gained a lot of momentum from its close integration and collaboration with Microsoft Azure and the focus on the data lakehouse.  The idea behind the data lakehouse as described in the Databricks paper is to have a single one-tier architecture with a data lake replace the typical two-tier architecture with a data lake for staging and a relational database for data warehouse on top. By adding SQL interfaces and ACID compliance through open-source Delta Lake format they can make a data lake work much like a relational database. It is a competitor to Azure Synapse and Snowflake, which offer three different approaches to how a data warehouse can be implemented. 

Key reasons for a data lakehouse according to Databricks: 
•    Big data (larger variety, volume, and velocity of data), and more AI workloads
•    Fixing the shortcomings of data lakes by adding e.g. support for transactions, consistency/isolation as well as enforcing data quality
•    Combining the best architectural concepts from both the data lake and data warehouse into a single system, reducing the need for multiple analytics systems down to potentially one

We believe: 
1) Big potential: Databricks is a promising technology with a lot a potential and rapid growth, especially for large-scale analytics and companies with big data ambitions and typically correspondingly big budgets. 
2) Tough competition: Databricks competes with Snowflake and Azure Synapse. The three technologies have different approaches, but all aim at achieving a complete and scalable data platform:  
a. Databricks: Making a data lake work like an MPP relational database.
b. Snowflake: Making an MPP relational database also have data lake capabilities.
c. Synapse: Bridging across data lake and MPP relational database with virtualization.
3) The relational database isn’t dead: For a data warehouse with normal-sized structured data, a relational database still has a set of strengths such as security, consistent performance, high availability of skilled workforce, and making self-service and automation easier as a database combines metadata and data (often separated in a data lake) 
4) Freedom of choice: Since there are several tradeoffs, our customers should have the freedom of choice. They should be able to try out any of Databricks, Snowflake, and Synapse, and choose the one best fitting their data and analytics needs. Also, they should be able to swap technologies later for continuous optimization of price–performance. 
5) Start small, scale up/out when you need: We believe automation should enable your organization to start small e.g. with a relational database, and at any time be able to upscale to Databricks without having to rewrite any code. This way, you save costs, minimize risks, and have full flexibility to get more performance when your business case and budget allow for it.

In summary, you now have the choice and flexibility to bring more automation and governance to both Azure and AWS, and to Databricks. 

The latest release of Xpert BI extends support for enterprise data platforms. To learn more about the release make sure to watch our release webinar

Sources:

• Read more about Databrick’s take on data lakehouse along with their key selling points
• The original Databricks paper
• Microsoft MVP, James Serra, defining a data lakehouse. In a call with James in January 2023 we discussed his blogs and how automation could bring value to data lakehouse and modern data warehouses. A key takeaway is that Xpert BI can allow itself to be specific towards architectures we want to build in a data lake or data warehouse, thus enabling standardization, guidance along best practices and automation of repeatable tasks.
• You may also read about the simplicity of Snowflake
• There are several places to look for inspiration, we typically recommend visiting Microsoft Azure Architecture Center, these 8 architectural patterns are worth looking at, some include Databricks:
  • Data warehousing and analytics
  • Modern data warehouse for small and medium business
  • Enterprise business intelligence
  • Analytics end-to-end with Azure Analytics
  • Big data analytics with Azure Data Explorer
  • Automated enterprise BI
  • Data warehousing in Microsoft Azure
  • Big data architectures