Performance tuning for SQL queries and data warehousing is an art that data management folks have been struggling with for decades…
Database Administrators or DBA’s for short, have spent a considerable amount of time, and resources on performance optimizations to make user queries as fast as possible. While some modern data warehouse vendors have promised automatic query optimization, removing the need for your DBA to tune anything.
Honestly, this is what you paid the DBA for, to use their professional judgment on how to make the tables in the warehouse performant, and the best ones did this efficiently, so costs don't grow out of control as the data platform scales up with business growth.
be wary of any vendor that tells you you can get rid of them with their product.
The best DBAs are the ones that you don't even know their names”. This is because if the DBA is great at their job, everything runs fine and no one ever complains, so you don’t know who they are.
A great starting place for DBAs would be the Query History page. The Query History lists all queries that have been executed in that workspace. From there, you can sort, filter, and find troublesome queries in dire need of tuning. When a user complains about query performance, you can go to this page, filter for that user, and observe the queries they have been submitting
With Delta Lake, one of the major innovations is the JSON transaction log to deliver ACID transactions on a data lake. Along with ACID transactions, Delta Lake also collects statistics of the column data in the Parquet files the data is stored
azure
service
