Cut Your Databricks Costs: The Ultimate Guide to Cost Control

00:00 - Enable System Tables- A robust source of data 02:17 - Tagging - The best way to organize costs 08:40 - AI/BI Cost Dashboard - Visualize and understand trends Enable System Tables- A robust source of data It is probably no surprise that the first step to understanding your usage is accessing data about it. Databricks provides usage data in the form of system tables, which are a Databricks-hosted analytical store of your account's operational data. In this post, we'll focus on using data in the billing system tables. Enable System tables: https://notebooks.databricks.com/demos/uc-04-system-tables/index.html# Tagging - The best way to organize costs As your company's usage of Databricks grows, you'll want to break the costs up by team, project or department to make it easier to tie costs to the value they produce. To do so, we recommend using tags, which allow you to create flexible key:value pairings. You can apply tags on clusters, SQL warehouses, jobs and serverless workloads. Tags are the best way to track what's happening in your Databricks environment. It's usually best to start with general keys that track usage to a business unit and a project, then move to more specific tags as needed to support optimizing your data and AI initiatives. Predefine the values for each key. Use policies to enforce usage of the tag key:value pair and control the tag values with the predefined list. Then you can use your dashboard to see usage by tag value and identify which workloads haven't been tagged yet. Cluster Policies: https://docs.databricks.com/aws/en/admin/clusters/policies Budget Policies https://docs.databricks.com/aws/en/admin/usage/budget-policies AI/BI Cost Dashboard - Visualize and understand trends To help you quickly understand your costs, we've included a pre-built AI/BI dashboard that allows you to view your usage over time and drill into your workspaces and SKUs. This can help you quickly identify areas that need your attention, such as a sudden spike in project costs. Additional Dashboards Job Cost Tracking: https://docs.databricks.com/aws/en/admin/system-tables/jobs-cost DBDemos https://www.databricks.com/resources/demos/tutorials/governance/system-tables