Azure Databricks Compliance Security Profile

When planning for enabling Azure Databricks Compliance Security Profile (CSP) in Azure, here is a list of high-priority items that may impact the deployment.

Key dates to remember

September 1, 2026: CSP becomes required for workspaces that process data protected under HIPAA, HITRUST, and IRAP.
February 1, 2027: Azure Virtual Network encryption enforcement begins.

Azure Virtual Network Encryption (VNet)

If Azure Virtual Networks connected to Databricks Workspaces are not encrypted, then Classic Compute Clusters may fail to start. Consult with your Cloud or DevOps Team to identify the Azure Virtual Networks and send them this link https://learn.microsoft.com/en-us/azure/virtual-network/how-to-create-encryption to enable VNet encryption. If you’re using Terraform to manage an Azure Virtual Network, the AzureRM Terraform resource azurerm_virtual_network sets this.

Azure Classic Compute Virtual Machine Type

Classic Compute Clusters cannot start when Databricks jobs and pipelines use legacy Azure virtual machine SKUs. Microsoft publishes supported virtual machine instance types at https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview#requirements.

Databricks Preview Features

Databricks rolls out many new features that may stop working and be disabled after enabling CSP. Thankfully Microsoft and Databricks publish supported preview features. Make note of the type of features that are enabled by a specific CSP, as some features support only workspaces configured for HIPAA.

Automatic Cluster Update

Automatic Cluster Update is another reason to consider moving workloads off of Classic Compute onto Serverless Workloads. Make sure to take inventory of long-running jobs and confirm that your jobs are written to withstand compute restarts. Databricks provides an example notebook to help with identifying long-running compute and we at 536 Technologies can help with this too! With Automatic Cluster Update, this feature will be enabled by default on the first Sunday of every month at 1:00 AM UTC. You can confirm that this is set up by going to the Azure Databricks Account Console and confirming if this is configured.

TLS 1.2 or Newer Communications

Microsoft states the following about this change:

Communications within the cluster and for egress use TLS 1.2 or higher, including communication with the metastore.

Take this opportunity to review all workloads that use Classic Compute Clusters and connect to external sources to confirm they’re not only secure but if they’re nececessary and can be disabled.

Enhanced Security Monitoring

Enabling CSP turns on Enhanced Security Monitoring. This feature uses an enhanced hardened disk image based on Ubuntu Advantage. The image includes additional agents for file integrity, antivirus, malware detection, and vulnerability scanning.

Databricks Audit Logs

The agents generate audit log events for file integrity and antivirus monitoring. Enable the audit log system table to review these events. We strongly suggest forwarding your logs using Audit Log Delivery.

Genie One

If you depend on the Genie One Dashboard at the account level then you will no longer see CSP enabled workspaces and their Genie assets. You can still visit the Genie One dashboard but only within the CSP enabled Databricks Workspace.

Cost

Databricks applies the Enhanced Security and Compliance add-on rate to eligible usage in each CSP-enabled workspace. Your account’s rate determines the cost. Microsoft and Databricks provide guidance on how to use the system.billing.usage table to help customers identify and monitor costs. They even provide a query that identifies add-on costs.

Conclusion

We at 536 Technologies understand the complexity of enabling the Databricks Compliance Security Profile. Our Azure and Databricks platform experience helps customers assess the impact and prepare their workspaces for applicable regulatory requirements.