Data

Databricks managed through Infrastructure As Code

Converted an existing Databricks environment into infrastructure as code, allowing data engineers to experiment more quickly and request Databricks resources through pull requests.

The problem

Data engineers could not provision new medallion schemas in Unity Catalog through a clear, repeatable process. Cloud, IAM, and Security teams received tickets with missing context, leaving governance as an afterthought.

How we worked

  1. Assess. We worked with Data Engineering to trace how requests moved between teams. The bottleneck became clear: requesting a new schema, workspace, or service principal required several tickets, with context and approvals split across Data Engineering, Cloud, IAM, and Security.

  2. Codify. We imported the existing Databricks workspace configuration and Unity Catalog resources into Terraform, creating a version-controlled representation of the live environment and a repeatable way to propose changes.

  3. Operationalize. We defined the governance boundaries for who could request, review, and approve each type of change. Pull requests gave Cloud and Security teams one place to validate changes against the permissions model the business intended to follow.

  4. Enable. Data Engineering received a runbook and an example showing how to make a change. Cloud Engineering received documentation explaining how infrastructure changes are tested, promoted across environments, applied, and recovered when necessary.

The outcome

The Unity Catalog changes in scope now move through pull-request review instead of separate ticket queues. Human access is standardized around team-owned groups, with service principals used for automation. Both are managed through version control.