🧱 Databricks Cluster Terraform Module

Provisions a single Databricks compute cluster and its attached library dependencies as one unit, against the databricks/databricks provider ~> 1.117.0.

• 🖥️ Creates one databricks_cluster — the keystone compute resource for interactive and automated workloads.
• 📦 Attaches zero or more libraries via separate databricks_library resources, keyed by a stable caller-chosen identifier, for_each-driven — never count, never the cluster's inline library block.
• 🔐 Defaults to USER_ISOLATION access mode and a 30-minute auto-termination window — a caller must type extra characters to loosen either.
• 🚫 Never accepts a credential, host, or account ID.
• 🌍 Workspace-plane only — this resource cannot be used with an account-level provider.

💡 Why it matters: a cluster is never created without its required libraries already declared, so library drift is caught at plan time rather than discovered at job-run time. Cost and access-mode defaults are chosen for a regulated, multi-tenant workspace, not the API's own looser defaults.

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

• ⭐ Star this repository to help others discover this Terraform module.
• 🤝 Connect with me on LinkedIn: linkedin.com/in/microsoftexpert
• ☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!

flowchart LR
 POLICY["terraform-databricks-cluster-policy"]
 style POLICY fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px

 POOL["terraform-databricks-instance-pool"]
 style POOL fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px

 THIS["terraform-databricks-cluster"]
 style THIS fill:#FF3621,color:#fff,stroke:#1B3139,stroke-width:1px

 JOB["terraform-databricks-job"]
 style JOB fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px

 PERMS["terraform-databricks-permissions"]
 style PERMS fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px

 POLICY -->|"id becomes policy_id (optional)"| THIS
 POOL -->|"id becomes instance_pool_id (optional)"| THIS
 THIS -->|"id becomes task cluster_id (planned)"| JOB
 THIS -->|"id becomes object_id (planned)"| PERMS

terraform-databricks-cluster-policy and terraform-databricks-instance-pool are optional upstream siblings, consumed only by ID (policy_id, instance_pool_id) — this module never creates either. terraform-databricks-job and terraform-databricks-permissions are downstream consumers of this module's id. Validated via the Mermaid Chart MCP before being embedded here.

ℹ️ terraform-databricks-cluster-policy and terraform-databricks-instance-pool were being authored in parallel with this module in the same session and did not yet have outputs.tf files when this README was written — the edges above and the end-to-end composition example (§ Example Library, item 14) reflect their planned id output contract, not a verified cross-module terraform plan.

flowchart TB
 subgraph INPUTS["var.*"]
 CORE["cluster_name / spark_version / node_type_id"]
 SIZE["autoscale / num_workers / autotermination_minutes"]
 REFS["policy_id / instance_pool_id (optional, by id)"]
 ESCAPE["spark_conf / custom_tags"]
 LIBS["libraries (keyed map)"]
 end

 KEYSTONE["databricks_cluster.this"]
 style KEYSTONE fill:#1B3139,color:#fff,stroke:#1B3139,stroke-width:1px

 CHILD["databricks_library.this[key] (separate resource type, for_each)"]
 style CHILD fill:#FF3621,color:#fff,stroke:#1B3139,stroke-width:1px

 subgraph OUTPUTS["outputs"]
 ID["id (cluster id)"]
 LIBIDS["library_ids (map of key to library id)"]
 end

 CORE --> KEYSTONE
 SIZE --> KEYSTONE
 REFS --> KEYSTONE
 ESCAPE --> KEYSTONE
 LIBS --> CHILD

 KEYSTONE -->|"cluster_id"| CHILD
 KEYSTONE --> ID
 CHILD --> LIBIDS

Resource inventory: two resource types — databricks_cluster.this (single, keystone) and databricks_library.this (for_each over var.libraries, keyed by library identifier). The child is a genuinely separate resource type in this pinned provider version, not a nested block — see "Schema notes that bite" below.

Schema notes that bite:

• databricks_cluster exposes two ways to attach libraries: an inline library nested block (nesting_mode: set) directly on the cluster, and the fully separate databricks_library resource keyed by its own required cluster_id. This module uses only the separate resource, via for_each — the inline block cannot be keyed by a stable identifier the way a separate resource can, and mixing both would create two divergent library-attachment paths for the same cluster. See SCOPE.md's "Design decisions."
• jar, whl, and requirements are plain string attributes on databricks_library in this pinned provider version, confirmed against schema.json — not nested blocks. pypi, maven, and cran remain nested blocks and are rendered via dynamic blocks in main.tf.
• databricks_library requires exactly one library type per entry (provider docs: "It's possible to set only one type of library within one resource. Otherwise, the plan will fail with an error."). This module enforces that with a validation {} block on var.libraries rather than letting a malformed entry fail only at plan time against a live workspace.
• terraform providers schema -json does not surface ForceNew/"requires replace" metadata — that information lives only in the provider's own documentation, not the schema dump. Any force-new claim in this README or Architecture Notes about node_type_id, spark_version, or similar cluster arguments is sourced from the live provider docs consulted at authoring time, not from schema.json, and should be re-verified at any future revision rather than assumed stale-safe from this file.
• node_type_id is Required unless instance_pool_id is set (or a referenced policy_id presets it) — confirmed in the live provider docs' argument reference. This module types it as optional and does not hard-validate the either/or relationship, for the same policy-driven- defaults reasoning as the autoscale/num_workers precedence rule (see Architecture Notes).
• data_security_mode enum discrepancy: this library's Secure-by-default convention previously named the legacy opt-out value "LEGACY_SINGLE_USER_STANDARD". That string does not exist in the live databricks/databricks provider schema or docs for ~> 1.117.0 — the real legacy values are LEGACY_PASSTHROUGH and LEGACY_TABLE_ACL (plus NONE/NO_ISOLATION). This module's validation {} block uses the confirmed live values. Flagged as a correction candidate for that convention, not silently patched over.
• databricks_library import is not currently supported, per the live provider docs — see Troubleshooting.
• autotermination_minutes accepts 0 (disabled) or an integer between 10 and 10000 inclusive — validated via validation {} since Terraform's type system can't express a bounded numeric range natively.

• Workspace-level: cluster creation permission (CAN_MANAGE at the workspace level, or the unrestricted cluster-creation entitlement, depending on the workspace's cluster-creation policy configuration — sourced from Databricks' access-control documentation for clusters, not the raw provider schema, which does not encode permission requirements).
• If policy_id is set: CAN_USE on the referenced cluster policy.
• Attaching a databricks_library requires the same cluster-management permission as the parent cluster; the provider does not expose a separate library-level permission.

• Workspace-level provider context (not account-level) — databricks_cluster and databricks_library are both workspace-plane resources per the live schema.
• If instance_pool_id is set, the referenced pool must already exist in the same workspace.
• If policy_id is set, the referenced cluster policy must already exist in the same workspace.

terraform-databricks-cluster/
├── providers.tf # required_providers only — no provider {} block
├── variables.tf # cluster_name, spark_version (required); autoscale, num_workers,
│ # data_security_mode, autotermination_minutes, policy_id,
│ # instance_pool_id, spark_conf, custom_tags, libraries (optional)
├── main.tf # databricks_cluster.this + databricks_library.this (for_each)
├── outputs.tf # id first, then library_ids
├── SCOPE.md # cross-module contract
├── README.md # this file
└── examples/
 └── basic/
 └── main.tf # smallest real, runnable call

module "shared_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "shared-autoscaling"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"

  autoscale = {
    min_workers = 1
    max_workers = 8
  }
}

The caller's root module configures provider "databricks" {} (host + auth) and passes it in implicitly, or explicitly via a providers = {} map for a multi-workspace root — this module accepts neither a provider block nor any credential-shaped variable.

Consumes:

Emits:

module "baseline_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "baseline"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
}

module "autoscaling_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "shared-autoscaling"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"

  autoscale = {
    min_workers = 1
    max_workers = 8
  }
}

module "fixed_size_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "fixed-size"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS4_v2"
  num_workers   = 4
}

module "single_user_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name       = "ml-single-user"
  spark_version      = "15.4.x-scala2.12"
  node_type_id       = "Standard_DS3_v2"
  data_security_mode = "SINGLE_USER"
  num_workers        = 2
}

module "legacy_table_acl_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name       = "legacy-shared"
  spark_version      = "13.3.x-scala2.12"
  node_type_id       = "Standard_DS3_v2"
  data_security_mode = "LEGACY_TABLE_ACL"
  num_workers        = 2
}

module "policy_driven_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "policy-scoped"
  spark_version = "15.4.x-scala2.12"
  policy_id     = "600123456789abcd"
}

module "pooled_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name     = "pooled"
  spark_version    = "15.4.x-scala2.12"
  instance_pool_id = "1234-567890-abcd123"

  autoscale = {
    min_workers = 2
    max_workers = 6
  }
}

module "tuned_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "tuned"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
  num_workers   = 4

  spark_conf = {
    "spark.databricks.io.cache.enabled"      = "true"
    "spark.databricks.io.cache.maxDiskUsage" = "50g"
  }

  custom_tags = {
    cost_center = "analytics-platform"
    environment = "prod"
  }
}

module "always_on_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name            = "always-on-shared"
  spark_version           = "15.4.x-scala2.12"
  node_type_id            = "Standard_DS3_v2"
  num_workers             = 2
  autotermination_minutes = 0
}

module "cli_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "cli-enabled"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
  num_workers   = 2

  libraries = {
    databricks-cli = {
      pypi = {
        package = "databricks-cli"
      }
    }
  }
}

module "multi_library_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "data-quality"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
  num_workers   = 4

  libraries = {
    fbprophet = {
      pypi = {
        package = "fbprophet==0.6"
      }
    }
    deequ = {
      maven = {
        coordinates = "com.amazon.deequ:deequ:1.0.4"
        exclusions  = ["org.apache.avro:avro"]
      }
    }
    internal-app = {
      jar = "/Volumes/analytics/raw/libraries/app-0.0.1.jar"
    }
  }
}

module "r_and_python_reqs_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "r-and-python"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
  num_workers   = 2

  libraries = {
    rkeops = {
      cran = {
        package = "rkeops"
      }
    }
    pinned-requirements = {
      requirements = "/Workspace/Shared/pipelines/requirements.txt"
    }
  }
}

module "no_library_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name  = "notebook-only"
  spark_version = "15.4.x-scala2.12"
  node_type_id  = "Standard_DS3_v2"
  num_workers   = 2
  # libraries defaults to {} — no databricks_library resources are created.
}

module "team_policy" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster-policy.git?ref=v1.0.0"

  name = "analytics-team-policy"
  #... policy definition fields omitted for brevity; not part of this module.
}

module "shared_pool" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-instance-pool.git?ref=v1.0.0"

  name         = "analytics-shared-pool"
  node_type_id = "Standard_DS3_v2"
  #... remaining pool sizing fields omitted for brevity; not part of this module.
}

module "analytics_cluster" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-cluster.git?ref=v1.0.0"

  cluster_name     = "analytics-shared"
  spark_version    = "15.4.x-scala2.12"
  policy_id        = module.team_policy.id
  instance_pool_id = module.shared_pool.id

  autoscale = {
    min_workers = 2
    max_workers = 10
  }

  libraries = {
    databricks-cli = {
      pypi = {
        package = "databricks-cli"
      }
    }
  }
}

# Planned — terraform-databricks-job is not yet authored in this catalog batch.
# module "nightly_etl" {
# source = "git::https://github.com/microsoftexpert/terraform-databricks-job.git?ref=v1.0.0"
#
# tasks = {
# etl = {
# existing_cluster_id = module.analytics_cluster.id
# notebook_task = {
# notebook_path = "/Shared/pipelines/nightly_etl"
# }
# }
# }
# }

variable "cluster_name" {
  type = string
  # validation: must not be empty
}

variable "spark_version" {
  type = string
  # validation: must not be empty
}

variable "node_type_id" {
  type    = string
  default = null
}

variable "instance_pool_id" {
  type    = string
  default = null
}

variable "policy_id" {
  type    = string
  default = null
}

variable "data_security_mode" {
  type    = string
  default = "USER_ISOLATION"
  # validation: one of USER_ISOLATION, SINGLE_USER, LEGACY_PASSTHROUGH, LEGACY_TABLE_ACL, NONE, NO_ISOLATION
}

variable "autotermination_minutes" {
  type    = number
  default = 30
  # validation: 0, or between 10 and 10000 inclusive
}

variable "autoscale" {
  type = object({
    min_workers = number
    max_workers = number
  })
  default = null
}

variable "num_workers" {
  type    = number
  default = null
}

variable "spark_conf" {
  type    = map(string)
  default = {}
}

variable "custom_tags" {
  type    = map(string)
  default = {}
}

variable "libraries" {
  type = map(object({
    jar          = optional(string)
    whl          = optional(string)
    requirements = optional(string)
    pypi = optional(object({
      package = string
      repo    = optional(string)
    }))
    maven = optional(object({
      coordinates = string
      repo        = optional(string)
      exclusions  = optional(list(string))
    }))
    cran = optional(object({
      package = string
      repo    = optional(string)
    }))
  }))
  default = {}
  # validation: each entry must set exactly one of jar, whl, requirements, pypi, maven, cran
}

• autoscale/num_workers precedence, not mutual-exclusivity validation. The Databricks API treats these as mutually exclusive, but this module does not enforce that with a hard validation {} block — a policy_id-scoped cluster can legitimately leave both null, relying entirely on the referenced cluster policy for sizing. main.tf instead implements a precedence rule: when autoscale is set, num_workers is always rendered null, regardless of what the caller passed to num_workers.
• for_each key stability for libraries. databricks_library.this is keyed by the caller's chosen string in var.libraries, not a list index — removing one library from the middle of the map does not re-index or force replacement of any other library, unlike a count-based or inline-block-driven approach would risk.
• Separate resource over inline block, deliberately. databricks_cluster exposes an inline library block, but this module attaches libraries exclusively through the standalone databricks_library resource so each attachment has its own stable identity and can be added, removed, or changed independently without touching the keystone cluster resource's plan.
• node_type_id optionality is real, not an oversight. It is typed optional(string) because it is legitimately absent when instance_pool_id or a referenced policy_id supplies the node type — see "Schema notes that bite."
• No sensitive outputs. Neither id nor library_ids carries a credential or secret; no output in this module is marked sensitive = true.

This table matches what main.tf/variables.tf actually implement — no row here that the code doesn't enforce, and no enforced default missing from this table.

cd terraform-databricks-cluster
terraform init -backend=false
terraform validate
terraform fmt -check

Pin consumers to an immutable tag — ?ref=v1.0.0 — never a branch. This module is plan-only; a human applies from CI after review.

terraform validate / terraform fmt -check catch: missing required arguments (cluster_name, spark_version), the data_security_mode enum validation, the autotermination_minutes range validation, and the libraries-exactly-one-type validation. They do not catch: whether the applying identity actually holds cluster-creation permission or CAN_USE on a referenced policy, whether a referenced instance_pool_id/policy_id actually exists in the target workspace, whether max_workers is genuinely greater than min_workers in autoscale (documented but not type-system-enforceable), or any real Databricks API-side constraint (quota, node-type availability, region). Those require an actual plan/apply against a live workspace, out of scope for this authoring process.

$ terraform output
id = "1234-567890-abcd1234"
library_ids = {
 "databricks-cli" = "1234-567890-abcd1234/databricks-cli"
}

• databricks_cluster provider resource
• databricks_library provider resource
• terraform-databricks-cluster-policy (optional upstream, provides policy_id)
• terraform-databricks-instance-pool (optional upstream, provides instance_pool_id)
• terraform-databricks-job (planned downstream consumer of id)
• terraform-databricks-permissions (planned downstream consumer of id)
• This module's SCOPE.md

💙 "Infrastructure as Code should be standardized, consistent, and secure."