Secrets Using Azure Key Vault

Azure Key Vault secrets facilitate the secure management of confidential credentials for authentication, ensuring they remain undisclosed within the notebook. This method preserves credential security while enabling monitoring of their usage and access across diverse services. In addition to managing secrets, Key Vault also supports key management and certificate management.

---

Azure Key Vault Secrets In Databricks

Creating A Secret Scope

To create a secret scope in Databricks, users can access the portal at https://<databricks-instance>#secrets/createScope, leveraging properties available from the Azure Key Vault portal.

Alternatively, users can create a Databricks-backed secret scope using the Databricks CLI or the Secrets API.

Access should be allowed to the Databricks workspace from the Networking tab within Azure Key Vault settings.

Accessing Secrets Within Notebooks

To access the secrets within notebooks, users need to utilize appropriate utility functions from the dbutils.secrets library.

Python

# Gets Secret
secretVar = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")


# Lists All Secrets
secretList = dbutils.secrets.list("<scope-name>")

Scala

// Gets Secret
val secretVar = dbutils.secrets.get(scope="<scope-name>", key="<key-name>")


// Lists All Secrets
val secretList = dbutils.secrets.list("<scope-name>")

---

Azure Key Vault Secrets In Synapse

Creating A Linked Service

In Synapse, establishing a connection to an Azure Key Vault necessitates creating a linked service to authenticate the workspace with the vault.

Access should be allowed to the Synapse workspace from the Networking tab within Azure Key Vault settings.

Accessing Secrets Within Notebooks

To access the secrets within notebooks, users need to utilize appropriate utility functions from the mssparkutils.credentials library.

Python

secretVar = mssparkutils.credentials.getSecret("<key-vault-name>", "<key-name>", "<linked-service-name>")

Scala

val secretVar = mssparkutils.credentials.getSecret("<key-vault-name>", "<key-name>", "<linked-service-name>")

Alternatively, users can access secrets directly by changing the Spark configuration within the notebook.

---

Key Differences

Databricks-Backed Secret Scopes

In Databricks, users have the capability to create and delete customized secrets within their Databricks-backed secret scopes. These actions are exclusively performed through the Databricks CLI or Secrets API but can be accessed from within notebooks.

In contrast, Synapse does not include the capability to create or delete custom secrets directly within the workspace. It exclusively accesses secrets and keys directly from Azure Key Vault.

---

References

• Secret scopes (Databricks)

• What is the Databricks CLI?

• Azure Databricks Secret API reference

• Secret workflow example (Databricks)

• Linked services in Azure Data Factory and Azure Synapse Analytics

• Secure credentials with linked services using the mssparkutils

• Secrets (Databricks-backed secrets)

---