» Versioned Key/Value Secret Engine

The Static Secrets guide introduced the basics of working with key-value secret engine. Vault 0.10 introduced K/V Secrets Engine v2 with Secret Versioning. This guide demonstrates the new features introduced by the key-value secret engine v2.

» Reference Material

» Estimated Time to Complete

10 minutes

» Challenge

The KV secret engine v1 does not provide a way to version or roll back secrets. This made it difficult to recover from unintentional data loss or overwrite when more than one user is writing at the same path.

» Solution

Run the version 2 of KV secret engine which can retain a configurable number of secret versions. This enables older versions' data to be retrievable in case of unwanted deletion or updates of the data. In addition, its Check-and-Set operations can be used to protect the data from being overwritten unintentionally.

Versioned KV

» Prerequisites

To perform the tasks described in this guide, you need to have a Vault environment. Refer to the Getting Started guide to install Vault. Make sure that your Vault server has been initialized and unsealed.

» Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# To view in Web UI
path "sys/mounts" {
  capabilities = [ "read", "update" ]
}

# Write and manage secrets in key-value secret engine
path "secret*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable secret engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

If you are not familiar with policies, complete the policies guide.

» Steps

This guide demonstrates the basic commands for working with KV secret engine v2.

You will perform the following:

  1. Check the KV secret engine version
  2. Write secrets
  3. Retrieve a specific version of secret
  4. Specify the number of versions to keep
  5. Delete versions of secret
  6. Permanently delete data

» Step 1: Check the KV secret engine version

(Persona: devops)

Before beginning, verify that you are using the v2 of the KV secret engine.

» CLI command

To check the KV secret engine version:

$ vault secrets list -format=json
...
"secret/": {
  "type": "kv",
  "description": "key/value secret storage",
  "accessor": "kv_f05b8b9c",
  "config": {
    "default_lease_ttl": 0,
    "max_lease_ttl": 0,
    "force_no_cache": false
  },
  "options": {
    "version": "2"
  },
  ...

The indicated version should be 2. If the version is 1, upgrade it to v2.

$ vault kv enable-versioning secret/

» API call using cURL

To check the KV secret engine version:

$ curl --header "X-Vault-Token: <TOKEN>" \
       <VAULT_ADDRESS>/v1/sys/mounts

Where <TOKEN> is your valid token, and <VAULT_ADDRESS> is where your vault server is running.

Example:

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/sys/mounts | jq
...
  "secret/": {
    "accessor": "kv_f05b8b9c",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "plugin_name": ""
    },
    "description": "key/value secret storage",
    "local": false,
    "options": {
      "version": "2"
    },
    "seal_wrap": false,
    "type": "kv"
  },
...

The indicated version should be 2. If the version is 1, upgrade it to v2.

$ cat payload.json
{
  "options": {
      "version": "2"
  }
}

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/sys/mounts/secret/tune

» Web UI

Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.

Web UI

If secret/ does not indicates v2, you can upgrade it from v1 to v2 by executing the following CLI command:

$ vault kv enable-versioning secret/

Alternatively, you can enable KV secret engine v2 at a different path by clicking Enable new engine. Select KV from the Secret engine type drop-down list. Be sure that the Version is set to be Version 2.

Enabling kv-v2

Click Enable Engine to complete.

» Step 2: Write Secrets

To understand how the versioning works, let's write some test data.

» CLI commands

To write secrets, run vault kv put command instead of vault write:

$ vault kv put secret/customer/acme name="ACME Inc." contact_email="jsmith@acme.com"
Key              Value
---              -----
created_time     2018-04-14T00:05:47.115378933Z
deletion_time    n/a
destroyed        false
version          1

To update the existing secret, run the vault kv put command again:

$ vault kv put secret/customer/acme name="ACME Inc." contact_email="john.smith@acme.com"
Key              Value
---              -----
created_time     2018-04-14T00:13:35.296018431Z
deletion_time    n/a
destroyed        false
version          2

Now you have two versions of the secret/customer/acme data. Run vault kv get to read the data.

$ vault kv get secret/customer/acme
====== Metadata ======
Key              Value
---              -----
created_time     2018-04-14T00:13:35.296018431Z
deletion_time    n/a
destroyed        false
version          2

======== Data ========
Key              Value
---              -----
contact_email    john.smith@acme.com
name             ACME Inc.

» API call using cURL

Write some data at secret/customer/acme:

$ tee payload.json <<EOF
{
  "data": {
    "name": "ACME Inc.",
    "contact_email": "jsmith@acme.com"
  }
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/secret/data/customer/acme

Notice that the endpoint for KV v2 is /secret/data/<path>; therefore, to write secrets at secret/customer/acme, the API endpoint becomes /secret/data/customer/acme.

Update the secret to create another version:

$ tee payload.json <<EOF
{
  "data": {
    "name": "ACME Inc.",
    "contact_email": "john.smith@acme.com"
  }
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/secret/data/customer/acme

Now you have two versions of the secret/customer/acme data. Read back the secret.

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/data/customer/acme
{
   "request_id": "7233b69d-35d9-6c1b-ae81-9a679a03082d",
   "lease_id": "",
   "renewable": false,
   "lease_duration": 0,
   "data": {
     "data": {
       "contact_email": "john.smith@acme.com",
       "name": "ACME Inc."
     },
     "metadata": {
       "created_time": "2018-04-14T00:59:11.27903511Z",
       "deletion_time": "",
       "destroyed": false,
       "version": 2
     }
   },
   "wrap_info": null,
   "warnings": null,
   "auth": null
}

» Web UI

In the Web UI, select secret/ and then click Create secret.

Write Secret

Click Save.

To update the existing secret, select Edit, change the contact_email value, and then click Save.

Write Secret

» Step 3: Retrieve a Specific Version of Secret

You may run into a situation where you need to view the secret before an update.

» CLI commands

To retrieve the version 1 of the secret written at secret/customer/acme:

$ vault kv get -version=1 secret/customer/acme
====== Metadata ======
Key              Value
---              -----
created_time     2018-04-14T00:05:47.115378933Z
deletion_time    n/a
destroyed        false
version          1

======== Data ========
Key              Value
---              -----
contact_email    jsmith@acme.com
name             ACME Inc.

To read the metadata of secret/customer/acme:

$ vault kv metadata get secret/customer/acme
======= Metadata =======
Key                Value
---                -----
created_time       2018-04-14T00:05:47.115378933Z
current_version    2
max_versions       0
oldest_version     0
updated_time       2018-04-14T00:13:35.296018431Z

====== Version 1 ======
Key              Value
---              -----
created_time     2018-04-14T00:05:47.115378933Z
deletion_time    n/a
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2018-04-14T00:13:35.296018431Z
deletion_time    n/a
destroyed        false

» API call using cURL

To retrieve the version 1 of the secret written at secret/customer/acme:

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/data/customer/acme?version=1 | jq
{
 "request_id": "3bf5a2c1-d89b-9dd5-9bb5-0bc61a4a6d83",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "data": {
     "contact_email": "jsmith@acme.com",
     "name": "ACME Inc."
   },
   "metadata": {
     "created_time": "2018-04-14T00:05:47.115378933Z",
     "deletion_time": "",
     "destroyed": false,
     "version": 1
   }
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

To read the metadata of secret/customer/acme:

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/metadata/customer/acme | jq
{
 "request_id": "34708262-59cd-9a94-247f-3b1db0909050",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "created_time": "2018-04-14T00:05:47.115378933Z",
   "current_version": 2,
   "max_versions": 0,
   "oldest_version": 0,
   "updated_time": "2018-04-14T00:13:35.296018431Z",
   "versions": {
     "1": {
       "created_time": "2018-04-14T00:05:47.115378933Z",
       "deletion_time": "",
       "destroyed": false
     },
     "2": {
       "created_time": "2018-04-14T00:13:35.296018431Z",
       "deletion_time": "",
       "destroyed": false
     }
   }
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

» Step 4: Specify the number of versions to keep

By default, the kv-v2 secret engine keeps up to 10 versions. Let's limit the maximum number of versions to keep to be 4.

» CLI command

To set the secret/ to keep up to 4 versions:

$ vault write secret/config max_versions=4
Success! Data written to: secret/config

# View the configuration settings
$ vault read secret/config
Key             Value
---             -----
cas_required    false
max_versions    4

Alternatively, to limit the number of versions only on the secret/customer/acme path rather than the entire secret/ engine:

$ vault kv metadata put -max-versions=4 secret/customer/acme

Overwrite the data a few more times to see what happens to the data.

$ vault kv metadata get secret/customer/acme
======= Metadata =======
Key                Value
---                -----
created_time       2018-04-14T00:42:25.677078177Z
current_version    6
max_versions       0
oldest_version     3
updated_time       2018-04-16T00:17:23.930473344Z

====== Version 3 ======
Key              Value
---              -----
created_time     2018-04-16T00:15:59.880368849Z
deletion_time    n/a
destroyed        false

====== Version 4 ======
Key              Value
---              -----
created_time     2018-04-16T00:16:18.941331243Z
deletion_time    n/a
destroyed        false

====== Version 5 ======
Key              Value
---              -----
created_time     2018-04-16T00:16:34.407951572Z
deletion_time    n/a
destroyed        false

====== Version 6 ======
Key              Value
---              -----
created_time     2018-04-16T00:17:23.930473344Z
deletion_time    n/a
destroyed        false

In this example, the current version is 6. Notice that version 1 and 2 do not show up in the metadata. Because the kv secret engine is configured to keep only 4 versions, the oldest two versions are permanently deleted and you won't be able to read them.

$ vault kv get -version=1 secret/customer/acme
No value found at secret/data/customer/data

» API call using cURL

To set the secret/ to keep up to 4 versions:

$ tee payload.json<<EOF
{
  "max_versions": 4,
  "cas_required": false
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json
       http://127.0.0.1:8200/v1/secret/config

To view the configuration:

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/config | jq
{
 "request_id": "8addfed1-41eb-6a19-8342-93f493c51538",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "cas_required": false,
   "max_versions": 4
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

Alternatively, to limit the number of versions only on the secret/customer/acme path rather than the entire secret/ engine:

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json
       http://127.0.0.1:8200/v1/secret/metadata/customer/acme

Invoke the secret/metadata/customer/acme endpoint instead.

Overwrite the data a few more times to see what happens to the data.

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/metadata/customer/acme | jq
{
 "request_id": "f2dd7f69-294c-e5c3-d582-f723005ea243",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "created_time": "2018-04-14T00:42:25.677078177Z",
   "current_version": 6,
   "max_versions": 0,
   "oldest_version": 3,
   "updated_time": "2018-04-16T00:17:23.930473344Z",
   "versions": {
     "3": {
       "created_time": "2018-04-16T00:15:59.880368849Z",
       "deletion_time": "",
       "destroyed": false
     },
     "4": {
       "created_time": "2018-04-16T00:16:18.941331243Z",
       "deletion_time": "",
       "destroyed": false
     },
     "5": {
       "created_time": "2018-04-16T00:16:34.407951572Z",
       "deletion_time": "",
       "destroyed": false
     },
     "6": {
       "created_time": "2018-04-16T00:17:23.930473344Z",
       "deletion_time": "",
       "destroyed": false
     }
   }
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

In this example, the current version is 6. Notice that version 1 and 2 do not show up in the metadata. Because the kv secret engine is configured to keep only 4 versions, the oldest two versions are permanently deleted and you won't be able to read them.

$ curl --header "X-Vault-Token: ..." \
       http://127.0.0.1:8200/v1/secret/data/customer/acme?version=1 | jq
{
 "errors": []
}

» Step 5: Delete versions of secret

» CLI command

Let's delete versions 4 and 5:

$ vault kv delete -versions="4,5" secret/customer/acme
Success! Data deleted (if it existed) at: secret/customer/acme

# Check the metadata
$ vault kv metadata get secret/customer/acme
...
====== Version 4 ======
Key              Value
---              -----
created_time     2018-04-16T00:12:25.404198622Z
deletion_time    2018-04-16T01:04:01.160426888Z
destroyed        false

====== Version 5 ======
Key              Value
---              -----
created_time     2018-04-16T00:12:47.527981267Z
deletion_time    2018-04-16T01:04:01.160427742Z
destroyed        false
...

The metadata on versions 4 and 5 reports its deletion timestamp (deletion_time); however, the destroyed parameter is set to false.

If version 5 was deleted by mistake and you wish to recover, invoke the vault kv undelete command:

$ vault kv undelete -versions=5 secret/customer/acme
Success! Data written to: secret/undelete/customer/acme

» API call using cURL

Let's delete versions 4 and 5:

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{ "versions":[4,5] }'
       http://127.0.0.1:8200/v1/secret/delete/customer/acme

# Check the metadata
$ curl --header "X-Vault-Token: ..." \
      http://127.0.0.1:8200/v1/secret/metadata/customer/acme | jq
...
"4": {
   "created_time": "2018-04-16T00:16:18.941331243Z",
   "deletion_time": "2018-04-16T01:17:42.003111567Z",
   "destroyed": false
 },
 "5": {
   "created_time": "2018-04-16T00:16:34.407951572Z",
   "deletion_time": "2018-04-16T01:17:42.003111978Z",
   "destroyed": false
 },
...

The metadata on versions 4 and 5 reports its deletion timestamp (deletion_time); however, the destroyed parameter is set to false.

If version 5 was deleted by mistake and you wish to recover, invoke the /secret/undelete endpoint:

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{ "versions":[5] }'
       http://127.0.0.1:8200/v1/secret/undelete/customer/acme

» Step 6: Permanently delete data

» CLI command

To permanently delete a version of secret:

$ vault kv destroy -versions=4 secret/customer/acme
Success! Data written to: secret/destroy/customer/acme

# Check the metadata
$ vault kv metadata get secret/customer/acme
...
====== Version 4 ======
Key              Value
---              -----
created_time     2018-04-16T00:12:25.404198622Z
deletion_time    2018-04-16T01:04:01.160426888Z
destroyed        true
...

The metadata indicates that Version 4 is destroyed.

If you wish to destroy all the keys and versions at secret/customer/acme, invoke the vault kv metadata delete command:

$ vault kv metadata delete secret/customer/acme
Success! Data deleted (if it existed) at: secret/metadata/customer/acme

» API call using cURL

To permanently delete a version of secret:

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{ "versions":[4] }'
       http://127.0.0.1:8200/v1/secret/destroy/customer/acme

# Check the metadata
$ curl --header "X-Vault-Token: ..." \
      http://127.0.0.1:8200/v1/secret/metadata/customer/acme | jq
...
  "4": {
    "created_time": "2018-04-16T00:16:18.941331243Z",
    "deletion_time": "2018-04-16T01:17:42.003111567Z",
    "destroyed": true
  },
...

The metadata indicates that Version 4 is destroyed.

If you wish to destroy all the keys and versions at secret/customer/acme, invoke the secret/metadata endpoint:

$ curl --header "X-Vault-Token: ..." \
       --request DELETE
       http://127.0.0.1:8200/v1/secret/metadata/customer/acme

» Additional Discussion

The v2 of KV secret engine supports a Check-And-Set operation to prevent unintentional secret overwrite. When you pass the cas flag to Vault, it first checks if the key already exists.

By default, Check-And-Set operation is not enabled on the KV secret engine; therefore, write is always allowed (no checking is performed).

$ vault read secret/config
Key             Value
---             -----
cas_required    false
max_versions    0

» CLI command

To enable the Check-And-Set operation:

# Enable cas_requied on the secret engine mounted at secret/
$ vault write secret/config cas-required=true

# Enable cas_requied only on the secret/partner path
$ vault kv metadata put -cas-required=true secret/partner

Once check-and-set is enabled, every write operation requires cas value to be passed. If you are sure that you want to overwrite the existing key-value, set cas to match the current version. Set cas to 0 if you want to write the secret only if the key does not exists.

Example:

# To write if the key does not already exists
$ vault kv put -cas=0 secret/partner name="Example Co." partner_id="123456789"
Key              Value
---              -----
created_time     2018-04-16T22:58:15.798753323Z
deletion_time    n/a
destroyed        false
version          1

# To overwrite the secret, you must specify the current version with -cas flag
$ vault kv put -cas=1 secret/partner name="Example Co." partner_id="ABCDEFGHIJKLMN"
Key              Value
---              -----
created_time     2018-04-16T23:00:28.66552289Z
deletion_time    n/a
destroyed        false
version          2

» API call using cURL

To enable the Check-And-Set operation:

$ tee payload.json<<EOF
{
  "max_versions": 10,
  "cas_required": true
}
EOF

# Enable cas_requied on the secret engine mounted at secret/
$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json
       http://127.0.0.1:8200/v1/secret/config

# Enable cas_requied only on the secret/partner path
$ curl --header "X-Vault-Token: ..." \
      --request POST \
      --data @payload.json
      http://127.0.0.1:8200/v1/secret/metadata/partner

Once check-and-set is enabled, every write operation requires cas value to be passed. If you are sure that you want to overwrite the existing key-value, set cas to match the current version. Set cas to 0 if you want to write the secret only if the key does not exists.

Example:

# Write if the key does not already exists
$ tee payload.json <<EOF
{
  "options": {
    "cas": 0
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "123456789"
  }
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/secret/data/partner

# To overwrite the secret, you must pass the current version
$ tee payload.json <<EOF
{
  "options": {
    "cas": 1
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "ABCDEFGHIJKLMN"
  }
}
EOF

$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/secret/data/partner


» Next steps

This guide introduced the CLI commands and API endpoints to read and write static secrets in the key-value secret engine. Read Secret as a Service: Dynamic Secrets guide to learn about the usage of database secret engine.