• Overview
    • Automated PKI Infrastructure
    • Data Encryption & Tokenization
    • Database Credential Rotation
    • Dynamic Secrets
    • Identity-based Access
    • Key Management
    • Kubernetes Secrets
    • Secrets Management
  • Enterprise
  • Tutorials
  • Docs
  • API
  • Community
GitHubTry Cloud
Download
    • v1.11.x (latest)
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • v1.7.x
    • v1.6.x
    • v1.5.x
    • v1.4.x
  • What is Vault?
  • Use Cases
    • CLI Quick Start
    • HCP Quick Start
    • Developer Quick Start

  • Browser Support
  • Installing Vault
    • Overview
    • Architecture
    • High Availability
    • Integrated Storage
    • Security Model
    • Telemetry
    • Token Authentication
    • Key Rotation
    • Replication
    • Limits and Maximums
    • Overview
    • 'Dev' Server
    • Seal/Unseal
    • Namespace API Lock
    • Lease, Renew, and Revoke
    • Authentication
    • Tokens
    • Identity
    • OIDC Provider
    • Response Wrapping
    • Policies
    • Password Policies
    • Username Templating
    • High Availability
    • Storage
      • Overview
      • Autopilot
    • PGP, GnuPG, and Keybase
    • Recovery Mode
    • Resource Quotas
      • Overview
      • FAQ
    • Transform
    • Mount Migration
    • Overview
      • Overview
      • TCP
    • replication
      • Overview
      • AliCloud KMS
      • AWS KMS
      • Azure Key Vault
      • GCP Cloud KMS
      • OCI KMS
      • HSM PKCS11 ENT
      • Vault Transit
    • sentinel
      • Overview
      • Consul
      • Kubernetes
      • Overview
      • Aerospike
      • Alicloud OSS
      • Azure
      • Cassandra
      • CockroachDB
      • Consul
      • CouchDB
      • DynamoDB
      • Etcd
      • Filesystem
      • FoundationDB
      • Google Cloud Spanner
      • Google Cloud Storage
      • In-Memory
      • Manta
      • MSSQL
      • MySQL
      • OCI Object Storage
      • PostgreSQL
      • Integrated Storage (Raft)
      • S3
      • Swift
      • Zookeeper
    • telemetry
    • ui
    • Log Completed Requests
    • Entropy Augmentation ENT
    • kms_library ENT
    • Overview
    • agent
      • Overview
      • disable
      • enable
      • list
      • Overview
      • disable
      • enable
      • help
      • list
      • move
      • tune
    • debug
    • delete
      • Overview
      • delete
      • destroy
      • enable-versioning
      • get
      • list
      • metadata
      • patch
      • put
      • rollback
      • undelete
      • Overview
      • lookup
      • renew
      • revoke
      • Overview
      • get
      • inspect
    • list
    • login
    • monitor
    • namespace
      • Overview
      • diagnose
      • generate-root
      • init
      • key-status
      • members
      • migrate
      • raft
      • rekey
      • rotate
      • seal
      • step-down
      • unseal
      • usage
    • path-help
      • Overview
      • deregister
      • info
      • list
      • register
      • reload
      • Overview
      • delete
      • fmt
      • list
      • read
      • write
    • read
      • Overview
      • disable
      • enable
      • list
      • move
      • tune
    • server
    • ssh
    • status
      • Overview
      • capabilities
      • create
      • lookup
      • renew
      • revoke
    • unwrap
    • version
    • version-history
    • write
    • Token Helpers
    • Overview
      • Overview
        • Overview
        • AliCloud
        • AppRole
        • AWS
        • Azure
        • Cert
        • CF
        • GCP
        • JWT
        • Kerberos
        • Kubernetes
        • Overview
        • File
      • Overview
        • Overview
        • Kubernetes
    • Templates
    • Windows service

    • Overview
    • Active Directory
    • AliCloud
    • AWS
    • Azure
    • Consul
    • Cubbyhole
      • Overview
      • Cassandra
      • Couchbase
      • Elasticsearch
      • HanaDB
      • IBM Db2
      • InfluxDB
      • MongoDB
      • MongoDB Atlas
      • MSSQL
      • MySQL/MariaDB
      • Oracle
      • PostgreSQL
      • Redshift
      • Snowflake
      • Custom
    • Google Cloud
    • Google Cloud KMS
      • Overview
      • Identity Tokens
      • OIDC Identity Provider
      • Overview
      • Azure Key Vault
      • AWS KMS
      • GCP Cloud KMS
      • Overview
      • K/V Version 1
      • K/V Version 2
    • KMIP ENTERPRISE
    • Kubernetes
    • MongoDB Atlas
    • Nomad
    • OpenLDAP
      • Overview
      • Setup and Usage
      • Quick Start - Root CA Setup
      • Quick Start - Intermediate CA Setup
      • Considerations
      • Rotation Primitives
    • RabbitMQ
      • Overview
      • Signed Certificates
      • SSH OTP
      • Dynamic Key
    • Terraform Cloud
    • TOTP
      • Overview
      • FF3-1 Tweak Usage
      • Tokenization Transform ENTERPRISE
    • Transit
    • Venafi (Certificates)
    • Overview
    • AppRole
    • AliCloud
    • AWS
    • Azure
    • Cloud Foundry
    • GitHub
    • Google Cloud
      • Overview
      • OIDC Providers
    • Kerberos
    • Kubernetes
    • LDAP
      • Overview
      • FAQ
    • Oracle Cloud Infrastructure
    • Okta
    • RADIUS
    • TLS Certificates
    • Tokens
    • Username & Password

    • App ID DEPRECATED
    • Overview
    • File
    • Syslog
    • Socket
    • Overview
    • Plugin Architecture
    • Plugin Development
    • Plugin Management
    • Plugin Portal
  • Vault Integration Program
  • Troubleshoot

    • Overview
      • Overview
      • Agent Injector vs. Vault CSI Provider
        • Overview
        • Running Vault
        • Enterprise Licensing
        • Running Vault on OpenShift
        • Configuration
          • Overview
          • Development
          • Standalone with Load Balanced UI
          • Standalone with TLS
          • Standalone with Audit Storage
          • External Vault
          • Using Kubernetes Auth Method
          • HA Cluster with Consul
          • HA Cluster with Raft
          • HA Enterprise Cluster with Raft
          • HA Enterprise DR Clusters with Raft
          • HA Enterprise Performance Clusters with Raft
          • Vault Agent Injector TLS Configuration
          • Vault Agent Injector TLS with Cert-Manager
        • Overview
        • Annotations
        • Installation
        • Examples
        • Overview
        • Installation
        • Configurations
        • Examples
      • Overview
      • Vault Lambda Extension
      • Running Vault
      • Overview
      • Installation
      • Configuration
      • Troubleshooting
      • Overview
      • Installation
      • Configuration
      • Upgrading
      • Troubleshooting

    • Overview
    • Upgrade Plugins
    • Upgrade to 1.11.x
    • Upgrade to 1.10.x
    • Upgrade to 1.9.x
    • Upgrade to 1.8.x
    • Upgrade to 1.7.x
    • Upgrade to 1.6.3
    • Upgrade to 1.6.2
    • Upgrade to 1.6.1
    • Upgrade to 1.6.0
    • Upgrade to 1.5.3
    • Upgrade to 1.5.2
    • Upgrade to 1.5.1
    • Upgrade to 1.5.0
    • Upgrade to 1.4.6
    • Upgrade to 1.4.5
    • Upgrade to 1.4.4
    • Upgrade to 1.4.1
    • Upgrade to 1.4.0
    • Upgrade to 1.3.10
    • Upgrade to 1.3.9
    • Upgrade to 1.3.8
    • Upgrade to 1.3.5
    • Upgrade to 1.3.4
    • Upgrade to 1.3.3
    • Upgrade to 1.3.2
    • Upgrade to 1.3.0
    • Upgrade to 1.2.7
    • Upgrade to 1.2.6
    • Upgrade to 1.2.5
    • Upgrade to 1.2.4
    • Upgrade to 1.2.1
    • Upgrade to 1.2.0
    • Upgrade to 1.1.2
    • Upgrade to 1.1.1
    • Upgrade to 1.1.0
    • Upgrade to 1.0.0
    • Upgrade to 0.11.6
    • Upgrade to 0.11.2
    • Upgrade to 0.11.0
    • Upgrade to 0.10.4
    • Upgrade to 0.10.2
    • Upgrade to 0.10.0
    • Upgrade to 0.9.6
    • Upgrade to 0.9.3
    • Upgrade to 0.9.2
    • Upgrade to 0.9.1
    • Upgrade to 0.9.0
    • Upgrade to 0.8.0
    • Upgrade to 0.7.0
    • Upgrade to 0.6.4
    • Upgrade to 0.6.3
    • Upgrade to 0.6.2
    • Upgrade to 0.6.1
    • Upgrade to 0.6.0
    • Upgrade to 0.5.1
    • Upgrade to 0.5.0

    • Overview
    • 1.11.0
    • 1.10.0
    • 1.9.0
    • 1.8.0
    • 1.7.0
    • 1.6.0
    • 1.5.0

    • Overview
    • FAQ

    • Overview
    • Feature Deprecation Notice and Plans
    • License
    • Client Count
    • Login MFA
    • Server Side Consistent Token

  • Glossary

    • Overview
      • Overview
      • Autoloading
      • FAQ
    • Replication
      • Overview
      • Behavioral Changes
      • Security
    • Automated Integrated Storage Snapshots
    • Automated Upgrades
    • Redundancy Zones
    • Lease Count Quotas
    • Entropy Augmentation
      • Overview
      • FIPS 140-2 Inside Vault
      • Seal Wrap for FIPS 140-2
    • Seal Wrap
    • Namespaces
    • Performance Standbys
    • Eventual Consistency
    • Control Groups
    • Managed Keys
      • Overview
      • Duo MFA
      • Okta MFA
      • PingID MFA
      • TOTP MFA
      • Overview
      • Examples
      • Properties
    • HCP Vault

The Vault website is being redesigned to help you find what you are looking for more effectively.

Type '/' to Search

»Recovery Mode

Vault can be started using the -recovery flag to bring it up in Recovery Mode. The main purpose of recovery mode is to allow direct access to storage in case Vault isn't starting up due to some newly discovered bug. This probably won't be helpful without a Vault expert on hand to advise.

Differences between recovery mode and regular Vault operation:

  • none of the usual subsystems run, e.g. expiration, clustering, RPCs from other nodes
  • instead of a regular unseal request, unseal a node by generating a recovery token
  • all requests are to sys/raw and are authenticated using the recovery token

»Recovery process

The usual way recovery mode is used is:

  • seal or stop all nodes in the cluster
  • if using Integrated Storage, run vault status on each node to find the highest-index ones (this will require they be running and sealed, as if unsealed a new leader might be elected and writes could happen, confusing the issue)
  • restart the target node in recovery mode
  • generate a recovery token on that node
  • use the recovery token to perform sys/raw requests to repair the node
  • if using Integrated Storage, reform the raft cluster as described below

»Integrated storage for HA only (ha_storage)

If Integrated Storage is used in hybrid mode (i.e. for ha_storage), recovery mode will not allow for changes to the Raft data but instead allow for modification of the underlying physical data that is associated with Vault's storage backend. This means that the notes regarding Integrated Storage in this doc do not apply.

»Integrated storage

With Integrated Storage, not all nodes are equal. It's possible that some nodes are further behind - i.e. haven't applied as many Raft logs. It is important when choosing a node to use for recovery that it has the highest AppliedIndex found in the cluster.

Each node's AppliedIndex value can be obtained by running vault status against the node sealed nodes of the cluster after bringing it down.

»Recovery tokens

Recovery tokens are issued in much the same way as root tokens are generated, only using a different endpoint, and the Vault node must be sealed first. Unlike root tokens, the recovery token is not persisted, so if Vault is restarted into recovery mode a new one must be generated.

Only a single recovery token can be generated. If lost, restart Vault and generate a new one.

»Raw requests

Requests can be issued to sys/raw in just the same way as in regular Vault server mode. The only difference is that in recovery mode, X-Vault-Token must contain a recovery token instead of a service or batch token.

»Reform the raft cluster

Recovery mode Vault automatically resizes the cluster to size 1. This is necessary because the Raft protocol won't allow changes to be made without a quorum, and in recovery mode we wish to make changes using a single node.

This means that after having used recovery mode, part of the procedure for returning to active service must include re-forming the raft cluster. There are two ways to do so: either delete the vault data directory on the other nodes and re-join them to the recovered node, or use the Manual Recovery Using peers.json approach to get all nodes to agree on what nodes are part of the cluster.

github logoEdit this page
DocsAPILearnCommunityPrivacySecurityPress KitConsent Manager