New Vault OSS Now Includes Multi-factor Authentication! Learn more
  • Overview
    • Automated PKI Infrastructure
    • Data Encryption & Tokenization
    • Database Credential Rotation
    • Dynamic Secrets
    • Identity-based Access
    • Key Management
    • Kubernetes Secrets
    • Secrets Management
  • Enterprise
  • Tutorials
  • Docs
  • API
  • Community
GitHubTry Cloud
Download
    • v1.10.x (latest)
    • v1.9.x
    • v1.8.x
    • v1.7.x
    • v1.6.x
    • v1.5.x
    • v1.4.x
  • What is Vault?
  • Use Cases
    • CLI Quick Start
    • HCP Quick Start
    • Developer Quick Start

  • Browser Support
  • Installing Vault
    • Overview
    • Architecture
    • High Availability
    • Integrated Storage
    • Security Model
    • Telemetry
    • Token Authentication
    • Key Rotation
    • Replication
    • Limits and Maximums
    • Overview
    • 'Dev' Server
    • Seal/Unseal
    • Namespace API Lock
    • Lease, Renew, and Revoke
    • Authentication
    • Tokens
    • Identity
    • OIDC Provider
    • Response Wrapping
    • Policies
    • Password Policies
    • Username Templating
    • High Availability
    • Storage
      • Overview
      • Autopilot
    • PGP, GnuPG, and Keybase
    • Recovery Mode
    • Resource Quotas
      • Overview
      • FAQ
    • Transform
    • Mount Migration
    • Overview
      • Overview
      • TCP
    • replication
      • Overview
      • AliCloud KMS
      • AWS KMS
      • Azure Key Vault
      • GCP Cloud KMS
      • OCI KMS
      • HSM PKCS11 ENT
      • Vault Transit
    • sentinel
      • Overview
      • Consul
      • Kubernetes
      • Overview
      • Aerospike
      • Alicloud OSS
      • Azure
      • Cassandra
      • CockroachDB
      • Consul
      • CouchDB
      • DynamoDB
      • Etcd
      • Filesystem
      • FoundationDB
      • Google Cloud Spanner
      • Google Cloud Storage
      • In-Memory
      • Manta
      • MSSQL
      • MySQL
      • OCI Object Storage
      • PostgreSQL
      • Integrated Storage (Raft)
      • S3
      • Swift
      • Zookeeper
    • telemetry
    • ui
    • Log Completed Requests
    • Entropy Augmentation ENT
    • kms_library ENT
    • Overview
    • agent
      • Overview
      • disable
      • enable
      • list
      • Overview
      • disable
      • enable
      • help
      • list
      • move
      • tune
    • debug
    • delete
      • Overview
      • delete
      • destroy
      • enable-versioning
      • get
      • list
      • metadata
      • patch
      • put
      • rollback
      • undelete
      • Overview
      • lookup
      • renew
      • revoke
      • Overview
      • get
      • inspect
    • list
    • login
    • monitor
    • namespace
      • Overview
      • diagnose
      • generate-root
      • init
      • key-status
      • members
      • migrate
      • raft
      • rekey
      • rotate
      • seal
      • step-down
      • unseal
      • usage
    • path-help
      • Overview
      • deregister
      • info
      • list
      • register
      • reload
      • Overview
      • delete
      • fmt
      • list
      • read
      • write
    • read
      • Overview
      • disable
      • enable
      • list
      • move
      • tune
    • server
    • ssh
    • status
      • Overview
      • capabilities
      • create
      • lookup
      • renew
      • revoke
    • unwrap
    • version
    • version-history
    • write
    • Token Helpers
    • Overview
      • Overview
        • Overview
        • AliCloud
        • AppRole
        • AWS
        • Azure
        • Cert
        • CF
        • GCP
        • JWT
        • Kerberos
        • Kubernetes
        • Overview
        • File
      • Overview
        • Overview
        • Kubernetes
    • Templates
    • Windows service

    • Overview
    • Active Directory
    • AliCloud
    • AWS
    • Azure
    • Consul
    • Cubbyhole
      • Overview
      • Cassandra
      • Couchbase
      • Elasticsearch
      • HanaDB
      • IBM Db2
      • InfluxDB
      • MongoDB
      • MongoDB Atlas
      • MSSQL
      • MySQL/MariaDB
      • Oracle
      • PostgreSQL
      • Redshift
      • Snowflake
      • Custom
    • Google Cloud
    • Google Cloud KMS
      • Overview
      • Azure Key Vault
      • AWS KMS
      • GCP Cloud KMS
    • KMIP ENTERPRISE
      • Overview
      • K/V Version 1
      • K/V Version 2
      • Overview
      • Identity Tokens
      • OIDC Identity Provider
    • MongoDB Atlas
    • Nomad
    • OpenLDAP
    • PKI (Certificates)
    • RabbitMQ
      • Overview
      • Signed Certificates
      • SSH OTP
      • Dynamic Key
    • Terraform Cloud
    • TOTP
      • Overview
      • FF3-1 Tweak Usage
      • Tokenization Transform ENTERPRISE
    • Transit
    • Venafi (Certificates)
    • Overview
    • AppRole
    • AliCloud
    • AWS
    • Azure
    • Cloud Foundry
    • GitHub
    • Google Cloud
      • Overview
      • OIDC Providers
    • Kerberos
    • Kubernetes
    • LDAP
      • Overview
      • FAQ
    • Oracle Cloud Infrastructure
    • Okta
    • RADIUS
    • TLS Certificates
    • Tokens
    • Username & Password

    • App ID DEPRECATED
    • MFA LEGACY / UNSUPPORTED
    • Overview
    • File
    • Syslog
    • Socket
    • Overview
    • Plugin Architecture
    • Plugin Development
    • Plugin Management
    • Plugin Portal
  • Vault Integration Program
  • Troubleshoot

    • Overview
      • Overview
      • Agent Injector vs. Vault CSI Provider
        • Overview
        • Running Vault
        • Enterprise Licensing
        • Running Vault on OpenShift
        • Configuration
          • Overview
          • Development
          • Standalone with Load Balanced UI
          • Standalone with TLS
          • Standalone with Audit Storage
          • External Vault
          • Using Kubernetes Auth Method
          • HA Cluster with Consul
          • HA Cluster with Raft
          • HA Enterprise Cluster with Raft
          • HA Enterprise DR Clusters with Raft
          • HA Enterprise Performance Clusters with Raft
          • Vault Agent Injector TLS Configuration
          • Vault Agent Injector TLS with Cert-Manager
        • Overview
        • Annotations
        • Installation
        • Examples
        • Overview
        • Installation
        • Configurations
        • Examples
      • Overview
      • Vault Lambda Extension
      • Running Vault
      • Overview
      • Installation
      • Configuration
      • Troubleshooting
      • Overview
      • Installation
      • Troubleshooting

    • Overview
    • Upgrade Plugins
    • Upgrade to 1.10.x
    • Upgrade to 1.9.x
    • Upgrade to 1.8.x
    • Upgrade to 1.7.x
    • Upgrade to 1.6.3
    • Upgrade to 1.6.2
    • Upgrade to 1.6.1
    • Upgrade to 1.6.0
    • Upgrade to 1.5.3
    • Upgrade to 1.5.2
    • Upgrade to 1.5.1
    • Upgrade to 1.5.0
    • Upgrade to 1.4.6
    • Upgrade to 1.4.5
    • Upgrade to 1.4.4
    • Upgrade to 1.4.1
    • Upgrade to 1.4.0
    • Upgrade to 1.3.10
    • Upgrade to 1.3.9
    • Upgrade to 1.3.8
    • Upgrade to 1.3.5
    • Upgrade to 1.3.4
    • Upgrade to 1.3.3
    • Upgrade to 1.3.2
    • Upgrade to 1.3.0
    • Upgrade to 1.2.7
    • Upgrade to 1.2.6
    • Upgrade to 1.2.5
    • Upgrade to 1.2.4
    • Upgrade to 1.2.1
    • Upgrade to 1.2.0
    • Upgrade to 1.1.2
    • Upgrade to 1.1.1
    • Upgrade to 1.1.0
    • Upgrade to 1.0.0
    • Upgrade to 0.11.6
    • Upgrade to 0.11.2
    • Upgrade to 0.11.0
    • Upgrade to 0.10.4
    • Upgrade to 0.10.2
    • Upgrade to 0.10.0
    • Upgrade to 0.9.6
    • Upgrade to 0.9.3
    • Upgrade to 0.9.2
    • Upgrade to 0.9.1
    • Upgrade to 0.9.0
    • Upgrade to 0.8.0
    • Upgrade to 0.7.0
    • Upgrade to 0.6.4
    • Upgrade to 0.6.3
    • Upgrade to 0.6.2
    • Upgrade to 0.6.1
    • Upgrade to 0.6.0
    • Upgrade to 0.5.1
    • Upgrade to 0.5.0

    • Overview
    • 1.10.0
    • 1.9.0
    • 1.8.0
    • 1.7.0
    • 1.6.0
    • 1.5.0

    • Overview
    • FAQ

    • Overview
    • Feature Deprecation Notice and Plans
    • License
    • Client Count
    • Login MFA
    • Server Side Consistent Token

  • Glossary

    • Overview
      • Overview
      • Autoloading
      • FAQ
    • Replication
      • Overview
      • Behavioral Changes
      • Security
    • Automated Integrated Storage Snapshots
    • Lease Count Quotas
    • Entropy Augmentation
    • Seal Wrap / FIPS 140-2
    • Namespaces
    • Performance Standbys
    • Eventual Consistency
    • Control Groups
    • Managed Keys
      • Overview
      • Duo MFA
      • Okta MFA
      • PingID MFA
      • TOTP MFA
      • Overview
      • Examples
      • Properties
    • HCP Vault
Type '/' to Search

»Replication (Vault Enterprise)

Vault Enterprise 0.7 adds support for multi-datacenter replication. Before using this feature, it is useful to understand the intended use cases, design goals, and high level architecture.

Replication is based on a primary/secondary (1:N) model with asynchronous replication, focusing on high availability for global deployments. The trade-offs made in the design and implementation of replication reflect these high level goals.

»Use Cases

Vault replication is based on a number of common use cases:

  • Multi-Datacenter Deployments: A common challenge is providing Vault to applications across many datacenters in a highly-available manner. Running a single Vault cluster imposes high latency of access for remote clients, availability loss or outages during connectivity failures, and limits scalability.

  • Backup Sites: Implementing a robust business continuity plan around the loss of a primary datacenter requires the ability to quickly and easily fail to a hot backup site.

  • Scaling Throughput: Applications that use Vault for Encryption-as-a-Service or cryptographic offload may generate a very high volume of requests for Vault. Replicating keys between multiple clusters allows load to be distributed across additional servers to scale request throughput.

»Design Goals

Based on the use cases for Vault Replication, we had a number of design goals for the implementation:

  • Availability: Global deployments of Vault require high levels of availability, and can tolerate reduced consistency. During full connectivity, replication is nearly real-time between the primary and secondary clusters. Degraded connectivity between a primary and secondary does not impact the primary's ability to service requests, and the secondary will continue to service reads on last-known data.

  • Conflict Free: Certain replication techniques allow for potential write conflicts to take place. Particularly, any active/active configuration where writes are allowed to multiple sites require a conflict resolution strategy. This varies from techniques that allow for data loss like last-write-wins, or techniques that require manual operator resolution like allowing multiple values per key. We avoid the possibility of conflicts to ensure there is no data loss or manual intervention required.

  • Transparent to Clients: Vault replication should be transparent to clients of Vault, so that existing thin clients work unmodified. The Vault servers handle the logic of request forwarding to the primary when necessary, and multi-hop routing is performed internally to ensure requests are processed.

  • Simple to Operate: Operating a replicated cluster should be simple to avoid administrative overhead and potentially introducing security gaps. Setup of replication is very simple, and secondaries can handle being arbitrarily behind the primary, avoiding the need for operator intervention to copy data or snapshot the primary.

»Architecture

The architecture of Vault replication is based on the design goals, focusing on the intended use cases. When replication is enabled, a cluster is set as either a primary or secondary. The primary cluster is authoritative, and is the only cluster allowed to perform actions that write to the underlying data storage, such as modifying policies or secrets. Secondary clusters can service all other operations, such as reading secrets or sending data through transit, and forward any writes to the primary cluster. Disallowing multiple primaries ensures the cluster is conflict free and has an authoritative state.

The primary cluster uses log shipping to replicate changes to all of the secondaries. This ensures writes are visible globally in near real-time when there is full network connectivity. If a secondary is down or unable to communicate with the primary, writes are not blocked on the primary and reads are still serviced on the secondary. This ensures the availability of Vault. When the secondary is initialized or recovers from degraded connectivity it will automatically reconcile with the primary.

Lastly, clients can speak to any Vault server without a thick client. If a client is communicating with a standby instance, the request is automatically forwarded to an active instance. Secondary clusters will service reads locally and forward any write requests to the primary cluster. The primary cluster is able to service all request types.

An important optimization Vault makes is to avoid replication of tokens or leases between clusters. Policies and secrets are the minority of data managed by Vault and tend to be relatively stable. Tokens and leases are much more dynamic, as they are created and expire rapidly. Keeping tokens and leases locally reduces the amount of data that needs to be replicated, and distributes the work of TTL management between the clusters. The caveat is that clients will need to re-authenticate if they switch the Vault cluster they are communicating with.

»Implementation Details

It is important to understand the high-level architecture of replication to ensure the trade-offs are appropriate for your use case. The implementation details may be useful for those who are curious or want to understand more about the performance characteristics or failure scenarios.

Using replication requires a storage backend that supports transactional updates, such as Consul. This allows multiple key/value updates to be performed atomically. Replication uses this to maintain a Write-Ahead-Log (WAL) of all updates, so that the key update happens atomically with the WAL entry creation. The WALs are then used to perform log shipping between the Vault clusters. When a secondary is closely synchronized with a primary, Vault directly streams new WALs to be applied, providing near real-time replication. A bounded set of WALs are maintained for the secondaries, and older WALs are garbage collected automatically.

When a secondary is initialized or is too far behind the primary there may not be enough WALs to synchronize. To handle this scenario, Vault maintains a merkle index of the encrypted keys. Any time a key is updated or deleted, the merkle index is updated to reflect the change. When a secondary needs to reconcile with a primary, they compare their merkle indexes to determine which keys are out of sync. The structure of the index allows this to be done very efficiently, usually requiring only two round trips and a small amount of data. The secondary uses this information to reconcile and then switches back into WAL streaming mode.

Performance is an important concern for Vault, so WAL entries are batched and the merkle index is not flushed to disk with every operation. Instead, the index is updated in memory for every operation and asynchronously flushed to disk. As a result, a crash or power loss may cause the merkle index to become out of sync with the underlying keys. Vault uses the ARIES recovery algorithm to ensure the consistency of the index under those failure conditions.

Log shipping traditionally requires the WAL stream to be synchronized, which can introduce additional complexity when a new primary cluster is promoted. Vault uses the merkle index as the source of truth, allowing the WAL streams to be completely distinct and unsynchronized. This simplifies administration of Vault Replication for operators.

»Caveats

Mismatched Cluster Versions: It is not safe to replicate from a newer version of Vault to an older version. When upgrading replicated clusters, ensure that upstream clusters are always on older version of Vault than downstream clusters. See Upgrading Vault for an example.

  • Read-After-Write Consistency: All write requests are forwarded from secondaries to the primary cluster in order to avoid potential conflicts. While replication is near real-time, it is not instantaneous, meaning there is a potential for a client to write to a secondary and a subsequent read to return an old value. Secondaries attempt to mask this from an individual client making subsequent requests by stalling write requests until the write is replicated or a timeout is reached (2 seconds). If the timeout is reached, the client will receive a warning.

  • Stale Reads: Secondary clusters service reads based on their locally-replicated data. During normal operation updates from a primary are received in near real-time by secondaries. However, during an outage or network service disruption, replication may stall and secondaries may have stale data. The cluster will automatically recover and reconcile any stale data once the outage has recovered, but reads in the intervening period may receive stale data.

github logoEdit this page
DocsAPILearnCommunityPrivacySecurityPress KitConsent Manager