New Vault OSS Now Includes Multi-factor Authentication! Learn more
  • Overview
    • Automated PKI Infrastructure
    • Data Encryption & Tokenization
    • Database Credential Rotation
    • Dynamic Secrets
    • Identity-based Access
    • Key Management
    • Kubernetes Secrets
    • Secrets Management
  • Enterprise
  • Tutorials
  • Docs
  • API
  • Community
GitHubTry Cloud
Download
    • v1.10.x (latest)
    • v1.9.x
    • v1.8.x
    • v1.7.x
    • v1.6.x
    • v1.5.x
    • v1.4.x
  • What is Vault?
  • Use Cases
    • CLI Quick Start
    • HCP Quick Start
    • Developer Quick Start

  • Browser Support
  • Installing Vault
    • Overview
    • Architecture
    • High Availability
    • Integrated Storage
    • Security Model
    • Telemetry
    • Token Authentication
    • Key Rotation
    • Replication
    • Limits and Maximums
    • Overview
    • 'Dev' Server
    • Seal/Unseal
    • Namespace API Lock
    • Lease, Renew, and Revoke
    • Authentication
    • Tokens
    • Identity
    • OIDC Provider
    • Response Wrapping
    • Policies
    • Password Policies
    • Username Templating
    • High Availability
    • Storage
      • Overview
      • Autopilot
    • PGP, GnuPG, and Keybase
    • Recovery Mode
    • Resource Quotas
      • Overview
      • FAQ
    • Transform
    • Mount Migration
    • Overview
      • Overview
      • TCP
    • replication
      • Overview
      • AliCloud KMS
      • AWS KMS
      • Azure Key Vault
      • GCP Cloud KMS
      • OCI KMS
      • HSM PKCS11 ENT
      • Vault Transit
    • sentinel
      • Overview
      • Consul
      • Kubernetes
      • Overview
      • Aerospike
      • Alicloud OSS
      • Azure
      • Cassandra
      • CockroachDB
      • Consul
      • CouchDB
      • DynamoDB
      • Etcd
      • Filesystem
      • FoundationDB
      • Google Cloud Spanner
      • Google Cloud Storage
      • In-Memory
      • Manta
      • MSSQL
      • MySQL
      • OCI Object Storage
      • PostgreSQL
      • Integrated Storage (Raft)
      • S3
      • Swift
      • Zookeeper
    • telemetry
    • ui
    • Log Completed Requests
    • Entropy Augmentation ENT
    • kms_library ENT
    • Overview
    • agent
      • Overview
      • disable
      • enable
      • list
      • Overview
      • disable
      • enable
      • help
      • list
      • move
      • tune
    • debug
    • delete
      • Overview
      • delete
      • destroy
      • enable-versioning
      • get
      • list
      • metadata
      • patch
      • put
      • rollback
      • undelete
      • Overview
      • lookup
      • renew
      • revoke
      • Overview
      • get
      • inspect
    • list
    • login
    • monitor
    • namespace
      • Overview
      • diagnose
      • generate-root
      • init
      • key-status
      • members
      • migrate
      • raft
      • rekey
      • rotate
      • seal
      • step-down
      • unseal
      • usage
    • path-help
      • Overview
      • deregister
      • info
      • list
      • register
      • reload
      • Overview
      • delete
      • fmt
      • list
      • read
      • write
    • read
      • Overview
      • disable
      • enable
      • list
      • move
      • tune
    • server
    • ssh
    • status
      • Overview
      • capabilities
      • create
      • lookup
      • renew
      • revoke
    • unwrap
    • version
    • version-history
    • write
    • Token Helpers
    • Overview
      • Overview
        • Overview
        • AliCloud
        • AppRole
        • AWS
        • Azure
        • Cert
        • CF
        • GCP
        • JWT
        • Kerberos
        • Kubernetes
        • Overview
        • File
      • Overview
        • Overview
        • Kubernetes
    • Templates
    • Windows service

    • Overview
    • Active Directory
    • AliCloud
    • AWS
    • Azure
    • Consul
    • Cubbyhole
      • Overview
      • Cassandra
      • Couchbase
      • Elasticsearch
      • HanaDB
      • IBM Db2
      • InfluxDB
      • MongoDB
      • MongoDB Atlas
      • MSSQL
      • MySQL/MariaDB
      • Oracle
      • PostgreSQL
      • Redshift
      • Snowflake
      • Custom
    • Google Cloud
    • Google Cloud KMS
      • Overview
      • Azure Key Vault
      • AWS KMS
      • GCP Cloud KMS
    • KMIP ENTERPRISE
      • Overview
      • K/V Version 1
      • K/V Version 2
      • Overview
      • Identity Tokens
      • OIDC Identity Provider
    • MongoDB Atlas
    • Nomad
    • OpenLDAP
    • PKI (Certificates)
    • RabbitMQ
      • Overview
      • Signed Certificates
      • SSH OTP
      • Dynamic Key
    • Terraform Cloud
    • TOTP
      • Overview
      • FF3-1 Tweak Usage
      • Tokenization Transform ENTERPRISE
    • Transit
    • Venafi (Certificates)
    • Overview
    • AppRole
    • AliCloud
    • AWS
    • Azure
    • Cloud Foundry
    • GitHub
    • Google Cloud
      • Overview
      • OIDC Providers
    • Kerberos
    • Kubernetes
    • LDAP
      • Overview
      • FAQ
    • Oracle Cloud Infrastructure
    • Okta
    • RADIUS
    • TLS Certificates
    • Tokens
    • Username & Password

    • App ID DEPRECATED
    • MFA LEGACY / UNSUPPORTED
    • Overview
    • File
    • Syslog
    • Socket
    • Overview
    • Plugin Architecture
    • Plugin Development
    • Plugin Management
    • Plugin Portal
  • Vault Integration Program
  • Troubleshoot

    • Overview
      • Overview
      • Agent Injector vs. Vault CSI Provider
        • Overview
        • Running Vault
        • Enterprise Licensing
        • Running Vault on OpenShift
        • Configuration
          • Overview
          • Development
          • Standalone with Load Balanced UI
          • Standalone with TLS
          • Standalone with Audit Storage
          • External Vault
          • Using Kubernetes Auth Method
          • HA Cluster with Consul
          • HA Cluster with Raft
          • HA Enterprise Cluster with Raft
          • HA Enterprise DR Clusters with Raft
          • HA Enterprise Performance Clusters with Raft
          • Vault Agent Injector TLS Configuration
          • Vault Agent Injector TLS with Cert-Manager
        • Overview
        • Annotations
        • Installation
        • Examples
        • Overview
        • Installation
        • Configurations
        • Examples
      • Overview
      • Vault Lambda Extension
      • Running Vault
      • Overview
      • Installation
      • Configuration
      • Troubleshooting
      • Overview
      • Installation
      • Troubleshooting

    • Overview
    • Upgrade Plugins
    • Upgrade to 1.10.x
    • Upgrade to 1.9.x
    • Upgrade to 1.8.x
    • Upgrade to 1.7.x
    • Upgrade to 1.6.3
    • Upgrade to 1.6.2
    • Upgrade to 1.6.1
    • Upgrade to 1.6.0
    • Upgrade to 1.5.3
    • Upgrade to 1.5.2
    • Upgrade to 1.5.1
    • Upgrade to 1.5.0
    • Upgrade to 1.4.6
    • Upgrade to 1.4.5
    • Upgrade to 1.4.4
    • Upgrade to 1.4.1
    • Upgrade to 1.4.0
    • Upgrade to 1.3.10
    • Upgrade to 1.3.9
    • Upgrade to 1.3.8
    • Upgrade to 1.3.5
    • Upgrade to 1.3.4
    • Upgrade to 1.3.3
    • Upgrade to 1.3.2
    • Upgrade to 1.3.0
    • Upgrade to 1.2.7
    • Upgrade to 1.2.6
    • Upgrade to 1.2.5
    • Upgrade to 1.2.4
    • Upgrade to 1.2.1
    • Upgrade to 1.2.0
    • Upgrade to 1.1.2
    • Upgrade to 1.1.1
    • Upgrade to 1.1.0
    • Upgrade to 1.0.0
    • Upgrade to 0.11.6
    • Upgrade to 0.11.2
    • Upgrade to 0.11.0
    • Upgrade to 0.10.4
    • Upgrade to 0.10.2
    • Upgrade to 0.10.0
    • Upgrade to 0.9.6
    • Upgrade to 0.9.3
    • Upgrade to 0.9.2
    • Upgrade to 0.9.1
    • Upgrade to 0.9.0
    • Upgrade to 0.8.0
    • Upgrade to 0.7.0
    • Upgrade to 0.6.4
    • Upgrade to 0.6.3
    • Upgrade to 0.6.2
    • Upgrade to 0.6.1
    • Upgrade to 0.6.0
    • Upgrade to 0.5.1
    • Upgrade to 0.5.0

    • Overview
    • 1.10.0
    • 1.9.0
    • 1.8.0
    • 1.7.0
    • 1.6.0
    • 1.5.0

    • Overview
    • FAQ

    • Overview
    • Feature Deprecation Notice and Plans
    • License
    • Client Count
    • Login MFA
    • Server Side Consistent Token

  • Glossary

    • Overview
      • Overview
      • Autoloading
      • FAQ
    • Replication
      • Overview
      • Behavioral Changes
      • Security
    • Automated Integrated Storage Snapshots
    • Lease Count Quotas
    • Entropy Augmentation
    • Seal Wrap / FIPS 140-2
    • Namespaces
    • Performance Standbys
    • Eventual Consistency
    • Control Groups
    • Managed Keys
      • Overview
      • Duo MFA
      • Okta MFA
      • PingID MFA
      • TOTP MFA
      • Overview
      • Examples
      • Properties
    • HCP Vault
Type '/' to Search

»External Plugin Architecture

Vault's external plugins are completely separate, standalone applications that Vault executes and communicates with over RPC. This means the plugin process does not share the same memory space as Vault and therefore can only access the interfaces and arguments given to it. This also means a crash in a plugin can not crash the entirety of Vault.

It is possible to enable a custom plugin with a name that's identical to a built-in plugin. In such a situation, Vault will always choose the custom plugin when enabling it.

»External Plugin Lifecycle

Vault external plugins are long-running processes that remain running once they are spawned by Vault, the parent process. Plugin processes can be started by Vault's active node and performance standby nodes. Additionally, there are cases where plugin processes may be terminated by Vault. These cases include, but are not limited to:

  • Vault active node step-down
  • Vault barrier seal
  • Vault graceful shutdown
  • Disabling a Secrets Engine or Auth method that uses external plugins
  • Database configured connection deletion
  • Database configured connection update
  • Database configured connection reset request
  • Database root credentials rotation
  • WAL Rollback from a previously failed root credentials rotation operation

The lifecycle of plugin processes are managed automatically by Vault. Termination of these processes are typical in certain scenarios, such as the ones listed above. Vault will start plugin processes when needed, typically by lazily loading the plugin when a request that requires the plugin is received by Vault. A plugin process may be started or terminated through other internal processes within Vault as well. Since Vault manages and tracks the lifecycle of its plugins, these processes should not be terminated by anything other than Vault.

»External Plugin Scaling Characteristics

External plugins are able to leverage Performance Standbys without any explicit action by a plugin author. The default behavior of Vault Enterprise is to attempt to handle all requests, including requests to plugins, on performance standbys. If the plugin request makes any attempt to modify storage, the request will receive a readonly error, and the request routing code will then forward the full original request transparently to the active node. In other words, plugins can scale horizontally on Vault Enterprise without any effort on the plugin author's part.

»Plugin Communication

Vault creates a mutually authenticated TLS connection for communication with the plugin's RPC server. Database secrets engines make use of the AutoMTLS feature of go-plugin which will automatically negotiate mTLS for transport authentication. For all other plugins, Vault passes a wrapping token to the plugin process' environment. This token is single use and has a short TTL. Once unwrapped, it provides the plugin with a uniquely generated TLS certificate and private key for it to use to talk to the original Vault process.

The api_addr must be set in order for the plugin process to establish communication with the Vault server during mount time. If the storage backend has HA enabled and supports automatic host address detection (e.g. Consul), Vault will automatically attempt to determine the api_addr as well.

Note: Prior to Vault version 1.9.2, reading the original connection's TLS connection state is not supported in plugins.

»Plugin Registration

An important consideration of Vault's plugin system is to ensure the plugin invoked by Vault is authentic and maintains integrity. There are two components that a Vault operator needs to configure before external plugins can be run- the plugin directory and the plugin catalog entry.

»Plugin Directory

The plugin directory is a configuration option of Vault and can be specified in the configuration file. This setting specifies a directory in which all plugin binaries must live; this value cannot be a symbolic link. A plugin cannot be added to Vault unless it exists in the plugin directory. There is no default for this configuration option, and if it is not set, plugins cannot be added to Vault.

Warning: By default, Vault expects the plugin directory and files to be owned by the user running Vault. It also expects no write or execute permissions for group or others. Vault allows operators to specify the user and permissions of the plugin directory and binaries using parameters plugin_file_uid and plugin_file_permissions in config if an operator needs those to be different.
This check can be disabled via the environment variable VAULT_DISABLE_FILE_PERMISSIONS_CHECK.

»Plugin Catalog

The plugin catalog is Vault's list of approved plugins. The catalog is stored in Vault's barrier and can only be updated by a Vault user with sudo permissions. Upon adding a new plugin, the plugin name, SHA256 sum of the executable, and the command that should be used to run the plugin must be provided. The catalog will ensure the executable referenced in the command exists in the plugin directory. When added to the catalog, the plugin is not automatically executed, but becomes visible to backends and can be executed by them. For more information on the plugin catalog please see the Plugin Catalog API docs.

An example of plugin registration in current versions of Vault:

$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    secret \                  # type
    myplugin-database-plugin

Success! Registered plugin: myplugin-database-plugin
$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    secret \                  # type
    myplugin-database-plugin

Success! Registered plugin: myplugin-database-plugin

Vault versions prior to v0.10.4 lacked the vault plugin operator and the registration step for them is:

$ vault write sys/plugins/catalog/database/myplugin-database-plugin \
     sha256=<SHA256 Hex value of the plugin binary> \
     command="myplugin"

Success! Data written to: sys/plugins/catalog/database/myplugin-database-plugin
$ vault write sys/plugins/catalog/database/myplugin-database-plugin \
     sha256=<SHA256 Hex value of the plugin binary> \
     command="myplugin"

Success! Data written to: sys/plugins/catalog/database/myplugin-database-plugin

»Plugin Execution

When a backend wants to run a plugin, it first looks up the plugin, by name, in the catalog. It then checks the executable's SHA256 sum against the one configured in the plugin catalog. Finally Vault runs the command configured in the catalog, sending along the JWT formatted response wrapping token and mlock settings. Like Vault, plugins support the use of mlock when available.

Note: If Vault is configured with mlock enabled, then the Vault executable and each plugin executable in your plugins directory must be given the ability to use the mlock syscall.

»Plugin Upgrades

External plugins may be updated by registering and reloading them. More details on the upgrade procedure can be found in Upgrading Vault Plugins.

»Plugin Multiplexing

Database plugins can be made to implement plugin multiplexing, allowing a single plugin process to be used for multiple database connections. This single process, per database plugin, will be multiplexed across all Vault namespaces for mounts of this type. Multiplexing a plugin does not affect the current behavior of existing plugins.

To enable multiplexing, the plugin must be compiled with the ServeMultiplex function call from Vault's dbplugin package. At this time, there is no opt-out capability for plugins that implement multiplexing. To use a non-multiplexed plugin, run an older version of the plugin, i.e., the plugin calls the dbplugin.Serve function. More details on implementing plugin multiplexing can be found in Serving a Multiplexed Plugin.

»Troubleshooting

»Unrecognized remote plugin message

If the following error is encountered when enabling a plugin secret engine or auth method:

Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Verify whether the Vault process has mlock enabled, and if so, run the following command against the plugin binary:

$ sudo setcap cap_ipc_lock=+ep <plugin-binary>
$ sudo setcap cap_ipc_lock=+ep <plugin-binary>
github logoEdit this page
DocsAPILearnCommunityPrivacySecurityPress KitConsent Manager