Pegleg Secret Generation and Encryption¶
Pegleg is responsible for shepherding deployment manifest documents from their resting places in Git repositories to a consumable format that is ready for ingestion into Airship. This spec expands its responsibility to account for secure generation and encryption of secrets that are required within an Airship-based deployment.
Links¶
The work to author and implement this spec will be tracked under this Storyboard Story.
Problem description¶
Airship supports the ability to identify secret information required for functioning deployments, such as passwords and keys; to ingest it into the site in a least-privilege-oriented fashion; and to encrypt it at rest within Deckhand. However, lifecycle management of the secrets outside the site should be made automatable and repeatable, to facilitate operational needs such as periodic password rotation, and to ensure that unencrypted secrets are only accessible by authorized individuals.
Impacted components¶
The following Airship components will be impacted by this solution:
- Pegleg: enhanced to generate, rotate, encrypt, and decrypt secrets.
- Promenade: PKICatalog will move to Pegleg.
- Treasuremap: update site manifests to use new Catalogs.
- Airship-in-a-Bottle: update site manifests to use new Catalogs.
Proposed change¶
PeglegManagedDocument¶
With this spec, the role of Pegleg grows from being a custodian of deployment
manifests to additionally being the author of certain manifests. A new YAML
schema will be created to describe these documents:
pegleg/PeglegManagedDocument/v1
.
Documents of this type will have one or both of the following data elements,
although more may be added in the future: generated
, encrypted
.
PeglegManagedDocuments serve as wrappers around other documents, and the
wrapping serves to capture additional metadata that is necessary, but
separate from the managed document proper.
The managed document data will live in the data.managedDocument
portion
of a PeglegManagedDocument.
If a PeglegManagedDocument is generated
, then its contents have been
created by Pegleg, and it must include provenance information per this
example:
schema: pegleg/PeglegManagedDocument/v1
metadata:
name: matches-document-name
schema: metadata/Document/v1
labels:
matching: wrapped-doc
layeringDefinition:
abstract: false
# Pegleg will initially support generation at site level only
layer: site
storagePolicy: cleartext
data:
generated:
at: <timestamp>
by: <author>
specifiedBy:
repo: <...>
reference: <git ref-head or similar>
path: <PKICatalog/PassphraseCatalog details>
managedDocument:
schema: <as appropriate for wrapped document>
metadata:
storagePolicy: encrypted
schema: <as appropriate for wrapped document>
<metadata from parent PeglegManagedDocument>
<any other metadata as appropriate>
data: <generated data>
If a PeglegManagedDocument is encrypted
, then its contents have been
encrypted by Pegleg, and it must include provenance information per this
example:
schema: pegleg/PeglegManagedDocument/v1
metadata:
name: matches-document-name
schema: metadata/Document/v1
labels:
matching: wrapped-doc
layeringDefinition:
abstract: false
layer: matching-wrapped-doc
storagePolicy: cleartext
data:
encrypted:
at: <timestamp>
by: <author>
managedDocument:
schema: <as appropriate for wrapped document>
metadata:
storagePolicy: encrypted
schema: <as appropriate for wrapped document>
<metadata from parent PeglegManagedDocument>
<any other metadata as appropriate>
data: <encrypted string blob>
A PeglegManagedDocument that is both generated via a Catalog, and encrypted
(as specified by the catalog) will contain both generated
and
encrypted
stanzas.
Note that this encrypted
key has a different purpose than the Deckhand
storagePolicy: encrypted
metadata, which indicates an intent for Deckhand
to store a document encrypted at rest in the cluster. The two can be used
together to ensure security, however: if a document is marked as
storagePolicy: encrypted
, then automation may validate that it is only
persisted (e.g. to a Git repository) if it is in fact encrypted within
a PeglegManagedDocument.
Note also that the Deckhand storagePolicy
of the PeglegManagedDocument
itself is always cleartext
, since its data stanza is not encrypted – it
only wraps a document that is storagePolicy: encrypted
.
This should be implemented as a Pegleg lint rule.
Document Generation¶
Document generation will follow the pattern established by Promenade’s
PKICatalog pattern. In fact, PKICatalog management responsibility will move
to Pegleg as part of this effort. The types of documents that are expected
to be generated are certificates and keys, which are defined via PKICatalog
documents now, and passphrases, which will be defined via a new
pegleg/PassphraseCatalog/v1
document. Longer-term, these specifications
may be combined, or split further (into a CertificateCatalog and
KeypairCatalog), but this is not needed in the initial implementation in
Pegleg. A collection of manifests
may define more than one of each of these secret catalog documents if desired.
The documents generated via PKICatalog and PassphraseCatalog will follow the
PeglegManagedDocument schema above; note that this is a change to existing
PKICatalog behavior. The PKICatalog schema and associated code should be
copied to Pegleg (and renamed to pegleg/PKICatalog/v1
), and during a
transition period the old and new PKICatalog implementations will exist
side-by-side with slightly different semantics. Promenade’s PKICatalog can
be removed once all deployment manifests have been updated to use the new one.
Pegleg will place generated document files in <site>/secrets/passphrases/
,
<site>/secrets/certificates
, or <site>/secrets/keypairs
as appropriate:
- The generated filenames for passphrases will follow the pattern
<passphrase-doc-name>.yaml
. - The generated filenames for certificate authorities will follow the pattern
<ca-name>_ca.yaml
. - The generated filenames for certificates will follow the pattern
<ca-name>_<certificate-doc-name>_certificate.yaml
. - The generated filenames for certificate keys will follow the pattern
<ca-name>_<certificate-doc-name>_key.yaml
. - The generated filenames for keypairs will follow the pattern
<keypair-doc-name>.yaml
. - Dashes in the document names will be converted to underscores for consistency.
A PassphraseCatalog will capture the following example structure:
schema: pegleg/PassphraseCatalog/v1
metadata:
schema: metadata/Document/v1
name: cluster-passphrases
layeringDefinition:
abstract: false
layer: site
storagePolicy: cleartext
data:
passphrases:
- document_name: osh-nova-password
description: Service password for Nova
encrypted: true
- document_name: osh-nova-oslo-db-password
description: Database password for Nova
encrypted: true
length: 12
The nonobvious bits of the document described above are:
encrypted
is optional, and denotes whether the generated PeglegManagedDocument will beencrypted
, as well as whether the wrapped document will havestoragePolicy: encrypted
orstoragePolicy: cleartext
metadata. If absent,encrypted
defaults totrue
.document_name
is required, and is used to create the filename of the generated PeglegManagedDocument manifest, and themetadata.name
of the wrappeddeckhand/Passphrase/v1
document. In both cases, Pegleg will replace dashes in thedocument_name
with underscores.length
is optional, and denotes the length in characters of the generated cleartext passphrase data. If absent,length
defaults to24
. Note that with this length and the selected character set there will be less than 8x10^48 probability of getting a new passphrase that is identical to the previous passphrase. This is sufficiently random to ensure no duplication of rotated passphrases in practice.description
is optional.
The encrypted
key will be added to the PKICatalog schema, and adds the same
semantics to PKICatalog-based generation as are described above for
PassphraseCatalog.
Pegleg CLI Changes¶
The Pegleg CLI interface will be extended as follows. These commands will create PeglegManagedDocument manifests in the local repository. Committing and pushing the changes will be left to the operator or to script-based automation.
For the CLI commands below which encrypt or decrypt secrets, an environment
variable (e.g. PEGLEG_PASSPHRASE
will be use to capture the master
passphrase to use. pegleg site secrets rotate
will use a second variable
(e.g. PEGLEG_PREVIOUS_PASSPHRASE
) to hold the key/passphrase being rotated
out. The contents of these keys/passphrases are not generated by Pegleg,
but are created externally and set by a deployment engineer or tooling.
A configurable minimum length (default 24) for master passphrases will
be checked by all CLI commands which use the passphrase. All other criteria
around passphrase strength are assumed to be enforced elsewhere, as it is an
external secret that is consumed/used by Pegleg.
pegleg site secrets generate passphrases
: Generate passphrases according to
all PassphraseCatalog documents in the site.
Note that regenerating passphrases can be accomplished
simply by re-running pegleg site secrets generate passphrases
.
pegleg generate passphrase
: A standalone version of passphrase generation.
This generates a single passphrase based on the default length, character set,
and implementation described above, and outputs it to the console. The
PassphraseCatalog is not involved in this operation. This command is suitable
for generation of a highly-secure Pegleg master passphrase.
pegleg site secrets generate pki
: Generate certificates and keys according
to all PKICatalog documents in the site.
Note that regenerating certificates can be accomplished
simply by re-running pegleg site secrets generate pki
.
pegleg site secrets generate
: Combines the two commands above.
May be expanded in the future to include other manifest generation activities.
pegleg site bootstrap
: For now, a synonym for
pegleg site secrets generate
,
and may be expanded in the future to include other bootstrapping activities.
pegleg site secrets encrypt
: Encrypt all site documents which have
metadata.storagePolicy: encrypted
, and which are not already encrypted
within a wrapping PeglegManagedDocument. Note that the
pegleg site secrets generate
commands encrypt generated secrets as
specified, so pegleg site secrets encrypt
is intended mainly for
external-facing secrets which a deployment engineer brings to the site
manifests.
The output PeglegManagedDocument will be written back to the filename that
served as its source.
pegleg site secrets decrypt <document YAML file>
: Decrypt a specific
PeglegManagedDocument manifest, unwrapping it and outputting the cleartext
original document YAML to standard output. This is intended to be used when
an authorized deployment engineer needs to determine a particular cleartext
secret for a specific operational purpose.
pegleg site secrets rotate passphrases
: This action re-encrypts
encrypted passphrases with a new key/passphrase, and it takes the
previously-used key and a new key as input. It accomplishes its task via
two activities:
- For encrypted passphrases that were imported from outside of Pegleg
(i.e. PeglegManagedDocuments which lack the
generated
stanza), decrypt them with the old key (in-memory), re-encrypt them with the new key, and output the results. - Perform a fresh
pegleg site secrets generate passphrases
process using the new key. This will replace allgenerated
passphrases with new secret values for added security. There is an assumption here that the only actors that need to know generated secrets are the services within the Airship-managed cluster, not external services or deployment engineers, except perhaps for point-in-time troubleshooting or operational exercises.
Similar functionality for rotating certificates (which is expected to have a different cadence than passphrase rotation, typically) will be added in the future.
Driving deployment of a site directly via Pegleg is follow-on functionality
which will
collect site documents, use them to create the genesis.sh
script, and then
interact directly with Shipyard to drive deployments. Its details are beyond
the scope of this spec, but when implemented, it should decrypt documents
wrapped by applicable PeglegManagedDocuments at the lst responsible moment,
and take care not to write, log, or stdout them to disk as cleartext.
Note that existing pegleg collect
functionality should not be changed
to decrypt encrypted secrets; this is because it writes its output to disk.
If pegleg collect
is called, at this point in time, the
PeglegManagedDocuments will be written (encrypted) to disk.
To enable special case full site secret decryption, a --force-decrypt
flag
will be added to pegleg collect
to do this under controlled circumstances,
and to help bridge the gap with existing CICD pipelines until Pegleg-driven
site deployment is in place. It will leverage the PEGLEG_PASSPHRASE
variable described above.
Secret Generation¶
The rstr
library should be invoked to generate secrets of the
appropriate length and character set.
This library uses the os.urandom()
function,
which in turn leverages /dev/urandom
on Linux,
and it is suitable for cryptographic purposes.
Characters in generated secrets will be evenly distributed across lower- and upper-case letters, digits, and punctuation in !”#$%&’()*+,-./:;<=>?@[]^_`{|}~. Note this is equivalent to the union of Python string.ascii_letters, string.digits, and string.punctuation.
Secret Encryption¶
The Python cryptography
library has been chosen to implement the
encryption and decryption of secrets within Pegleg. cryptography
aims to be the standard cryptographic approach for Python, and takes
pains to make it difficult to do encryption poorly (via its recipes
layer), while still allowing access to the algorithmic details when
truly needed (via its hazmat
layer). cryptography
is actively
maintained and is the target encryption library for OpenStack as well.
The cryptography.fernet
module will be used for symmetric encryption.
It uses AES with a 128-bit key for encryption, and HMAC using SHA256
for encryption.
Fernet requires as input a URL-safe, base64-encoded 32-byte encryption key,
which will be derived from the master passphrase passed into Pegleg via
PEGLEG_PASSPHRASE
as described above.
The example for password-based encryption from the Fernet documentation
should be followed as a guide. The salt
to be used in key derivation
will be configurable, and will be set to a fixed value within a built
Pegleg container via an environment variable passed into the Pegleg
Dockerfile. This will allow the salt to be different on an
operator-by-operator basis.
The cryptography.exceptions.InvalidSignature
exception is thrown by
cryptography
when an attempt is made to decrypt a message with a key that
is different than the one used to encrypt a message, i.e., when the user has
supplied an incorrect phassphrase. It should be handled gracefully by Pegleg,
resulting in an informative message back to the user.
Security impact¶
These changes will result in a system that handles site secrets in a highly secure manner, in the face of multiple roles and day 2 operational needs.
Performance impact¶
Performance impact to existing flows will be minimal. Pegleg will need to additionally decrypt secrets as part of site deployment, but this will be an efficient operation performed once per deployment.
Alternatives¶
The Python secrets
library presents a convenient interface for generating
random strings. However, it was introduced in Python 3.6, and it would be
limiting to introduce this constraint on Airship CICD pipelines.
The strgen
library presents an even more convenient interface for
generating pseudo-random strings; however, it leverages the Python random
library, which is unsuitably random for cryptographic purposes.
Deckhand already supports a storagePolicy
element which indicates whether
whether Deckhand will persist document data in an encrypted state, and this
flag could have been re-used by Pegleg to indicate whether a secret is
(or should be) encrypted. However, “should this data be encrypted” is a
fundamentally different question than “is this data encrypted now”, and
additional metadata-esque parameters (generated
, generatedLength
)
were desired as well, so this proposal adds data.encrypted
to indicate
the point-in-time encryption status. storagePolicy
is still valuable
in this context to make sure everything that should be encrypted is,
prior to performing actions with it (e.g. Git commits).
The PyCrypto
library is a popular solution for encryption in Python;
however, it is no longer actively maintained. Following the lead of OpenStack
and others, we opted instead for the cryptography
library.
This proposed implementation writes the output of generation/encryption events back to the same source files from which the original data came. This is a destructive operation; however, it wasn’t evident that it is problematic in any anticipated workflow. In addition, it sidesteps challenges around naming of generated files, and cleanup of original files.
Implementation¶
Please refer to the Storyboard Story for implementation planning information.
Dependencies¶
This work should be based on the patchset to add Git branch and revision support to Pegleg, if it is not merged by the time implementation begins. This patchset alters the CLI interface and Git repository management code, and basing on it will avoid future refactoring.