.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode .. index:: single: Pegleg single: Security ======================================= Pegleg Secret Generation and Encryption ======================================= Pegleg is responsible for shepherding deployment manifest documents from their resting places in Git repositories to a consumable format that is ready for ingestion into Airship. This spec expands its responsibility to account for secure generation and encryption of secrets that are required within an Airship-based deployment. Links ===== The work to author and implement this spec will be tracked under this `Storyboard Story`_. Problem description =================== Airship supports the ability to identify secret information required for functioning deployments, such as passwords and keys; to ingest it into the site in a least-privilege-oriented fashion; and to encrypt it at rest within Deckhand. However, lifecycle management of the secrets outside the site should be made automatable and repeatable, to facilitate operational needs such as periodic password rotation, and to ensure that unencrypted secrets are only accessible by authorized individuals. Impacted components =================== The following Airship components will be impacted by this solution: #. Pegleg: enhanced to generate, rotate, encrypt, and decrypt secrets. #. Promenade: PKICatalog will move to Pegleg. #. Treasuremap: update site manifests to use new Catalogs. #. Airship-in-a-Bottle: update site manifests to use new Catalogs. Proposed change =============== PeglegManagedDocument --------------------- With this spec, the role of Pegleg grows from being a custodian of deployment manifests to additionally being the author of certain manifests. A new YAML schema will be created to describe these documents: ``pegleg/PeglegManagedDocument/v1``. Documents of this type will have one or both of the following data elements, although more may be added in the future: ``generated``, ``encrypted``. PeglegManagedDocuments serve as wrappers around other documents, and the wrapping serves to capture additional metadata that is necessary, but separate from the managed document proper. The managed document data will live in the ``data.managedDocument`` portion of a PeglegManagedDocument. If a PeglegManagedDocument is ``generated``, then its contents have been created by Pegleg, and it must include provenance information per this example:: schema: pegleg/PeglegManagedDocument/v1 metadata: name: matches-document-name schema: metadata/Document/v1 labels: matching: wrapped-doc layeringDefinition: abstract: false # Pegleg will initially support generation at site level only layer: site storagePolicy: cleartext data: generated: at: by: specifiedBy: repo: <...> reference: path: managedDocument: schema: metadata: storagePolicy: encrypted schema: data: If a PeglegManagedDocument is ``encrypted``, then its contents have been encrypted by Pegleg, and it must include provenance information per this example:: schema: pegleg/PeglegManagedDocument/v1 metadata: name: matches-document-name schema: metadata/Document/v1 labels: matching: wrapped-doc layeringDefinition: abstract: false layer: matching-wrapped-doc storagePolicy: cleartext data: encrypted: at: by: managedDocument: schema: metadata: storagePolicy: encrypted schema: data: A PeglegManagedDocument that is both generated via a Catalog, and encrypted (as specified by the catalog) will contain both ``generated`` and ``encrypted`` stanzas. Note that this ``encrypted`` key has a different purpose than the Deckhand ``storagePolicy: encrypted`` metadata, which indicates an *intent* for Deckhand to store a document encrypted at rest in the cluster. The two can be used together to ensure security, however: if a document is marked as ``storagePolicy: encrypted``, then automation may validate that it is only persisted (e.g. to a Git repository) if it is in fact encrypted within a PeglegManagedDocument. Note also that the Deckhand ``storagePolicy`` of the PeglegManagedDocument itself is always ``cleartext``, since its data stanza is not encrypted -- it only wraps a document that *is* ``storagePolicy: encrypted``. This should be implemented as a Pegleg lint rule. Document Generation ------------------- Document generation will follow the pattern established by Promenade's PKICatalog pattern. In fact, PKICatalog management responsibility will move to Pegleg as part of this effort. The types of documents that are expected to be generated are certificates and keys, which are defined via PKICatalog documents now, and passphrases, which will be defined via a new ``pegleg/PassphraseCatalog/v1`` document. Longer-term, these specifications may be combined, or split further (into a CertificateCatalog and KeypairCatalog), but this is not needed in the initial implementation in Pegleg. A collection of manifests may define more than one of each of these secret catalog documents if desired. The documents generated via PKICatalog and PassphraseCatalog will follow the PeglegManagedDocument schema above; note that this is a change to existing PKICatalog behavior. The PKICatalog schema and associated code should be copied to Pegleg (and renamed to ``pegleg/PKICatalog/v1``), and during a transition period the old and new PKICatalog implementations will exist side-by-side with slightly different semantics. Promenade's PKICatalog can be removed once all deployment manifests have been updated to use the new one. Pegleg will place generated document files in ``/secrets/passphrases/``, ``/secrets/certificates``, or ``/secrets/keypairs`` as appropriate: * The generated filenames for passphrases will follow the pattern ``.yaml``. * The generated filenames for certificate authorities will follow the pattern ``_ca.yaml``. * The generated filenames for certificates will follow the pattern ``__certificate.yaml``. * The generated filenames for certificate keys will follow the pattern ``__key.yaml``. * The generated filenames for keypairs will follow the pattern ``.yaml``. * Dashes in the document names will be converted to underscores for consistency. A PassphraseCatalog will capture the following example structure:: schema: pegleg/PassphraseCatalog/v1 metadata: schema: metadata/Document/v1 name: cluster-passphrases layeringDefinition: abstract: false layer: site storagePolicy: cleartext data: passphrases: - document_name: osh-nova-password description: Service password for Nova encrypted: true - document_name: osh-nova-oslo-db-password description: Database password for Nova encrypted: true length: 12 The nonobvious bits of the document described above are: * ``encrypted`` is optional, and denotes whether the generated PeglegManagedDocument will be ``encrypted``, as well as whether the wrapped document will have ``storagePolicy: encrypted`` or ``storagePolicy: cleartext`` metadata. If absent, ``encrypted`` defaults to ``true``. * ``document_name`` is required, and is used to create the filename of the generated PeglegManagedDocument manifest, and the ``metadata.name`` of the wrapped ``deckhand/Passphrase/v1`` document. In both cases, Pegleg will replace dashes in the ``document_name`` with underscores. * ``length`` is optional, and denotes the length in characters of the generated cleartext passphrase data. If absent, ``length`` defaults to ``24``. Note that with this length and the selected character set there will be less than 8x10^48 probability of getting a new passphrase that is identical to the previous passphrase. This is sufficiently random to ensure no duplication of rotated passphrases in practice. * ``description`` is optional. The ``encrypted`` key will be added to the PKICatalog schema, and adds the same semantics to PKICatalog-based generation as are described above for PassphraseCatalog. Pegleg CLI Changes ------------------ The Pegleg CLI interface will be extended as follows. These commands will create PeglegManagedDocument manifests in the local repository. Committing and pushing the changes will be left to the operator or to script-based automation. For the CLI commands below which encrypt or decrypt secrets, an environment variable (e.g. ``PEGLEG_PASSPHRASE`` will be use to capture the master passphrase to use. ``pegleg site secrets rotate`` will use a second variable (e.g. ``PEGLEG_PREVIOUS_PASSPHRASE``) to hold the key/passphrase being rotated out. The contents of these keys/passphrases are not generated by Pegleg, but are created externally and set by a deployment engineer or tooling. A configurable minimum length (default 24) for master passphrases will be checked by all CLI commands which use the passphrase. All other criteria around passphrase strength are assumed to be enforced elsewhere, as it is an external secret that is consumed/used by Pegleg. ``pegleg site secrets generate passphrases``: Generate passphrases according to all PassphraseCatalog documents in the site. Note that regenerating passphrases can be accomplished simply by re-running ``pegleg site secrets generate passphrases``. ``pegleg generate passphrase``: A standalone version of passphrase generation. This generates a single passphrase based on the default length, character set, and implementation described above, and outputs it to the console. The PassphraseCatalog is not involved in this operation. This command is suitable for generation of a highly-secure Pegleg master passphrase. ``pegleg site secrets generate pki``: Generate certificates and keys according to all PKICatalog documents in the site. Note that regenerating certificates can be accomplished simply by re-running ``pegleg site secrets generate pki``. ``pegleg site secrets generate``: Combines the two commands above. May be expanded in the future to include other manifest generation activities. ``pegleg site bootstrap``: For now, a synonym for ``pegleg site secrets generate``, and may be expanded in the future to include other bootstrapping activities. ``pegleg site secrets encrypt``: Encrypt all site documents which have ``metadata.storagePolicy: encrypted``, and which are not already encrypted within a wrapping PeglegManagedDocument. Note that the ``pegleg site secrets generate`` commands encrypt generated secrets as specified, so ``pegleg site secrets encrypt`` is intended mainly for external-facing secrets which a deployment engineer brings to the site manifests. The output PeglegManagedDocument will be written back to the filename that served as its source. ``pegleg site secrets decrypt ``: Decrypt a specific PeglegManagedDocument manifest, unwrapping it and outputting the cleartext original document YAML to standard output. This is intended to be used when an authorized deployment engineer needs to determine a particular cleartext secret for a specific operational purpose. ``pegleg site secrets rotate passphrases``: This action re-encrypts encrypted passphrases with a new key/passphrase, and it takes the previously-used key and a new key as input. It accomplishes its task via two activities: * For encrypted passphrases that were imported from outside of Pegleg (i.e. PeglegManagedDocuments which lack the ``generated`` stanza), decrypt them with the old key (in-memory), re-encrypt them with the new key, and output the results. * Perform a fresh ``pegleg site secrets generate passphrases`` process using the new key. This will replace all ``generated`` passphrases with new secret values for added security. There is an assumption here that the only actors that need to know generated secrets are the services within the Airship-managed cluster, not external services or deployment engineers, except perhaps for point-in-time troubleshooting or operational exercises. Similar functionality for rotating certificates (which is expected to have a different cadence than passphrase rotation, typically) will be added in the future. Driving deployment of a site directly via Pegleg is follow-on functionality which will collect site documents, use them to create the ``genesis.sh`` script, and then interact directly with Shipyard to drive deployments. Its details are beyond the scope of this spec, but when implemented, it should decrypt documents wrapped by applicable PeglegManagedDocuments at the lst responsible moment, and take care not to write, log, or stdout them to disk as cleartext. Note that existing ``pegleg collect`` functionality should **not** be changed to decrypt encrypted secrets; this is because it writes its output to disk. If ``pegleg collect`` is called, at this point in time, the PeglegManagedDocuments will be written (encrypted) to disk. To enable special case full site secret decryption, a ``--force-decrypt`` flag will be added to ``pegleg collect`` to do this under controlled circumstances, and to help bridge the gap with existing CICD pipelines until Pegleg-driven site deployment is in place. It will leverage the ``PEGLEG_PASSPHRASE`` variable described above. Secret Generation ----------------- The ``rstr`` library should be invoked to generate secrets of the appropriate length and character set. This library uses the ``os.urandom()`` function, which in turn leverages ``/dev/urandom`` on Linux, and it is suitable for cryptographic purposes. Characters in generated secrets will be evenly distributed across lower- and upper-case letters, digits, and punctuation in !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. Note this is equivalent to the union of Python string.ascii_letters, string.digits, and string.punctuation. Secret Encryption ----------------- The Python ``cryptography`` library has been chosen to implement the encryption and decryption of secrets within Pegleg. ``cryptography`` aims to be the standard cryptographic approach for Python, and takes pains to make it difficult to do encryption poorly (via its ``recipes`` layer), while still allowing access to the algorithmic details when truly needed (via its ``hazmat`` layer). ``cryptography`` is actively maintained and is the target encryption library for OpenStack as well. The ``cryptography.fernet`` module will be used for symmetric encryption. It uses AES with a 128-bit key for encryption, and HMAC using SHA256 for encryption. Fernet requires as input a URL-safe, base64-encoded 32-byte encryption key, which will be derived from the master passphrase passed into Pegleg via ``PEGLEG_PASSPHRASE`` as described above. The example for password-based encryption from the `Fernet documentation`_ should be followed as a guide. The ``salt`` to be used in key derivation will be configurable, and will be set to a fixed value within a built Pegleg container via an environment variable passed into the Pegleg Dockerfile. This will allow the salt to be different on an operator-by-operator basis. The ``cryptography.exceptions.InvalidSignature`` exception is thrown by ``cryptography`` when an attempt is made to decrypt a message with a key that is different than the one used to encrypt a message, i.e., when the user has supplied an incorrect phassphrase. It should be handled gracefully by Pegleg, resulting in an informative message back to the user. Security impact =============== These changes will result in a system that handles site secrets in a highly secure manner, in the face of multiple roles and day 2 operational needs. Performance impact ================== Performance impact to existing flows will be minimal. Pegleg will need to additionally decrypt secrets as part of site deployment, but this will be an efficient operation performed once per deployment. Alternatives ============ The Python ``secrets`` library presents a convenient interface for generating random strings. However, it was introduced in Python 3.6, and it would be limiting to introduce this constraint on Airship CICD pipelines. The ``strgen`` library presents an even more convenient interface for generating pseudo-random strings; however, it leverages the Python ``random`` library, which is unsuitably random for cryptographic purposes. Deckhand already supports a ``storagePolicy`` element which indicates whether whether Deckhand will persist document data in an encrypted state, and this flag could have been re-used by Pegleg to indicate whether a secret is (or should be) encrypted. However, "should this data be encrypted" is a fundamentally different question than "is this data encrypted now", and additional metadata-esque parameters (``generated``, ``generatedLength``) were desired as well, so this proposal adds ``data.encrypted`` to indicate the point-in-time encryption status. ``storagePolicy`` is still valuable in this context to make sure everything that *should* be encrypted *is*, prior to performing actions with it (e.g. Git commits). The ``PyCrypto`` library is a popular solution for encryption in Python; however, it is no longer actively maintained. Following the lead of OpenStack and others, we opted instead for the ``cryptography`` library. This proposed implementation writes the output of generation/encryption events back to the same source files from which the original data came. This is a destructive operation; however, it wasn't evident that it is problematic in any anticipated workflow. In addition, it sidesteps challenges around naming of generated files, and cleanup of original files. Implementation ============== Please refer to the `Storyboard Story`_ for implementation planning information. Dependencies ============ This work should be based on the patchset to add `Git branch and revision support`_ to Pegleg, if it is not merged by the time implementation begins. This patchset alters the CLI interface and Git repository management code, and basing on it will avoid future refactoring. References ========== .. _Storyboard Story: https://storyboard.openstack.org/#!/story/2003708 .. _Git branch and revision support: https://review.openstack.org/#/c/577886/ .. _Fernet documentation: https://cryptography.io/en/latest/fernet/