Document Layering¶
Introduction¶
Layering provides a restricted data inheritance model intended to help reduce duplication in configuration. With layering, child documents can inherit data from parent documents. Through Layering Actions, child documents can control exactly what they inherit from their parent. Document layering, conceptually speaking, works much like class inheritance: A child class inherits all variables and methods from its parent, but can elect to override its parent’s functionality.
Goals behind layering include:
- model site deployment data hierarchically
- lessen data duplication across site layers (as well as other conceptual layers)
Document Abstraction¶
Layering works with Document Abstraction: child documents can inherit from abstract as well as concrete parent documents.
Pre-Conditions¶
A document only has one parent, but its parent is computed dynamically using the Parent Selection algorithm. That is, the notion of “multiple inheritance” does not apply to document layering.
Documents with different schema
values are never layered together (see the
Document Substitution section if you need to combine data from multiple types of
documents).
Document layering requires a LayeringPolicy to exist in the revision whose documents will be layered together (rendered). An error will be issued otherwise.
Terminology¶
Note
Whether a layer is “lower” or “higher” has entirely to do with its order of
initialization in a layerOrder
and, by extension, its precedence in the
Parent Selection algorithm described below.
- Layer - A position in a hierarchy used to control Parent Selection by
the Algorithm. It can be likened to a position in an
inheritance hierarchy, where
object
in Python can be likened to the highest layer in alayerOrder
in Deckhand and a leaf class can be likened to the lowest layer in alayerOrder
. - Child - Meaningful only in a parent-child document relationship. A document with a lower layer (but higher priority) than its parent, determined using using Parent Selection.
- Parent - Meaningful only in a parent-child document relationship. A document with a higher layer (but lower priority) than its child.
- Layering Policy - A control document that defines
the strict
layerOrder
in which documents are layered together. See LayeringPolicy documentation for more information. - Layer Order (
layerOrder
) - Corresponds to thedata.layerOrder
of the LayeringPolicy document. Establishes the layering hierarchy for a set of layers in the system. - Layering Definition (
layeringDefinition
) - Metadata in each document for controlling the following:layer
: the document layer itselfparentSelector
: Parent Selectionabstract
: Document Abstractionactions
: Layering Actions
- Parent Selector (
parentSelector
) - Key-value pairs or labels for identifying the document’s parent. Note that these key-value pairs are not unique and that multiple documents can use them. All the key-value pairs in theparentSelector
must be found among the target parent’smetadata.labels
: this means that theparentSelector
key-value pairs must be a subset of the target parent’smetadata.labels
key-value pairs. See Parent Selection for further details. - Layering Actions (
actions
) - A list of actions that control what data are inherited from the parent by the child. See Layering Actions for further details.
Algorithm¶
Layering is applied at the bottommost layer of the layerOrder
first and
at the topmost layer of the layerOrder
last, such that the “base” layers
are processed first and the “leaf” layers are processed last. For each
layer in the layerOrder
, the documents that correspond to that layer
are retrieved. For each document retrieved, the layerOrder
hierarchy
is resolved using Parent Selection to identify the parent document.
Finally, the current document is layered with its parent using
Layering Actions.
After layering is complete, the Document Substitution algorithm is applied to the current document, if applicable.
Layering Configuration¶
Layering is configured in 2 places:
- The
LayeringPolicy
control document (described in LayeringPolicy), which defines the valid layers and their order of precedence. - In the
metadata.layeringDefinition
section of normal (metadata.schema=metadata/Document/v1
) documents. For more information about document structure, reference Document Format.
An example layeringDefinition
may look like:
layeringDefinition:
# Controls whether the document is abstract or concrete.
abstract: true
# A layer in the ``layerOrder``. Must be valid or else an error is raised.
layer: region
# Key-value pairs or labels for identifying the document's parent.
parentSelector:
required_key_a: required_label_a
required_key_b: required_label_b
# Actions which specify which data to add to the child document.
actions:
- method: merge
path: .path.to.merge.into.parent
- method: delete
path: .path.to.delete
Layering Actions¶
Introduction¶
Layering actions allow child documents to modify data that is inherited from
the parent. What if the child document should only inherit some of the parent
data? No problem. A merge action can be performed, followed by delete
and replace
actions to trim down on what should be inherited.
Each layer action consists of an action
and a path
. Whenever any
action is specified, all the parent data is automatically inherited by the
child document. The path
specifies which data from the child document to
prioritize over that of the parent document. Stated differently, all data
from the parent is considered while only the child data at path
is
considered during an action. However, whenever a conflict occurs during an
action, the child data takes priority over that of the parent.
Layering actions are queued – meaning that if a merge
is
specified before a replace
then the merge
will necessarily be
applied before the replace
. For example, a merge
followed by a
replace
is not necessarily the same as a replace
followed by a
merge
.
Layering actions can be applied to primitives, lists and dictionaries alike.
Action Types¶
Supported actions are:
merge
- “deep” merge child data and parent data into the childdata
, at the specified JSONPathNote
For conflicts between the child and parent data, the child document’s data is always prioritized. No other conflict resolution strategy for this action currently exists.
merge
behavior depends upon the data types getting merged. For objects and lists, Deckhand uses JSONPath resolution to retrieve data from those entities, after which Deckhand applies merge strategies (see below) to combine merge child and parent data into the child document’sdata
section.Merge Strategies
Deckhand applies the following merge strategies for each data type:
- object: “Deep-merge” child and parent data together; conflicts are resolved by prioritizing child data over parent data. “Deep-merge” means recursively combining data for each key-value pair in both objects.
- array: The merge strategy involves:
- When using an index in the action
path
(e.g.a[0]
):- Copying the parent array into the child’s
data
section at the specified JSONPath. - Appending each child entry in the original child array into the parent
array. This behavior is synonymous with the
extend
list function in Python.
- Copying the parent array into the child’s
- When not using an index in the action
path
(e.g.a
):- The child’s array replaces the parent’s array.
- When using an index in the action
- primitives: Includes all other data types, except for
null
. In this case JSONPath resolution is impossible, so child data is prioritized over that of the parent.
Examples
Given:
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
When:
Merge Path: ``.``
Then:
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}`` All data from parent is automatically considered, all data from child is considered due to ``.`` (selects everything), then both merged.
When:
Merge Path: ``.a``
Then:
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}`` All data from parent is automatically considered, all data from child at ``.a`` is considered, then both merged.
When:
Merge Path: ``.b``
Then:
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}`` All data from parent is automatically considered, all data from child at ``.b`` is considered, then both merged.
When:
Merge Path: ``.c``
Then:
Error raised (``.c`` missing in child).
replace
- overwrite existing data with child data at the specified JSONPath.Examples
Given:
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
When:
Replace Path: ``.``
Then:
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` All data from parent is automatically considered, but is replaced by all data from child at ``.`` (selects everything), so replaces everything in parent.
When:
Replace Path: ``.a``
Then:
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}`` All data from parent is automatically considered, but is replaced by all data from child at ``.a``, so replaces all parent data at ``.a``.
When:
Replace Path: ``.b``
Then:
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}`` All data from parent is automatically considered, but is replaced by all data from child at ``.b``, so replaces all parent data at ``.b``. While ``.b`` isn't in the parent, it only needs to exist in the child. In this case, something (from the child) replaces nothing (from the parent).
When:
Replace Path: ``.c``
Then:
Error raised (``.c`` missing in child).
delete
- remove the existing data at the specified JSONPath.Examples
Given:
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
When:
Delete Path: ``.``
Then:
Rendered Data: ``{}`` Note that deletion of everything results in an empty dictionary by default.
When:
Delete Path: ``.a``
Then:
Rendered Data: ``{'c': 9}`` All data from Parent Data at ``.a`` was deleted, rest copied over.
When:
Delete Path: ``.c``
Then:
Rendered Data: ``{'a': {'x': 1, 'y': 2}}`` All data from Parent Data at ``.c`` was deleted, rest copied over.
When:
Replace Path: ``.b``
Then:
Error raised (``.b`` missing in child).
After actions are applied for a given layer, substitutions are applied (see the Document Substitution section for details).
Parent Selection¶
Parent selection is performed dynamically. Unlike Document Substitution,
parent selection does not target a specific document using schema
and
name
identifiers. Rather, parent selection respects the layerOrder
,
selecting the highest precedence parent in accordance with the algorithm that
follows. This allows flexibility in parent selection: if a document’s immediate
parent is removed in a revision, then, if applicable, the grandparent (in the
previous revision) can become the document’s parent (in the latest revision).
Selection of document parents is controlled by the parentSelector
field and
works as follows:
A given document,
C
, that specifies aparentSelector
, will have exactly one parent,P
. If comparing layering with inheritance, layering, then, does not allow multi-inheritance.Both
C
andP
must have the sameschema
.Both
C
andP
should have differentmetadata.name
values except in the case of Document Replacement.Document
P
will be the highest-precedence document whosemetadata.labels
are a superset of document C’sparentSelector
. Where:Highest precedence means that
P
belongs to the lowest layer defined in thelayerOrder
list from theLayeringPolicy
which is at least one level higher than the layer forC
. For example, ifC
has layersite
, then its parentP
must at least have layertype
or above in the followinglayerOrder
:--- ... layerOrder: - global # Highest layer - type - site # Lowest layer
Superset means that
P
at least has all the labels in itsmetadata.labels
that childC
references via itsparentSelector
. In other words, parentP
can have more labels thanC
uses to reference it, butC
must at least have one matching label in itsparentSelector
withP
.
Deckhand will select
P
if it belongs to the highest-precedence layer. For example, ifC
belongs to layersite
,P
belongs to layertype
, andG
belongs to layerglobal
, then Deckhand will useP
as the parent forC
. IfP
is non-existent, thenG
will be selected instead.
For example, consider the following sample documents:
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: global-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: region-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: region
parentSelector:
key1: value1
actions:
- method: replace
path: .a
data:
a:
z: 3
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: site-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
data:
b: 4
When rendering, the parent chosen for site-1234
will be region-1234
,
since it is the highest precedence document that matches the label selector
defined by parentSelector
, and the parent chosen for region-1234
will be
global-1234
for the same reason. The rendered result for site-1234
would
be:
---
schema: example/Kind/v1
metadata:
name: site-1234
data:
a:
z: 3
b: 4
If region-1234
were later removed, then the parent chosen for site-1234
would become global-1234
, and the rendered result would become:
---
schema: example/Kind/v1
metadata:
name: site-1234
data:
a:
x: 1
y: 2
b: 4