Testing Guidelines

This document lays out several guidelines to ensure quality and consistency throughout airshipctl's test bed.

Testing packages

The airshipctl project uses the testify library, a thin wrapper around Go's builtin testing package. The testify package provides the following packages:

Test coverage

Tests should cover at least 80% of the codebase. Anything less will cause the CI gates to fail. A developer should assert that their code meets this criteria before submitting a change. This check can be performed with one of the following make targets:

# Runs all unit tests, then computes and reports the coverage
make cover

Good practice is to assert that the changed packages have not decreased in coverage. The coverage check can be run for a specific package with a command such as the following.

make cover PKG=./pkg/foo

Additional testing should be done to ensure that the proposed change meets an expected level of quality. These tests include:

# Tidy, to ensure go.mod is up to date
make tidy

# Lint, to ensure code meets linting requirements
make lint

# Update-golden, to ensure the golden test data reflects the current test cases
make update-golden

When the above are done, if you would like to perform the same dockerized container testing as the CI gates you can do so via:

make docker-image-test-suite

NOTE: If test cases are deleted you must first run make update-golden, and commit your changes prior to running the docker-image-test-suite make target. Otherwise the docker-image-test-suite make target (and the CI job) will fail.

Test directory structure

Test files end in _test.go, and sit next to the tested file. For example, airshipctl/pkg/foo/foo.go should be tested by airshipctl/pkg/foo/foo_test.go. A test's package name should also end in _test, unless that file intends to test unexported fields and method, at which point it should be in the package under test.

Go will ignore any files stored in a directory called testdata, therefore all non-Go test files (such as expected output or example input) should be stored there.

Any mocks for a package should be stored in a sub-package ending in mocks. Each mocked struct should have its own file, where the filename describes the struct, i.e. a file containing a mocked Fooer should be stored at mocks/fooer.go. Mocked files can be either handwritten or generated via mockery. The mockery tool can generate files in this fashion with the following command.

mockery -all -case snake

An example file structure might look something like the following.

airshipctl/pkg/foo
├── foo.go
├── foo_test.go
├── mocks
│   └── fooer.go
└── testdata
    └── example-input.yaml

Testing guidelines

This section annotates various standards for unit tests in airshipctl. These should be thought of as "guidelines" rather than "rules".

How to write unit tests for files listed under the cmd package

Go files listed under the cmd package should be relatively slim. Their purpose is to be a client of the pkg package. Most of these files will contain no more than a single function which creates and returns a cobra.Command. Nonetheless, these functions need to be tested. To help alleviate some of the difficulties that come with testing a CLI, airshipctl provides several helper structs and functions under the testutil package.

As an example, suppose you have the following function:

func NewVersionCommand() *cobra.Command {
    versionCmd := &cobra.Command{
        Use:   "version",
        Short: "Show the version number of airshipctl",
        Run: func(cmd *cobra.Command, args []string) {
            out := cmd.OutOrStdout()
            clientV := version.clientVersion()
            w := util.NewTabWriter(out)
            defer w.Flush()
            fmt.Fprintf(w, "%s:\t%s\n", "airshipctl", clientV)
        },
    }
    return versionCmd
}

Testing this functionality is easy with the use of the pre-built testutil.CmdTest:

func TestVersion(t *testing.T) {
    versionCmd := cmd.NewVersionCommand()
    cmdTests := []*testutil.CmdTest{
        {
            Name:    "version",
            CmdLine: "",
            Cmd:     versionCmd,
            Error:   nil,
        },
        {
            Name:    "version-help",
            CmdLine: "--help",
            Cmd:     versionCmd,
            Error:   nil,
        },
    }

    for _, tt := range cmdTests {
        testutil.RunTest(t, tt)
    }
}

The above test uses CmdTest structs, which are then fed to the RunTest function. This function provides abstraction around running a command on the command line and comparing its output to a "golden file" (the pre-determined expected output). The following describes the fields of the CmdTest struct.

Once you've written your test, you can generate the associated golden files by running make update-golden, which invokes the "update" mode for testutil.RunTest. When the command has completed, you can view the output in the associated files in the testdata directory next to your command. Note that these files are easily discoverable from the output of git status. When you're certain that the golden files are correct, you can add them to the repo.