The new direction is to move functional testing out of tempest and into the projects. This means that we need to provide guidance for how this is to be done in Keystone. This spec answers the following questons:
We’ve also decided at the Kilo mid-cycle meeting that the Work Items will be updated in subsequent commits to reflect new functional tests scenarios that will be added. If someone decides that they would like to add tests to verify feature X, they will submit a change to this spec to add a subsection to Work Items based on the template.
This spec may lead to moving some unit tests into the functional tests, but is not advocating any substantial changes to how unit tests work today.
From the move functional testing post:
Put the burden for a bunch of these tests back on the projects as “functional” tests. Basically a custom devstack environment that a project can create with a set of services that they minimally need to do their job. These functional tests will live in the project tree, not in Tempest, so can be atomically landed as part of the project normal development process.
What exactly is a functional test? If the answer of any of the following questions is ‘yes’, then you have a functional test.
$ pwd
/opt/stack/keystone
keystone/tests/functional
├── federation
│ ├── __init__.py
│ └── {{test_something}}.py
└── shared
├── __init__.py
└── {{test_something}}.py
2 directories, 4 files
Functional shared tests should be run by developers before submitting changes to Gerrit. This should not be much of a barrier because those are tests that need to run on any running Keystone configuration.
When developers are working on feature for a specific configuration they should run the config specific tests.
To run all of the shared tests (unlikely that you want to do this):
tox -e functional
To run all of the federation config tests:
tox -e functional -- keystone.tests.functional.federation
The tests will be written in a style that is similar to unit tests. tox will be used to run the tests. Test classes are subclasses to testtools.
The difference comes in when you start talking about the contents of the tests themselves. Functional tests should be written using the black box model. The test should have no knowledge of the backends being used. Ideally all tests can be written using primarily keystoneclient and requests to interact with Keystone.
Certain elements of the tests must be controllable via environment variables. For example, the base URL for Keystone. This allows the developer running the tests to point to any Keystone instance. Beyond the Keystone URLs there is no official list of what must be configurable.
I have not really investigated alternatives. This spec is intentionally a bit vague so that we can innovate a little in this space.
None. This is about tests and does’t directly impact production code.
None. This is about tests and does’t directly impact production code.
None. This is about tests and does’t directly impact production code.
None. This is about tests and does’t directly impact production code.
None. This is about tests and does’t directly impact production code.
Certain functional tests may require a specific environment to be available. For example, to run the federation tests your Keystone instance will have to be configured to use federation.
The developer documentation will need to be updated to explain how and when to run the functional tests. A new document will need to be created to explain how to create and extend the functional tests.