Policies¶

The policy classes are driving the tests by controlling the evolution of the test parameters over time. The parameters changed are applied to the test through the Channels.

MultivariableExplorerPolicy¶

class performance.driver.classes.policy.MultivariableExplorerPolicy(config, eventbus, parameterBatch)[source]¶

The Multi-Variable Exploration Policy is running one scale test for every product of the parameters defined in the matrix.

policies:
  - class: policy.MultivariableExplorerPolicy

    # The following rules describe the permutation matrix
    matrix:
      # The key is the name of the parameter to control
      param:
        ...

      # A "discreet" parameter can take one of the specified values
      apps:
        type: discreet
        values: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096]

      # A "sample" parameter takes any value within a numerical range
      size:
        type: sample
        min: 0 # Default
        max: 1 # Default
        step: 1 # Default
        samples: 10 # Default

      # A "range" parameter takes all the values within the given range
      instances:
        type: range
        min: 0
        max: 1
        step: 1

    # The event binding configuration
    events:

      # Signals are events that define a terminal condition and it's status
      #
      # For example, in this case when the `MarathonDeploymentSuccessEvent`
      # is received, the test will be completed with status `OK`
      #
      signal:
        OK: MarathonDeploymentSuccessEvent
        FAILED: MarathonDeploymentFailedEvent
        ... : ...

      # [Optional] Wait for the given number of signal events before
      # considering the test complete.
      #
      # This parameter is an expression evaluated at run-time, so you
      # could use anything that can go within a python's `eval` statement
      #
      # For example: "discreet + 2"
      #
      signalEventCount: 1

      # [Optional] Start the tests with this event is received
      start: EventToWaitUntilReady

This policy is first computing all possible combinations of the parameter matrix given and is then running the tests for every one.

The policy will start immediately when the test driver is ready unless the start event is specified. In that case the policy will wait for this event before starting with the first test.

The policy continues with the next test only when a signal event is received. Such events are defined in the signal dictionary. Since a test can either complete successfully or fail you are expected to provide the status indication for every signal event.

It is also possible to wait for more than one signal event before considering the test complete. To specify the number of events to expect you can use the signalEventCount parameter.

However since the number of events to expect depends on an arbitrary number of factors, it’s possible to use an expression instead of a value. For the expression you can use the names of the parameters taking part in the matrix and the special variable _i that contains the number of the test, starting from 1.

For example signalEventCount: "apps + size/2"

TimeEvolutionPolicy¶

class performance.driver.classes.policy.TimeEvolutionPolicy(config, eventbus, parameterBatch)[source]¶

The Time Evolution Policy is changing a parameter monotonically as the time evolves.

policies:
  - class: policy.TimeEvolutionPolicy

    # Configure which parameters to evolve over time
    evolve:

      # The name of the parameter to change
      - parameter: parameterName

        # [Optional] The interval (in seconds) at which to evolve the
        # parameter (default is 1 second)
        interval: 1

        # [Optional] By how much to increment the parameter (default is 1)
        step: 1

        # [Optional] The initial value of the parameter (default is 0)
        min: 0

        # [Optional] The final value of the parameter, after which the
        # test is completed.
        max: 10

    # The event binding configuration
    events:

      # [Optional] Terminate the tests when this event is received.
      end: EventToWaitForCompletion

      # [Optional] Start the tests with this event is received
      start: EventToWaitUntilReady

This policy is first computing all possible combinations of the parameter matrix given and is then running the tests for every one.

MultiStepPolicy¶

class performance.driver.classes.policy.MultiStepPolicy(config, eventbus, parameterBatch)[source]¶

The Step Policy evolves various parameters through various steps.

policies:
  - class: policy.MultiStepPolicy

    # Configure the policy steps
    steps:

      # The name of the step
      - name: First Step

        # The values to explore
        values:

          # 1) A fixed-value parameter
          - parameter: name
            value: 1

          # 2) A fixed-value parameter calculated with an expression
          - parameter: name
            value: "log(parameter1) + parameter2"

          # 3) A fixed list of values
          - parameter: name
            values: [1, 2, 3, 4, 5 ...]

          # 4) A calculated range of values
          - parameter: name
            min: 0
            max : 100
            step: 1

            # Set to `no` if you don't want to include the `max` value
            inclusive: no

          # 5) A sampled subset of a uniformly distributed values
          - parameter: name
            min: 0
            max: 10000
            step: 1
            sample: 100

            # Set to `no` if you don't want to include the `max` value
            inclusive: no

        # [Optional] Trigger the following named tasks
        tasks:

          # [Optional] Run this named task before start
          start: startTask

          # [Optional] Run this named task after step completion
          end: endTask

          # [Optional] Run this named before a value change
          pre_value: advanceTask

          # [Optional] Run this named after a value change
          post_value: advanceTask

        # [Optional] Event configuration
        events:

          # [Optional] Wait for this to start the step
          start: EventName

          # [Optional] Wait for this event to end the step
          end: EventName

          # [Optional] Wait for this event to fail this step
          fail: EventName

          # [Optional] Wait for this event before advancing to the next value
          advance: EventName

        # [Optional] Custom end condition
        end_condition:

          # [Optional] Wait for the specified number of end/fail events
          # before considering the step completed
          events: 1

          # [Optional] Wait for the specified number of seconds after the
          # final step is completed before completing the test
          linger: 0

        # [Optional] Custom advance condition
        advance_condition:

          # [Optional] Wait for the specified number of advance events
          # before considering the value ready to advance. Note that this
          # can be a python expression
          events: ""

          # [Optional] If the step was not advanced by itself in the given
          # time, marked is a timed out and continue with the next
          timeout: 10s

          # [Optional] What flag to set on the run that advanced due to a
          # timeout. Set this to `OK` to make timeout a legit action or
          # `FAILED` to make timeout a critical failure.
          timeout_status: TIMEOUT

This policy is first computing all possible combinations of the parameter matrix given and is then running the tests for every one.

Important

The MultiStepPolicy is respecting the event tracing principle. This means that all the events in the events section will be matched only if they derive from the same step of a policy action.

If the events you are listening for do not belong on a trace initiated by the current step, use the :notrace indicator.

For example, let’s say that policy sets the number of instances to 3, that triggers a deployment that eventually triggers a DeploymentCompletedEvent when completed. In this case you can listen for advance: DeploymentCompletedEvent events.

However, if you are advancing at clock ticks, they are not part of a trace initiated by the policy and therefore you must use: `advance: TickEvent:notrace

SimplePolicy¶

class performance.driver.classes.policy.SimplePolicy(config, eventbus, parameterBatch)[source]¶

The Simple Policy submits a single parameter change event and terminates when the designated end condition is met. No repetition or parameter exploration is performed.

policies:
  - class: policy.SimplePolicy

    # Which parameters to submit
    parameters:
      apps: 1

    # The event binding configuration
    events:

      # [Optional] Terminate the tests when this event is received.
      end: EventToWaitForCompletion

      # [Optional] Start the tests with this event is received
      start: EventToWaitUntilReady

    # [Optional] Maximum running time for this policy
    timeout: 10s

This policy can be used if you are not interested about parameter exploration or any other feature of the driver, but you rather want to observe the system response on a particular condition.