Reporters

The reporter classes are reporting the results into a human-readable, or machine-processable format.

CSVReporter

class performance.driver.classes.reporter.CSVReporter(config, generalConfig, eventbus)[source]

The CSV Reporter is creating a comma-separated value (.csv) file with the axis values and summarised metric values for every run.

reporters:
  - class: reporter.CSVReporter

    # [Optional] The filename to write the csv file into
    filename: results.csv

    # [Optional] The column separator character to use
    separator: ","

    # [Optional] Which value to use if a parameter is missing
    default: 0

This reporter is writing the summarised results in a CSV file. The resulting file will have the following columns:

Parameters Summarised Metrics Flags
p1 p2 … m1 (sum) m2 (sum) … f1 f2 …

The first line will contain the names of the parameters, metrics and flags.

Note

To configure which summariser(s) to use on every metric, use the summarize parameter in the config.metrics config.

DataDogReporter

class performance.driver.classes.reporter.DataDogReporter(config, generalConfig, eventbus)[source]

The DataDog Reporter is uploading the indicators into DataDog for archiving and alerting usage.

reporters:
  - class: reporter.DataDogReporter

    # The API Key to use
    api_key: 1234567890abcdef

    # The App Key to use
    app_key: 1234567890abcdef

    # The data points to submit
    points:

      # The name of the metric to submit to DataDog and the
      # indicator to read the data from
      - name: memory
        indicator: meanMemoryUsage

    # [Optional] The hostname to use as the agent name in datadog
    # If missing the network name of the machine will be used
    hostname: test.host

    # [Optional] Prefix of the metrics (Defaults to `dcos.perf.`)
    prefix: "dcos.perf."

The DataDog reporter is using the DataDog API to submit one or more indicator values as data points.

Note

This reporter is only collecting the indicators. Metric values or summaries cannot be reported to DataDog. Use the reporter.DataDogMetricReporter instead.

DataDogMetricReporter

class performance.driver.classes.reporter.DataDogMetricReporter(config, generalConfig, eventbus)[source]

The DataDog Metric Reporter uploads the raw metric values to DataDog the moment they are collected.

reporters:
  - class: reporter.DataDogMetricReporter

    # The API Key to use
    api_key: 1234567890abcdef

    # [Optional] The hostname to use as the agent name in datadog
    # If missing the network name of the machine will be used
    hostname: test.host

    # [Optional] Prefix of the metrics (Defaults to `dcos.perf.`)
    prefix: "dcos.perf."

    # [Optional] How frequently to flush the metrics to DataDog
    flushInterval: 5s

    # [Optional] Report configuration. If missing, the default behavior
    # is to report the summarized metrics at the end of the test.
    report:

      # [Optional] The string `all` indicates that all the metrics should
      # be submitted to DataDog
      metrics: all

      # [Optional] OR a list of metric names can be provided
      metrics:
        - metricA
        - metricB

      # [Optional] OR you can use a dictionary to provide an alias
      metrics:
        metricA: aliasedMetricA
        metricB: aliasedMetricB

      # [Optional] Set to `yes` to submit the raw values the moment they
      # are collected.
      raw: yes

      # [Optional] Set to `yes` to submit summarized values of the metrics
      # at the end of the test run.
      summarized: yes

      # [Optional] The string `all` indicates that all the indicators should
      # be submitted to DataDog
      indicators: all

      # [Optional] OR a list of indicator names can be provided
      indicators:
        - indicatorA
        - indicatorB

      # [Optional] OR you can use a dictionary to provide an alias
      indicators:
        indicatorA: aliasedIndicatorA
        indicatorB: aliasedIndicatorB

The DataDog reporter is using the DataDog API to submit the values of the test metrics to DataDog in real-time.

PlotReporter

class performance.driver.classes.reporter.PlotReporter(config, generalConfig, eventbus)[source]

The Plot Reporter is creating a PNG plot with the measured values and storing it in the results folder.

reporters:
  - class: reporter.PlotReporter

    # [Optional] Default parameter value to use if not specified
    default: 0

    # [Optional] Filename prefix and suffix (without the extension)
    prefix: "plot-"
    suffix: ""

    # [Optional] The X and Y axis scale (for all plots)
    # Can be one of: 'linear', 'log', 'log2', 'log10'
    xscale: linear
    yscale: log2

    # [Optional] The colormap to use when plotting 2D plots
    # Valid options from: https://matplotlib.org/examples/color/colormaps_reference.html
    colormap: winter

    # [Optional] Plot the raw values as a scatter plot and not the summarised
    raw: False

    # [Optional] Reference data structure
    reference:

      # Path to raw reference JSON
      data: http://path.to/refernce-raw.json

      # [Optional] The colormap to use when plotting the reference 2D plots
      ratiocolormap: bwr

      # [Optional] Name of the reference data
      name: ref

      # [Optional] Headers to send along with the request
      headers:
        Authentication: "token={{token}}"

This reporter will generate an image plot for every metric defined. The axis is the 1 or 2 parameters of the test.

Warning

The PlotReporter can be used only if the total number of parameters is 1 or 2, since it’s not possible to display plots with more than 3 axes.

Trying to use it otherwise will result in an exception being thrown.

PostgRESTReporter

class performance.driver.classes.reporter.PostgRESTReporter(config, generalConfig, eventbus)[source]

The PostgREST Reporter is uploading the full set of results in a structured manner in a Postgres database using a PostgREST API endpoint.

reporters:
  - class: reporter.PostgRESTReporter

    # The URL to the PostgREST endpoint
    url: "http://127.0.0.1:4000"

    # [Optional] The database table prefix
    prefix: "profile_data_"

This reporter is uploading the following information

Important

The Postgres database is using uuid-based lookup for every parameter and metric. Therefire it’s required to include the uuid parameter in the config.metrics and config.parameters configuration.

This de-couples the representation of the metric across different projects or versions of the same project.

Related:

RawReporter

class performance.driver.classes.reporter.RawReporter(*args, **kwargs)[source]

The Raw Reporter is creating a raw dump of the results in the results folder in JSON format.

reporters:
  - class: reporter.RawReporter

    # Where to dump the results
    filename: "results-raw.json"

    # [Optional] Include event traces
    events:

      # [Optional] Include events that pass through the given expression
      include: FilterExpression

      # [Optional] Exclude events that pass through the given expression
      exclude: FilterExpression

      # [Optional] Group the events to their traces
      traces: yes

The JSON structure of the data included is the following:

{

  // Timing information
  "time": {
    "started": "",
    "completed": ""
  },

  // The configuration used to run this test
  "config": {
    ...
  },

  // The values for the indicators
  "indicators": {
    "indicator": 1.23,
    ...
  },

  // The metadata of the run
  "meta": {
    "test": "1-app-n-instances",
    ...
  },

  // Raw dump of the timeseries for every phase
  "raw": [
    {

      // One or more status flags collected in this phase
      "flags": {
        "status": "OK"
      },

      // The values of all parameter (axes) in this phase
      "parameters": {
        "apps": 1,
        "instances": 1
      },

      // The time-series values for every phase
      "values": {
        "metricName": [

          // Each metric is composed of the timestamp of it's
          // sampling time and the value
          [
            1499696193.822527,
            11
          ],
          ...

        ]
      }
    }
  ],

  // Summarised dump of the raw timeseries above, in the same
  // structure
  "sum": [
    {

      // One or more status flags collected in this phase
      "flags": {
        "status": "OK"
      },

      // The values of all parameter (axes) in this phase
      "parameters": {
        "apps": 1,
        "instances": 1
      },

      // The summarised values of each timeseries
      "values": {
        "metricName": {

          // Here are the summarisers you selected in the `metric`
          // configuration parameter.
          "sum": 123.4,
          "mean": 123.4,
          ...
        }
      }
    }
  ]
}

RawEventsReporter

class performance.driver.classes.reporter.RawEventsReporter(*args)[source]

The Raw Events Reporter is dumping every event in the eventBus to a file that can be used for offline event processing. You can also use this reporter for debugging the performance driver internals.

reporters:
  - class: reporter.RawEventsReporter

    # [Optional] Set to `yes` to track TickEvents
    # Note that including tick events might introduce a lot of noise to
    # your data and/or increase the reporting impact.
    tickEvents: no

    # Where to dump the events
    filename: "events.dump"

The log file is encoded with the following rules:

  1. The events are encoded in plain-text
  2. Each event is separated with a new line
  3. Each line contains two columns separated with semicolon
  4. The first column contains the unix timestamp of the event
  5. The second column contains the name of the event
  6. The third column contains the field values for the event encoded as a JSON string.

For example:

//   Timestamp  //    Name    //      Properties   //
1500891843.976068;SomeEventName;{"prop":"value", ...}
...

This format allows for simple grepping and more elaborate parsing. For example

cat event.dump | grep ';TickEvent;' | wc -l

S3Reporter

class performance.driver.classes.reporter.S3Reporter(*args, **kwargs)[source]

The S3 Reporter is uploading a raw dump of the results in a bucket in Amazon’s S3 services

reporters:
  - class: reporter.S3Reporter

    # The name of the S3 bucket
    bucket: dcos-perf-test-results

    # [Optional] If ommited, you must provide the AWS_ACCESS_KEY_ID
    # environment variable
    aws_access_key_id: ...

    # [Optional] If ommited, you must provide the AWS_SECRET_ACCESS_KEY
    # environment variable
    aws_secret_access_key: ...

    # [Optional] The path in the bucket where to save the file
    path: results-raw.json

    # [Optional] A canned ACL. One of: private, public-read,
    # public-read-write, authenticated-read, bucket-owner-read,
    # bucket-owner-full-control
    acl: private

    # [Optional] Put the filename uploaded on the given index file
    index:

      # The path to the index JSON file
      path: path/to/index.json

      # The index entry to update
      entry: some_key

      # [Optional] How many items to keep under this key
      max_entries: 100

      # [Optional] The bucket name if different than the above
      bucket: dcos-perf-test-results

This reporter behaves exactly like RawReporter, but the generated JSON blob is uploaded to an S3 bucket instead of a file in your local filesystem.