3.3.22. Statistics Service
The statistics service (or stats service) is implemented in buildbot.statistics.stats_service.
Please see stats-service for more information.
Here is a diagram demonstrating the workings of the stats service:
 
3.3.22.1. Stats Service
- class buildbot.statistics.stats_service.StatsService
- An instance of this class functions as a - BuildbotService. The instance of the running service is initialized in the master configuration file (see- stats-servicefor more information). The running service is accessible everywhere in Buildbot via the- BuildMaster. The service is available at- self.master.namedServices['<service-name>']. It takes the following initialization arguments:- storage_backends
- A list of storage backends. These are instance of subclasses of - StatsStorageBase.
- name
- (str) The name of this service. This name can be used to access the running instance of this service using - self.master.namedServices[name].
 - Please see - stats-servicefor examples.- checkConfig(self, storage_backends)
- storage_backends
- A list of storage backends. 
 - This method is called automatically to verify that the list of storage backends contains instances of subclasses of - StatsStorageBase.
 - reconfigService(self, storage_backends)
- storage_backends
- A list of storage backends. 
 - This method is called automatically to reconfigure the running service. 
 - registerConsumers(self)
- Internal method for this class called to register all consumers (methods from Capture classes) to the MQ layer. 
 - stopService(self)
- Internal method for this class to stop the stats service and clean up. 
 - removeConsumers(self)
- Internal method for this class to stop and remove consumers from the MQ layer. 
 - yieldMetricsValue(self, data_name, post_data, buildid)
- data_name
- (str) The name of the data being sent for storage. 
- post_data
- A dictionary of key-value pairs that is sent for storage. 
- buildid
- The integer build id of the current build. Obtainable in all - BuildSteps.
 - This method should be called to post data that is not generated and stored as build-data in the database. This method generates the - stats-yield-dataevent to the mq layer which is then consumed in- postData.
 
3.3.22.2. Storage backends
Storage backends are responsible for storing any statistics/data sent to them. A storage backend will generally be some sort of a database-server running on a machine.
Note
This machine may be different from the one running BuildMaster.
Data is captured according to the master config file and then is sent to each of the storage backends provided by the master configuration (see stats-service).
Each storage backend has a Python client defined as part of buildbot.statistics.storage_backends to aid in posting data by StatsService.
Currently, only InfluxDB is supported as a storage backend.
- class buildbot.statistis.storage_backends.base.StatsStorageBase
- An abstract class for all storage services. It cannot be directly initialized - it would raise a - TypeErrorotherwise.- thd_postStatsValue(self, post_data, series_name, context)
- post_data
- A dict of key-value pairs that is sent for storage. The keys of this dict can be thought of as columns in a database and the value is the data stored for that column. 
- series_name
- (str) The name of the time-series for this statistic. 
- context
- (Optional) Any other contextual information about the data. It is a dict of key-value pairs. 
 - An abstract method that needs to be implemented by every child class of this class. Not doing so will result in a - TypeErrorwhen starting Buildbot.
 
- class buildbot.statistics.storage_backends.influxdb_client.InfluxStorageService
- InfluxDB is a distributed time series database that employs a key-value pair storage system. - This class is a Buildbot client to the InfluxDB storage backend. It is available in the configuration as - statistics.InfluxStorageService. It takes the following initialization arguments:- url
- (str) The URL where the service is running. 
- port
- (int) The port on which the service is listening. 
- user
- (str) Username of an InfluxDB user. 
- password
- (str) Password for - user.
- db
- (str) The name of database to be used. 
- captures
- A list of instances of subclasses of - Capture. This tells which stats are to be stored in this storage backend.
- name=None
- (Optional) (str) The name of this storage backend. 
 - thd_postStatsValue(self, post_data, series_name, context={})
- post_data
- A dict of key-value pairs that is sent for storage. The keys of this dict can be thought of as columns in a database and the value is the data stored for that column. 
- series_name
- (str) The name of the time-series for this statistic. 
- context
- (Optional) Any other contextual information about the data. It is a dict of key-value pairs. 
 - This method constructs a dictionary of data to be sent to InfluxDB in the proper format and then sends the data to the InfluxDB instance. 
 
3.3.22.3. Capture Classes
Capture classes are used for declaring which data needs to captured and sent to storage backends for storage.
- class buildbot.statistics.capture.Capture
- This is the abstract base class for all capture classes. Not to be used directly. It’s initialized with the following parameters: - routingKey
- (tuple) The routing key to be used by - StatsServiceto register consumers to the MQ layer for the subclass of this class.
 - callback
- The callback registered with the MQ layer for the consumer of a subclass of this class. Each subclass must provide a default callback for this purpose. 
 - _defaultContext(self, msg)
- A method for providing default context to the storage backends. 
 - consume(self, routingKey, msg)
- This is an abstract method - each subclass of this class should implement its own consume method. If not, then the subclass can’t be instantiated. The consume method, when called (from the mq layer), receives the following arguments: - routingKey
- The routing key which was registered to the MQ layer. Same as the - routingKeyprovided to instantiate this class.
 - msg
- The message that was sent by the producer. 
 
 - _store(self, post_data, series_name, context)
- This is an abstract method of this class. It must be implemented by all subclasses of this class. It takes the following arguments: - post_data
- (dict) The key-value pair being sent to the storage backend. 
 - series_name
- (str) The name of the series to which this data is stored. 
 - context
- (dict) Any additional information pertaining to the data being sent. 
 
 
- class buildbot.statistics.capture.CapturePropertyBase
- This is a base class for both - CapturePropertyand- CapturePropertyAllBuildersand abstracts away much of the common functionality between the two classes. It cannot be initialized directly as it contains an abstract method and raises- TypeErrorif tried. It is initialized with the following arguments:- property_name
- (str) The name of property needed to be recorded as a statistic. This can be a regular expression if - regex=True(see below).
 - callback=None
- The callback function that is used by - CaptureProperty.consumerto post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this:
 - regex=False
- If this is set to - True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
 - consume(self, routingKey, msg)
- The consumer for all CaptureProperty classes described below. This method filters out the correct properties as per the configuration file and sends those properties for storage. The subclasses of this method do not need to implement this method as it takes care of all the functionality itself. See - Capturefor more information.
 - _builder_name_matches(self, builder_info)
- This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the - consumemethod mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:- builder_info
- (dict) The dictionary returned by the data API containing the builder information. 
 
 
- class buildbot.statistics.capture.CaptureProperty
- The capture class for capturing build properties. It is available in the configuration as - statistics.CaptureProperty.- It takes the following arguments: - builder_name
- (str) The name of builder in which the property is recorded. 
 - property_name
- (str) The name of property needed to be recorded as a statistic. 
 - callback=None
- The callback function that is used by - CaptureProperty.consumerto post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (see- CapturePropertyBasefor more information).
 - regex=False
- If this is set to - True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
 - _builder_name_matches(self, builder_info)
- See - CapturePropertyBasefor more information on this method.
 
- class buildbot.statistics.capture.CapturePropertyAllBuilders
- The capture class to use for capturing build properties on all builders. It is available in the configuration as - statistics.CapturePropertyAllBuilders.- It takes the following arguments: - property_name
- (str) The name of property needed to be recorded as a statistic. 
 - callback=None
- The callback function that is used by - CaptureProperty.consumerto post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (see- CapturePropertyBasefor more information).
 - regex=False
- If this is set to - True, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
 - _builder_name_matches(self, builder_info)
- See - CapturePropertyBasefor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildTimes
- A base class for all Capture classes that deal with build times (start/end/duration). Not to be used directly. It’s initialized with: - builder_name
- (str) The name of builder whose times are to be recorded. 
 - callback
- The callback function that is used by a subclass of this class to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this. Each subclass must provide a default callback that is used in initialization of this class should the user not provide a callback. 
 - consume(self, routingKey, msg)
- The consumer for all subclasses of this class. See - Capturefor more information.
 - _retValParams(self, msg)
- This is an abstract method which needs to be implemented by subclasses. This method needs to return a list of parameters that will be passed to the - callbackfunction. See individual build- CaptureBuild*classes for more information.
 - _err_msg(self, build_data, builder_name)
- A helper method that returns an error message for the - consumemethod.
 - _builder_name_matches(self, builder_info)
- This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the - consumemethod mentioned above. It checks whether a builder is allowed to send build times to the storage backend according to the configuration file. It takes one argument:- builder_info
- (dict) The dictionary returned by the data API containing the builder information. 
 
 
- class buildbot.statistics.capture.CaptureBuildStartTime
- A capture class for capturing build start times. It takes the following arguments: - builder_name
- (str) The name of builder whose times are to be recorded. 
 - callback=None
- The callback function for this class. See - CaptureBuildTimesfor more information.- The default callback: 
 - _retValParams(self, msg)
- Returns a list containing one Python datetime object (start time) from - msgdictionary.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildStartTimeAllBuilders
- A capture class for capturing build start times from all builders. It is a subclass of - CaptureBuildStartTime. It takes the following arguments:- callback=None
- The callback function for this class. See - CaptureBuildTimesfor more information.- The default callback: - See - CaptureBuildStartTime.__init__for the definition.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildEndTime
- A capture class for capturing build end times. Takes the following arguments: - builder_name
- (str) The name of builder whose times are to be recorded. 
 - callback=None
- The callback function for this class. See - CaptureBuildTimesfor more information.- The default callback: 
 - _retValParams(self, msg)
- Returns a list containing two Python datetime object (start time and end time) from - msgdictionary.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildEndTimeAllBuilders
- A capture class for capturing build end times from all builders. It is a subclass of - CaptureBuildEndTime. It takes the following arguments:- callback=None
- The callback function for this class. See - CaptureBuildTimesfor more information.- The default callback: - See - CaptureBuildEndTime.__init__for the definition.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildDuration
- A capture class for capturing build duration. Takes the following arguments: - builder_name
- (str) The name of builder whose times are to be recorded. 
 - report_in='seconds'
- Can be one of three: - 'seconds',- 'minutes', or- 'hours'. This is the units in which the build time will be reported.
 - callback=None
- The callback function for this class. See - CaptureBuildTimesfor more information.- The default callback: - default_callback(start_time, end_time)
- It returns the duration of the build as per the - report_inargument. It receives the following arguments:
 
 - _retValParams(self, msg)
- Returns a list containing one Python datetime object (end time) from - msgdictionary.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureBuildDurationAllBuilders
- A capture class for capturing build durations from all builders. It is a subclass of - CaptureBuildDuration. It takes the following arguments:- callback=None
- The callback function for this class. See - CaptureBuildTimesfor more.- The default callback: - See - CaptureBuildDuration.__init__for the definition.
 - _builder_name_matches(self, builder_info)
- See - CaptureBuildTimesfor more information on this method.
 
- class buildbot.statistics.capture.CaptureDataBase
- This is a base class for both - CaptureDataand- CaptureDataAllBuildersand abstracts away much of the common functionality between the two classes. Cannot be initialized directly as it contains an abstract method and raises- TypeErrorif tried. It is initialized with the following arguments:- data_name
- (str) The name of data to be captured. Same as in - yieldMetricsValue.
 - callback=None
- The callback function for this class. - The default callback: - The default callback takes a value - xand return it without changing. As such,- xitself acts as the- post_datasent to the storage backends.
 - consume(self, routingKey, msg)
- The consumer for this class. See - Capturefor more.
 - _builder_name_matches(self, builder_info)
- This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the - consumemethod mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:- builder_info
- (dict) The dictionary returned by the data API containing the builder information. 
 
 
- class buildbot.statistics.capture.CaptureData
- A capture class for capturing arbitrary data that is not stored as build-data. See - yieldMetricsValuefor more. Takes the following arguments for initialization:- data_name
- (str) The name of data to be captured. Same as in - yieldMetricsValue.
 - builder_name
- (str) The name of the builder on which the data is captured. 
 - callback=None
- The callback function for this class. - The default callback: - See - CaptureDataBaseof definition.
 - _builder_name_matches(self, builder_info)
- See - CaptureDataBasefor more information on this method.
 
- class buildbot.statistics.capture.CaptureDataAllBuilders
- A capture class to capture arbitrary data on all builders. See - yieldMetricsValuefor more. It takes the following arguments:- data_name
- (str) The name of data to be captured. Same as in - yieldMetricsValue.
 - callback=None
- The callback function for this class. 
 - _builder_name_matches(self, builder_info)
- See - CaptureDataBasefor more information on this method.