The configuration file: config.yml

jenkins-pipeline-library relies on a YAML file in order to compose dinamically the stages that tackle the fulfillment of a requirement or good practice as defined by the SQA baseline.

The configuration file must exist under .sqa/config.yml path, relative to the root directory of your code repository.

Below is an example YAML file which shows the most common configuration options:

# .sqa/config.yml - jenkins-pipeline-library configuration file

# generic configuration: workspace, agents
config:
  project_repos:
    worsica-processing:
      repo: 'https://github.com/WORSICA/worsica-processing.git'
      branch: master

sqa_criteria:
  qc_style:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
            - stylecheck

environment:
  GIT_COMMITTER_NAME: Person1
  GIT_COMMITTER_EMAIL: person1@example.org
  LANG: C.UTF-8

timeout: 600

Supported settings

The .sqa/config.yml file shall contain the following main settings:

config 

Here you can define the generic parameters, such as the workspace and execution agents.

Type:: map
Parameters:: node_agent, deploy_template, project_repos
Required:: true

Example:

config:
  project_repos:
    worsica-processing:
      repo: 'https://github.com/WORSICA/worsica-processing.git'
      branch: master

node_agent

The type of agent to drive the execution of the dynamic stages. Docker Compose is used by default.

Type:: string
Options:: docker_compose
Default:: docker_compose
Location:: config:node_agent

Example:

config:
  node_agent: 'docker_compose'

deploy_template

Path to the template containing the definition of the services for the node_agent in use.

Type:: path
Default:: .sqa/docker-compose.yml
Location:: config:deploy_template

Example:

config:
  deploy_template: '.sqa/docker-compose.yml'

project_repos

Describes the code repositories that the pipeline will deal with.

Type:: map
Required:: true
Location:: config:project_repos

Example:

config:
  project_repos:
    worsica-processing:
      repo: 'https://github.com/WORSICA/worsica-processing.git'
      branch: master

The set of allowed parameters for the definition of the code repository’s description within the project_repos setting are herein described:

repo

URL pointing to the root path of the code repository.

Type:: url
Required:: true
Location:: config:project_repos:repo

branch

Branch name to be checked out.

Type:: string
Default:: master
Location:: config:project_repos:branch

sqa_criteria 

This setting allows to define the criteria from the SQA baseline to be checked. Each requirement has a unique identifier and an associated set of mandatory and optional attributes.

Type:: map
Parameters:: qc_style, qc_functional, qc_coverage, qc_security, qc_doc
Required:: true

Example:

sqa_criteria:
  qc_style:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
             - stylecheck

Note

The sqa_criteria setting is the most relevant section of the .sqa/config.yml as it defines the different stages that will be dynamically added to the pipeline.

The relationship between the identifiers used in the definition of the sqa_criteria (see Options above) and the ones used in the SQA criteria is summarized as follows:

sqa_criteria ID	SQA baseline category
qc_style	QC.Sty
qc_coverage	QC.Uni
qc_functional	QC.Fun
qc_security	QC.Sec
qc_doc	QC.Doc

The previous table lists the set of criteria that is currently supported by the current version of the jenkins-pipeline-library. The settings described in this section are common to all, which are applicable and defined per-repository, and thus, they must be used within the repos map setting, as showed in the following examples.

Note

The repositories used under repos must be previously defined in the config:project_repos setting. They are referred by the identifiers used there.

Examples:

sqa_criteria:
  qc_style:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
              - stylecheck

sqa_criteria:
  qc_coverage:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
              - coverage
      worsica-portal:
        container: celery
        tox:
          testenv:
              - coverage

sqa_criteria:
  qc_functional:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
              - unittest
      worsica-portal:
        container: celery
        tox:
          testenv:
              - functional

sqa_criteria:
  qc_security:
    repos:
      worsica-processing:
        container: processing
        tox:
          testenv:
              - security

sqa_criteria:
  qc_doc:
    repos:
      worsica-cicd:
        container: processing
        commands:
          - python setup.py build_sphinx

Next, we will describe those available settings, some of them used in the previous examples, that can be defined for each repository associated with the former criteria:

container

Allows to specify the Docker container where the given criteria assessment will take place. If using docker_compose, the value could be any of the services defined in the docker-compose.yml.

Type:: string
Required:: true
Location:: sqa_criteria:<qc_xxx>:repos:<repo>:container

tox

Built-in support tox application. It is only applicable for Python environments.

Type:: map
Parameters:: testenv, tox_file
Location:: sqa_criteria:<qc_xxx>:repos:<repo>:tox

testenv

Identifier of the test environment that tox shall use.

Type:: list
Required:: true
Location:: sqa_criteria:<qc_xxx>:repos:<repo>:tox:testenv

tox_file

Specifies the path to the tox configuration file.

Type:: path
Default:: tox.ini
Location:: sqa_criteria:<qc_xxx>:repos:<repo>:tox:tox_file

Note

If using tox without container, the jenkins-pipeline-library will automatically select an appropriate Docker container for running the tool.

commands

Allows to include a list of commands. This is helpful whenever there is no built-in support for the tool you use for building purposes.

Type:: list
Default:: []
Location:: sqa_criteria:<qc_xxx>:repos:<repo>:commands

Example:

sqa_criteria:
  qc_sec:
    repos:
     worsica-processing:
       commands:
         - bundle exec brakeman --exit-on-error

Note

commands requires the presence of the container setting, which must have available all the tools –and dependencies– used by the list of commands. Also the commands runs relative to the root directory /. As a hacking solution is possible to use Docker Compose’s Environment variables to define the expected workspace in docker-compose.yml context, as a solution for current release.

environment 

Contains the environment variables required to execute the previouos SQA criteria checks.

Type:: list
Default:: []

Example:

environment:
  GIT_COMMITTER_NAME: Person1
  GIT_COMMITTER_EMAIL: person1@example.org
  LANG: C.UTF-8

Note

environment variables are only usable by the deployment (for example with docker_compose) or defined features in current version. This environment will not be available inside the containers. For that, you should use for example, docker-compose.yml environment definitions instead.

Note

The following JPL-prefixed environment variables have a special purpose:

JPL vars	Purpose
JPL_DOCKERPUSH	Space-separated list of defined docker-compose services whose image will be pushed to the Docker registry. If `ALL` value is used, it will push all locally built images defined in docker-compose.yml
JPL_IGNOREFAILURES	If set, by using any random string value (without spaces), it will ignore any push-related failure
JPL_DOCKERFORCEBUILD	Forcedly rebuild all images with build clause in docker-compose.yml
JPL_DOCKERSERVER	Sets Docker registry server. By default it will use Docker Hub
JPL_DOCKERUSER	Sets username of Docker registry credentials
JPL_DOCKERPASS	Sets password of Docker registry credentials

timeout 

Sets the timeout for the pipeline execution.

Type:: integer
Default:: 600

Example:

timeout: 60

Docker Registry: upload images

As mentioned in special purpose environment variables note, pushing images to docker registry is supported using the following environment variables:

JPL vars	Purpose
JPL_DOCKERPUSH	Space-separated list of defined docker-compose services whose image will be pushed to the Docker registry. If `ALL` value is used, it will push all locally built images defined in docker-compose.yml
JPL_IGNOREFAILURES	If set, by using any random string value (without spaces), it will ignore any push-related failure
JPL_DOCKERSERVER	Sets Docker registry server. By default it will use Docker Hub
JPL_DOCKERUSER	Sets username of Docker registry credentials
JPL_DOCKERPASS	Sets password of Docker registry credentials

Example 1: upload specific images to dockerhub registry ignoring failures

config.yml example with minimal required configurations:

config:
  credentials:
    - id: my-dockerhub-token
      username_var: JPL_DOCKERUSER
      password_var: JPL_DOCKERPASS

environment:
  JPL_DOCKERPUSH: "docs service1 service4"
  JPL_IGNOREFAILURES: "defined"

In this example there are three services:

service1: main service that have is Dockerfile in the service1 directory inside git repository.
service2: same as service1 with Dockerfile inside directory service2 and depends on service1 to be built.
docs: service to generate the project documentation.

The docker-compose.yml file that would work with previous configuration can be as the following (mandatory the build keyword in service definition):

version: "3.7"

services:
  service1:
     build:
        context: "."
        dockerfile: "./service1/Dockerfile"
     image: "organization/service1:${GIT_BRANCH}"

  service2:
     build:
        context: "."
        dockerfile: "./service2/Dockerfile"
        cache_from:
           - "organization/service1:${GIT_BRANCH}"
     image: "organization/service2:${GIT_BRANCH}"
     depends_on:
        - service1

  docs:
     build:
        context: "."
        dockerfile: "./docs/Dockerfile"
     image: "organization/docs:${GIT_BRANCH}"

Example 2: upload all images to independent registry and fail with push failures

config:
  credentials:
    - id: my-dockerhub-token
      username_var: JPL_DOCKERUSER
      password_var: JPL_DOCKERPASS

environment:
  JPL_DOCKERPUSH: "ALL"
  JPL_DOCKERSERVER: "mydockerregistry.example.com:8080"

Note

When using custom docker registry is also expected that docker-compose.yml have the expected configuration for the image references, following the official documentation.

Warning

The docker-compose.yml file for this example could be any. With ‘ALL’ value it will upload all loaded images to the custom registry. This also includes all images pulled from Dockerhub or other docker registry without a build section defined in docker-compose.yml.

A Note on Docker images and registries

Images are defined in docker-compose.yml file and there is no relation of those with defined service names. Also the docker registry repository needs to be previously created before running the last step of the generated pipeline. Last step will be always the image push to docker registry. Images that can be pushed in the end require the build keyword in service definition at docker-compose configuration. In next examples the sqa_criteria property is being omitted to focus only in the required configurations to push images to a docker registry. Also project_repos in config section is being removed since is not required, so it turns the examples more clear. Jenkins environment variable ${GIT_BRANCH} receives the branch or tag from git repository.

The configuration file: config.yml

Supported settings

config

node_agent

deploy_template

project_repos

sqa_criteria

container

tox

commands

environment

timeout

Docker Registry: upload images

A Note on Docker images and registries

config 

sqa_criteria 

environment 

timeout 