Part 2: Go Libraries, Pipeline Templates and Versioning
Having already created a simple pipeline in Part 1 of this article series, we dive a little deeper in Part 2. We create a library for our Go example project that will contain shared code with other potential Go microservices. In doing so, we have the challenge that its Git repository should remain private in Azure DevOps. Next, we will create a pipeline for a Python application. To avoid duplicate code in the pipeline scripts, we create a pipeline template from which both pipelines inherit their basic structure. We also extend the pipelines with automatic versioning using Git tags, which will significantly simplify dependency management and the use of the published Docker images.
Integrating your own Go Libraries from Azure repositories
Dependency management in Go is relatively simple: it only requires the URL to a Git repository and a Git tag. Go then checks out the commit with the corresponding tag from the Git repository and makes the code available during compilation. For public Git repositories, for example on GitHub, not much can go wrong. For private Git repositories, on the other hand, a few extra steps are needed.
First, we create a second Git repository named example-go-library with a single function, which we want to call later inside the example-go-project.
For our library to be able to be referenced later in other Go projects, it is imperative to use the full URL as the module name:
Otherwise, we will get error message like the following one:
The import in the example-go-project then looks as following:
If the library is a private Git repo, as in this case, the following settings are also necessary (both in the local development environment and later in the Dockerfile):
- The environment variable GOPRIVATE. It prevents the library from being loaded via a public Go proxy (which has no access to the private Git repository).
- The following git setting for authentication (a PAT can be created in Azure DevOps under the menu item “Personal Access Tokens” in the user menu at the top right corner of the page):
We take the small HTTP server from part 1 of this article series as a basis and replace the import “github.com/sirupsen/logrus” with “dev.azure.com/dennishellerdigatus/CICD-Test/_git/example-go-library.git/log” and all calls to logrus.Info with log.Message.
The Dockerfile must also be adapted accordingly so that we have access to the private Git repository there. So we also set the environment variable GOPRIVATE here and the Git setting with the PAT:
We also have to adapt the pipeline. There, too, we make the necessary preparations before the go mod download step. Fortunately, we don’t have to publish our personal PAT in the Dockerfile, but get an automatically generated PAT, since the build is already being executed in the privileged setting of our Azure DevOps project. We get this via the variable $(System.AccessToken) and it is only valid for the duration of the build. The complete pipeline now looks like this:
Running the pipeline, we see that the PAT is used (and masked) by Azure DevOps:
Afterwards, a test in a local shell confirms that everything still works as before:
Even if everything looks the same on the outside, we now have the advantage that we can move any code into the library and reuse it in other Go projects. Especially in a microservice landscape with many small services written in Go, there is usually a common code base, so you can save a lot of duplicate code by introducing such a private library.
The second CI Pipeline – Python and Docker
In order to bring some variety into our microservice landscape, we now switch to antoher language: let’s create a microservice in Python, which will call the REST endpoint of our Go microservice. Of course, the new microservice also gets a pipeline. Since Python, in contrast to Go, is interpreted at runtime, the build step is omitted here. The test steps look similar to the Go pipeline and the Docker steps are identical. Later, we will extract the similarities of both pipelines into a pipeline template in order to avoid duplicate code and to be flexible for future pipelines.
Here is the simple code of our Python application, which simply calls our Go microservice every 3 seconds and logs the result:
For the sake of completeness, we create a small unit test that mocks the Go microservice and uses capsys to catch the standard output:
Then we wrap the application into a Docker image. The environment variable PYTHONUNBUFFERED=1 is important to ensure that we can see the log output in real time. We install the dependencies with pip – in this case only a single library, otherwise we would use a requirements file.
We create the pipeline in the same way like the Go pipeline. As a first step, we again tell Azure DevOps which language and which version we want to work with – in this case Python 3.11.3. Next, we install the dependencies and then we run the tests using pytest. The parameter –capture=tee-sys ensures that we can capture the standard output in the test. With –cov=main we calculate the code coverage and with –junit-xml=report.xml we create the classic test report. Here, too, there is already a tool for preparing the code coverage: the Python package coverage. Without further parameters, it is by default compatible with the result format of pytest. The remaining steps for publishing the test results and for building and pushing the Docker image are identical to the Go pipeline:
Our first Python pipeline is ready. The result is impressive:
Also the test results and the coverage:
Thinking with foresight: Pipeline Templates
If we now compare our two pipelines – Go and Python – we notice that we have some identical steps and some different steps. To save work for future pipelines, Azure DevOps offers us the possibility to create pipeline templates. Also, if we later want to change or extend a common part of the pipeline, we then only have to do this once in the common template and not in each individual pipeline. The template hierarchy could even be continued further, so that we would have, for example, a common template for the entire company, then a sub-template for the project and further sub-templates for CI and CD, different languages and frameworks up to the final pipeline for a microservice. The keyword for using templates, as in object-oriented programming, is extends:. The important thing is that a pipeline can only inherit from exactly one template. The key to making templates extendable are parameters that can be used to fill the placeholders in the template. These parameters can be simple text values, numbers, lists, complex objects and even lists of complete pipeline steps. Default values are also possible. The parameters of a template are declared at the very top of the template under the parameters: section and can then be used in the code of the template with the following notation: {{ parameters.xxx }}. The template hierarchy and the parameters are evaluated when the pipeline is compiled to create a single large pipeline script in which the parameters are already replaced. In contrast to the parameters, there are so-called variables. These are used with the following notation and are only interpreted at runtime: $(variable). Template files are saved as YAML files like normal pipelines. Since they are used in several other Git repositories, it makes sense to create a separate Git repository for them, in our case we call it example-pipeline-templates.
Back to our two pipelines: the basic structure (publishing the test results and building and uploading the Docker image) is identical. Only the middle part, the building and testing, differs. Here is an overview of all the steps in the two pipelines:
| Go-Pipeline | Python-Pipeline |
|---|---|
|
|
|
|
displayName: Go set environment variables |
|
displayName: Git config |
|
displayName: Go mod download |
displayName: Prepare environment |
displayName: Go build |
|
displayName: Go test |
displayName: Execute tests |
|
|
|
|
displayName: Build Docker image |
displayName: Build Docker image |
displayName: Push Docker image |
displayName: Push Docker image |
It therefore makes sense to move the checkout: step and the last 4 steps into a common template and to use a placeholder with parameters for the middle part. This looks like this:
If a parameter is noted as a single YAML list entry but contains a list, Azure DevOps expands it automatically without us having to write an extra each loop here. The template basically looks like a normal pipeline and could also be used as such. If we were to create a pipeline from this template file in Azure DevOps, we would have to fill the parameters manually when starting the pipeline, which is not possible for the stepList type. Therefore, the default value would be taken here: an empty list.
We now change our two previous pipelines so that they inherit from this template and we set the values for the parameters. To do this, we first have to specify the Git repository that contains the template and name it with an alias. Then we can specify the template with extends: and template:. The syntax here is <relativer Pfad>@<repository-Alias>. If a pipeline contains extends: at the top level, it must not contain its own stages:, jobs: oder steps: next to it. Instead, the complete pipeline must be built through the skeleton provided by the parent template and all individual changes must be realised via parameters. As already mentioned, Azure DevOps assembles a single large pipeline script from the template hierarchy before executing the pipeline, so that we end up with exactly the same result.
Versioning with Git Tags
Currently we use the Git commit hash as a Docker image tag, which is hard to remember. A hard-coded tag like latest would have the disadvantage that we could only use one version in parallel at a time. So it makes sense to introduce a versioning concept based on semantic versioning and to integrate it into the pipeline so that the version number is automatically incremented and used as a Docker image tag at the same time. In addition, we create a Git tag for each build so that we can later assign the Docker images to the source code. Since the logic for this is quite complex, we put it in a bash script. Most likely, we will also need it in future CI pipelines, so we create another template azure-pipelines-ci.yaml, which serves as the new base template for azure-pipelines-ci-docker.yaml. Thus we already have a template hierarchy with three levels.
To explain the script – we distinguish between different cases:
- If there is already a version tag directly on the commit for which the pipeline is running, we take it and do not change anything in the version.
- If the pipeline was started from a pull request, we don’t care about the versioning. We only want to find out whether the code and the Docker image can be built and whether the tests are successful. So we just take the last previous version we can find in the Git history and don’t change anything in the version.
- Otherwise, we search the Git history in the past for the closest version tag. If we are on a merge commit, we search in both directions and take the higher version.
- If we cannot find a previous version, we start with version 0.1.
- Otherwise, we increase the version as follows:
- On the master branch, we increase the minor version by 1.
- On the dev branch, we increase the patch level by 1.
- On feature branches we do not increase the version number.
- We also add a suffix to the version number:
- On the dev-Branch -dev.
- On feature branches, a slimmed-down variant of the branch name.
- In Go projects, it is important that the tags are not moved because the Go client caches the complete Git repository locally and stores a signature for each version. Once created, that signature must not change. For this use case we introduce the parameter uniqueGitTags. If this is set to true, we create a separate unique version for each commit by appending another suffix that contains the commit hash and a timestamp.
From the bash script we create a runtime pipeline variable called Version. This is possible with the help of a so-called logging command. To do this, we just have to write a special command to the standard output of the bash script: echo “##vso[task.setvariable variable=<NAME>;isreadonly=true]<WERT>”. This way, the version can be used in subsequent pipeline steps, for example as a Docker image tag. We only push the version tag back to the remote git repository if the pipeline has run successfully, otherwise it should be ignored. To do this, we force Azure DevOps to check out the Git repository completely clean every time a pipeline is executed by setting the clean: true parameter in the checkout: step. We also have to set the parameter persistCredentials: true, otherwise the credentials for the remote Git repository would be deleted after the checkout: step due to security reasons and we would have no permissions to push the Git tag back to the remote Git repository.
The azure-pipeline-ci-docker.yaml is then shortened to the two Docker steps. The file name azure-pipeline-ci.yaml is sufficient for referencing the template, as both files are in the same Git repository. It is important now to change the Docker image tag from ‘$(Build.SourceVersion)’ to ‘$(Version)’ in order to use the version number as the Docker image tag. If we are on the master branch, we set the latest tag next to the version number, as is usual with Docker images. We also add a condition: to the “Push Docker Image” step, which allows the step to be skipped in the context of a pull request. Just as with versioning, in the context of a pull request we are only interested in checking the validity of the code and not in publishing anything.
Now we start the pipeline for the example-python project. For the time being, there is not much to see of the major rebuilding behind the scenes. The build steps Versioning and Create git tag are new. Since we don’t have another version tag yet, we get the following message: Updating from “” to “0.0.1”.
The Docker Push now uses the version number as the tag, as desired:
Unfortunately, pushing the Git tag still fails:
This is because pipelines in Azure DevOps run on behalf of a virtual user called <project name> Build Service (<organisation name>). For security reasons, this user does not have write access to the Git repositories by default. In our case, however, this is desired. To grant the necessary permission, we navigate through the menu via the cogwheel at the bottom left to “Project Settings” à “Repositories” à “Security” tab à “Users” à “<project name> Build Service (<organisation name>)” and change the value for Contribute in the table on the right from Not Set to Allow:
Clicking on Rerun failed jobs in the failed pipeline shows us that we have caught the right setting in the permission jungle of Azure DevOps. The pipeline now runs through successfully:
We also see the new tags in the Git log and in the Docker Hub:
Conclusion
Azure DevOps offers a variety of possibilities to create both simple and complex pipelines. What’s nice is that – unlike other CI/CD systems – you can quickly create simple pipelines without having to worry about many details. This is because the default values and settings are very well balanced between the necessary freedom and security. The more complex features are encapsulated so that they don’t bother you when you don’t need them. Later, you can step by step break out of this borders, which means that you have to write more code yourself and change settings, but you also have more features available. In this way, template hierarchies and Bash scripts can be used to implement even complex scenarios. The reusability and extensibility of pipeline scripts through the template function – especially the insertion of entire step lists – is definitely one of the advantages of Azure DevOps. And if the standard tasks are not enough, there is always the option to switch to the more extensive Bash level. All the necessary tools are also installed there, such as git, go and python. If we use a self-hosted build agent, we could install any other tools, such as jq, make or kustomize. Fortunately, we have not had to worry much about permissions until now. There, we have also many options for configuration, but unfortunately it is sometimes very difficult to understand the effect and the context of the permissions. A plus point here, however, is that Azure DevOps offers a “Quick Fix” button for frequently occurring authorisation problems, which automatically adjusts the permissions on the spot without having to wade through complex menus.
