Git Lab CI for docker build enabled! You can enable it using .gitlab-ci.yml in your project. Check file template at https://gitlab.bio.di.uminho.pt/snippets/5

README.md 10.6 KB
Newer Older
1
# Merlin-Continuous-Integration tutorial
2

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
This tutorial showcases how to implement a few simple tests for merlin classes and plugins. 
In this example, we have tested a small class, "RulesParser" from the merlin-utilities plugin

Outline:

* A simple Java 8 application with tests
* Unit tests written with [JUnit 5](https://junit.org/junit5/)
* A Maven build that puts it all together
* Deploy maven build to nexus repository with GitLab CI/CD tool

## Running the tests

* To run the unit tests locally, call `mvn test` from the project root directory

## Conventions

This example follows the following basic conventions:

| | unit test |
| --- | --- |
| **resides in:** | `src/test/java/*Test.java` |
| **executes in Maven phase:** | test |
| **handled by Maven plugin:** | [surefire](http://maven.apache.org/surefire/maven-surefire-plugin/) | [failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/) |

# How to deploy Maven projects to Nexus with GitLab CI/CD

## Introduction

In this project, we show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/)
to build a [Maven](https://maven.apache.org/) project and deploy it to [Nexus](https://nexus.bio.di.uminho.pt/).

You'll use this project as example to test Gitlab CI using Maven and our Nexus.

We assume that you already have a GitLab account on [gitLab.bio.di.uminho.pt](http://gitlab.bio.di.uminho.pt/), and that you know the basic usage of Git and [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/).
We also assume that the Nexus is available and reachable from the internet, and that you have valid credentials to deploy on it. (Please ask for your Nexus credentials from our Servers/Resources Administrators) 

## Create the Project

First, you need an application to work with. 
In this specific case we will use a small sample of the merlin-utilities project. 
However, it could have been any Maven application. 
This application will become the dependency that we will package it up and deploy in Nexus repository, 
so that it can be available to other projects.

### Prepare the dependency application

For this tutorial, you can clone this very same project and its hierarchy. 

1. Log in to your GitLab account
1. Create a new project by selecting **Import project from > Repo by URL**
1. Add the following URL:

   ```plaintext
Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
56
   https://gitlab.bio.di.uminho.pt/tutorials/merlin-ci-cd-tutorial
57 58 59 60 61
   ```

1. Click **Create project**

This application comprehends one basic java class, RulesParser, in the main directory. 
Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
62
Then, there is one JUnit, RulesParserTest, for the RulesParser in the test directory.
63 64 65 66 67 68
Thus, the project structure is simple.

Additionaly, there is also one important resource to manage configurations and dependencies of our application:

- `pom.xml`: project object model (POM) configuration file

Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
Maven plugins for testing should be specified in the project's `pom.xml` file as follows:

```xml
<project>
   ...
    <build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>3.0.0-M5</version>
			</plugin>
		</plugins>
	</build>
    ...
</project>
   ```

Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
### Configure the Nexus deployment

### For Snapshots

Commonly, `Snapshots` are compiled java code that was not tested manually. Their deployment into Nexus encompasses either 
changing the `Snapshot` version or deploying the same version. It is worth noting that the compiled `.jar` files
will be deployed into Nexus, either appending the `.jar` file to the current version directory or creating a new one for
the new version.

The `Snapshot` version is declared in `pom.xml` as follows:

```xml
   <project>
	<modelVersion>4.0.0</modelVersion>

	<groupId>pt.uminho.ceb.biosystems.merlin</groupId>
	<artifactId>merlin-project</artifactId>
	<version>4.0.6.1-SNAPSHOT</version>
        ...
    </project>
   ```

This version can be incremented or let as it is, depending on the developer's scope.

### For Releases

Commonly, `Releases` are compiled java code that was manually tested and is ready for the client. Their deployment into Nexus is ONLY possible
if no version of the `Releases` being deployed exists. In other words, Nexus does not append to an existing directory as it does for `Snapshots`.
On contrary, the deployment is not performed. Hence, please increment the version of the release in the project's `pom.xml` file.


The `Release` version is declared in `pom.xml` as follows:

```xml
   <project>
	<modelVersion>4.0.0</modelVersion>

	<groupId>pt.uminho.ceb.biosystems.merlin</groupId>
	<artifactId>merlin-project</artifactId>
	<version>4.0.6-RELEASE</version>
        ...
    </project>
   ```

#### Generic specifications (both for Releases and Snapshots)

133 134 135 136

The application is ready to use, but you need some additional steps to deploy it in Nexus:

1. Log in to Nexus with your user's credentials.
Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
137
1. From the main screen, click on the `Snapshots` and `Releases` item in the **Repositories** panel.
138 139 140 141 142 143 144 145
1. Copy the configuration snippet under the **Summary** Tab to a text editor.
1. Change the `url` value to have it configurable by using variables.
1. Copy the snippet in the `pom.xml` file for your project, just after the
   `dependencies` section. The snippet should look like this:

   ```xml
   <distributionManagement>
     <repository>
Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
146 147 148 149
       <id>releases</id>
       <url>${MAVEN_REPO_URL}/content/repositories/releases</url>
     </repository>
     <snapshotRepository>
150 151
       <id>snapshots</id>
       <url>${MAVEN_REPO_URL}/content/repositories/snapshots</url>
Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
152
     </snapshotRepository>
153 154 155
   </distributionManagement>
   ```

Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
156

157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264
Now it is time to configure the authentication data. 
Maven searches for the authentication data in a file called `settings.xml`.
The settings file must be in a `.m2` subdirectory within the project root. 

Note that, you also have the settings file in a `.m2` subdirectory within the user homedir to run maven locally.
However, since this tutorial is designed to use a runner to automatically deploy the application, you
should create this file in the project root directory:

1. Create a folder called `.m2` in the root of your repository
1. Create a file called `settings.xml` in the `.m2` folder
1. Copy the following content into a `settings.xml` file:

   ```xml
   <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"
       xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
     <servers>
       <server>
         <id>releases</id>
         <username>${MAVEN_REPO_USER}</username>
         <password>${MAVEN_REPO_PASS}</password>
       </server>
       <server>
         <id>snapshots</id>
         <username>${MAVEN_REPO_USER}</username>
         <password>${MAVEN_REPO_PASS}</password>
       </server>
     </servers>
   </settings>
   ```

Username and password will be replaced by the correct values using the environment variables defined in the next step.

### Configure GitLab CI/CD 

Now, it is time to set up [GitLab CI/CD](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) to automatically build, test and deploy the dependency!

GitLab CI/CD uses an yaml file in the repository's root, named `.gitlab-ci.yml`. 
The yaml file contains the configurations for the jobs that will be executed by the runners. 
You can read more about this file in the [GitLab Documentation](https://docs.gitlab.com/ee/ci/yaml/README.md).

First of all, remember to set up variables for your deployment. Navigate to your project's **Settings > CI/CD > Environment variables** page
and add the following ones (replace them with your current values, of course):

- **MAVEN_REPO_URL**: `https://nexus.bio.di.uminho.pt` (our Nexus URL)
- **MAVEN_REPO_USER**: `nexususer` (your Nexus username)
- **MAVEN_REPO_PASS**: `nexuspassword` (your Nexus Password)

Now it's time to define jobs in `.gitlab-ci.yml` and push it to the repository:

```yaml
image: maven:3.6-jdk-8-slim

variables:
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
  MEVEN_DEPLOY_OPTS: "-DskipTests"
  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"

cache:
  paths:
    - .m2/repository/
    - target/

build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS compile

test:
  stage: test
  script:
    - mvn $MAVEN_CLI_OPTS test

deploy:
  stage: deploy
  script:
    - mvn $MAVEN_CLI_OPTS deploy $MEVEN_DEPLOY_OPTS
  only:
    - master
    - dev
```

The runner uses the latest [Maven Docker image](https://hub.docker.com/_/maven/),
which contains all tools and dependencies required to run the jobs.

Regarding configuration and dependencies, the environment variables are set to instruct Maven to use the project `homedir`, namely the repository root, 
rather than the user's homedir.

Caching the `.m2/repository folder` (where all the Maven files are stored), and the `target` folder (where our application will be created), is useful for speeding up the process
by running all Maven phases in a sequential order, therefore, executing `mvn test` will automatically run `mvn compile` if necessary.

Both `build` and `test` jobs leverage the `mvn` command to compile the application and to test it as defined in the test suite that is part of the application.

Deploy to Nexus is done as defined by the variables we have just set up.
The deployment occurs only if we're pushing or merging to `master` branch, so that the development versions are tested but not published.

Done! Now you have all the changes in the GitLab repository, and a pipeline has already been started for this commit. In the **Pipelines** tab you can see what's happening.
If the deployment has been successful, the deploy job log will output:

```plaintext
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.983 s
```

>**Note**:
the `mvn` command downloads a lot of files from the internet, so you'll see a lot of extra activity in the log the first time you run it.

Fernando João Pereira da Cruz's avatar
Fernando João Pereira da Cruz committed
265
Yay! You did it! Check Nexus to confirm a new artifact available in the `Snapshots` or `Releases` repository.
266 267 268 269 270 271 272 273 274 275 276

>**Note**:
Be aware that Nexus doesn't allow replacements of the same version of a java package release without Nexus Administrator authorization!
   

## Conclusion

In this article we covered the basic steps to use the Nexus Maven repository to automatically publish and consume artifacts.

A similar approach could be used to interact with any other Maven compatible Binary Repository Manager.
Obviously, you can improve these examples, optimizing the `.gitlab-ci.yml` file to better suit your needs, and adapting to your workflow.