Easy-Peasy CI/CD With CircleCI


CI/CD is an important topic for modern software development, and my personal favorite tool for this is CircleCI. Simple tool, great user experience, great design, and overall, easy to use.

So why CircleCI?

1. Extremely user-friendly

  • Easy YAML config
  • Excellent documentation
  • Starts building as soon as you add and push your stuff. Takes ~5–10 minutes, or 15 minutes if you’re new (that’s how long it took me to learn the first time I used it).*

*depends on how complex your requirements / workflows are. Most of my side-projects are simple builds and running of unit tests.

2. Intuitive, nice interface

It’s easy. I can access specific output, from specific steps, from specific jobs, from specific workflows, for specific repositories. See example below.

CircleCI Job Details Page

3. Simple Setup & SSH Capabilities

All of your jobs run in Docker containers, a Linux VM, or a Mac OS machine (costs extra). If a build fails, you can “Rerun w/ SSH” and then SSH into your environment to diagnose issues. Hot damn!

4. Great pricing

You get about 1,000 free build minutes and one “container” (i.e., job runner) for free. Always. Even for private projects.

For paid plans, the new pricing charges you based on how much compute you use and a user fee. See pricing plans here.

5. Cloud solution w/ capability for self-hosting

Gone are the days of trying to setup and administer your own Jenkins or GoCD instance. Onboarding a project is as simple as “adding” it on CircleCI, writing your YAML file, and then pushing it.

Of course, if you require self-hosting then the option does exist.

6. Everything is built-in functionality

No more dealing w/ 3rd-party plugins and sh*t documentation. You write all of your configs and functionality in the YAML file, and anything else you need to configure can be done per project or globally in the easy-to-use interface.

Since everything runs inside Docker containers, you can easily write your own Docker images to test with if you require specific setups/dependencies.

7. Powerful, handy API

You can use their handy API to trigger jobs w/ specific parameters and arguments in other projects. This is great for complex, super-apps w/ inter-project dependencies.

Or if not, you can use their API for various other things, like triggering a job from Slack or fetching build artifacts automatically from IFTTT. It’s just a bunch of REST calls.

8. Excellent user experience

I have no idea if CircleCI hired any UX designers, but the app is just pleasant to use.

It looks great, handles well, it’s fast and responsive… Features and settings are intuitive. Everything is where you expect it to be, even if you’ve never used it before.

Note: I understand for each of these points, there may be one or more other CI/CD apps that are similar or superior. However, I’ve yet to use a solution that has an equal balance of all benefits/features above. If you know of one, please suggest it, and I will check it out.

Also Note: No, CircleCI is not compensating me in any way for this article. I don’t think they even know I’m writing this.

Your first CircleCI build

1. Decide which project you want to set up CircleCI for.

You can setup CircleCI for your own project since I’ve generalized this guide. But if you don’t have one or don’t want to, you can fork or duplicate our example project on GitHub. It’s a Go Hello World program.

2. Let’s write the first part of our config.

Every project built in CircleCI will need to have a folder .circleci at the top of the repository. Inside that folder, you will have a config.yml with the configuration for your project’s jobs. Go ahead and make the folder, touch the file, then open it in your favorite text editor.

1
2
3
4
5
git clone https://github.com/Static-Void-Academy/circleci-hello
cd circleci-hello
mkdir .circleci
touch .circleci/config.yml
nvim .circleci/config.yml # Replace nvim w/ whatever you use

All CircleCI config files start with the version. For a while they supported their 1.0 alongside the new 2.0 but 1.0 has been discontinued.

Next, we have the concept of a job, which would be something like building source code or running unit tests. It’s the exact same as a Jenkins job, if that is your background.

“Jobs” is a parent-level dictionary in the YAML and each mapping underneath is a job you define. If you want a nice quick tutorial of YAML, Taurus has written a great YAML intro for their own project.

We will define a job called “build”. Here’s how your file should look now:

1
2
3
4
# Top of file
version: 2.1
jobs:
build:

GOTCHA #1: CircleCI will automatically expect a job called “build.” If you want to name your job something else, you’ll need to define a workflow to tell CircleCI what jobs to look for. But I’ll go over that later - for now just name it “build.”

Within “build”, you need to define the executor. This is the environment your project builds in. You can choose between a Linux VM (not recommended in most cases), a Mac OS, or from Docker images. Here’s CircleCI’s list of prebuilt Docker images for testing.

The reason the Linux VM (defined as machine: true in the config) isn’t recommended is because it’s actually a full Ubuntu 16.04 AMI in Amazon that has to start. So it can be a bit slower to boot, and on the free plan, minutes are precious.

Also, it’s Ubuntu 16.04, and apparently there’s no way of changing that. Blegh. Who uses crap made from more than 3 years ago? coughPython2cough

Our example project is a Go “Hello World” program. So we will use their prebuilt Go Docker image.

1
2
3
4
5
6
7
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello

BEST PRACTICE #1: Make sure you set the exact version for the Docker image. If you just use the latest by default then sometimes you’ll find builds randomly breaking because a language upgraded (i.e., everyone and their mother dropping Java 10 support back in October).

BEST PRACTICE #2: By default, CircleCI will clone your source code into “~/project” inside the environment. However, we recommend setting the directory yourself to something you know and will remember.

In our example project’s case, because it’s a Go project, it must adhere to the Go Workspace requirements. So it’s pretty much mandatory to set the working directory to where it would be in the Gopath.

Within each job, there are steps you define. These will be individual commands being executed to achieve your goals. Many jobs start off by checking out the code, which is what we’ll do. After checking out the code, we will also define a simple run step that simply prints “Hello World!”.

Run steps are just shell execution commands.

1
2
3
4
5
6
7
8
9
10
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: echo "Hello World!"

That’s good for now. Add this change to git (preferably on master branch if possible) and push.

GOTCHA #2: The reason I suggest master branch is because yes, you can have different CircleCI configs by branch. This has bitten me in the ass before. And CircleCI will execute the different configs differently on separate branches.

So good.

All pushed? Ok. All CircleCI jobs are triggered on push*. You don’t have to fiddle with refspecs or tokens or webhooks or any of that garbage - it’s done for you. So good.

*You actually can exercise fine-grained control over what jobs run on what branches and when, etc., or whether they’re triggered at all. I won’t cover that in this starter guide.

3. Let’s get started w/ CircleCI by going to their website.

Click the Green “Start Building for Free” button and sign up using your BB or GH account. It has to be the same account your project is hosted on.

Sign up on CircleCI's website

After logging in, click on the left-hand side on “Add Projects”. Then click on “Set Up Project” on the right hand side next to your desired project.

Setup 1: Add Projects

Select Linux then language required for your project. We’ve already written our own file, so we won’t need to use their template. Go ahead and click “Start Building”

Setup 2: Configuring Project

After you start building, you’ll be taken to the “Workflow” page. This isn’t relevant to us for now, so go back to the left-hand side and click on “Jobs.” This will show the list of all of your projects, and all of the jobs for those projects. An example is shown below.

The reason my page view has more jobs is because I have more projects I’ve setup. Yours should have just one. Go ahead and click on it.

Jobs List Page Yay it worked!

On this page you’ll see all of the steps the job runs as dropdown sections. Click on any step to see the output to the console.

Pretty nifty huh? My favorite thing about this is the step output separation. It doesn’t just gurgle everything to window.

Congratulations! You’ve got a working CircleCI job. But it’s useless. Let’s have it actually do something.

4. Modify the config.yml file to actually build and artifact things.

BEST PRACTICE #3: Generally speaking, if you have larger or more complex projects, you’ll want separate jobs to do separate things (i.e., build vs. test). Despite the fact our example project is dead-ass simple, we will do this just to demonstrate the functionality.

Open up the config.yml file again, and replace your run command and/or add more run commands with what you need to set up your project for testing. For example, Node.js projects would require npm i or yarn commands to download dependencies.

Our example Go project doesn’t have any dependencies, but you would normally need to download them before doing anything so I will add that step. Then I will add another step to build the executable.

Afterwards, another common thing people do is to artifact the files that are generated. This is similar in concept to Jenkins artifacts (if you come from that background). But basically it’s making certain files available for access. I will add a step to artifact the built executable as well.

BEST PRACTICE #4: Don’t ever artifact the entire project folder unless you REALLY need every single file for whatever reason. Artifacting can be really slow and waste precious time depending on how many files there are and how big they are.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: go get -d ./...
- run:
name: Build the executable
command: |
echo "Other important command here as example"
go build
- store_artifacts:
path: ./circleci-hello # executable

BEST PRACTICE #5: If you have a simple shell command you can just use the run: value format. But if you have a complicated step, then you can use the second run step’s format above. You can put a name towards that particular step as a clarifying short description, and you can chain multiple commands in a single step using YAML’s built-in literal block scalar style.

Ah, I love it when things just work

So now I’ve got a proper build job that will build and artifact my build files. If you push your changes and go to the next job that runs, you’ll be able to see the new steps and the file(s) you artifact!

5. Modify the config.yml file to run a separate test job.

You have a build job…but we didn’t test. Oops. Let’s fix that. Let’s copy-pasta the build job and modify our commands to test instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: go get -d ./...
- run:
name: Build the executable
command: |
echo "Other important command here as example"
go build
- store_artifacts:
path: ./circleci-hello # executable
test:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: go get -d ./...
- run: go test

You might have noticed… we’re defining the docker and working directory for both jobs. Isn’t that redundant? And that leads to…

GOTCHA #3: Every job runs in a different, new, fresh container, even if both jobs are for the same repository, even if both jobs use the same Docker image. If you need to share files between jobs, then you need to use workspaces. See the diagram below.

Source: CircleCI Workflows

Again, this example project is dead simple and doesn’t need it… but I’ll show you how to use workspaces.

We’ll modify our build job to save all the pertinent stuff in the folder to a workspace. Then our test job will use that workspace, effectively eliminating the need to clone the repo again.

The persist_to_workspace takes two parameters: root, which is either an absolute path or relative path from working_directory, and paths, which takes a glob to match files to a pattern.

After persisting to a workspace in one, we will then attach the workspace to the other. Our file should look like the following after done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: go get -d ./...
- run:
name: Build the executable
command: |
echo "Other important command here as example"
go build
- persist_to_workspace:
root: . # Persist current working directory
paths: ./* # Glob. Will persist everything in folder
- store_artifacts:
path: ./circleci-hello # executable
test:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- attach_workspace:
at: .
- checkout # Checks out the source code
- run: go get -d ./...
- run: go test

Now add it and push! If you switch over to CircleCI and look then you’ll see…

build ran great. But test didn’t run. Uh oh. :(

Remember when I mentioned above how CircleCI by default just looks for “build”? Now that you’ve defined a different job w/ a different name, you’ll need to tell CircleCI about it. This is where workflows come in.

6. Define the workflow for all of your jobs.

Let’s refer to our diagram again from our friends at CircleCI.

Source: CircleCI Workflows

You can see here that in their workflows, they are showing separate “Build”, “Test”, and “Deploy” jobs and how one leads to the next. We’ll now define our workflow to run our “Build” and “Test” jobs.

Add a workflows section to your config. It’s a top-level item just like jobs is, it will need its own version parameter, and every child we define in workflows will be a different workflow.

We’ll define one called build-test, and specify our build and test jobs like so:

1
2
3
4
5
6
7
... # rest of file above
workflows:
version: 2
build_test:
jobs:
- build
- test

Now, if you left this alone as-is and pushed it, it would spawn both jobs simultaneously, and whichever one happened to spawn first would get your one free container.

We don’t want that - especially above, we defined the test job to depend on the workspace persisted by the build job. So now we’ll want to add that dependency to the workflow.

1
2
3
4
5
6
7
8
9
... # rest of file above
workflows:
version: 2
build_test:
jobs:
- build
- test:
requires:
- build

Now our test job will not run unless our build job succeeds. This is how you prevent unwanted failures from occurring in jobs down the workflow.

So now your whole file should look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# Top of file
version: 2.1
jobs:
build:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- checkout # Checks out the source code
- run: go get -d ./...
- run:
name: Build the executable
command: |
echo "Other important command here as example"
go build
- persist_to_workspace:
root: . # Persist current working directory
paths: ./* # Glob. Will persist everything in folder
- store_artifacts:
path: ./circleci-hello # executable
test:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello
steps:
- attach_workspace:
at: .
- checkout # Checks out the source code
- run: go get -d ./...
- run: go test
workflows:
version: 2
build_test:
jobs:
- build
- test:
requires:
- build

Go ahead and push and watch the action. You can watch your “Build” job pop up and run as soon as you push, and then your “Test” job will pop up immediately after your “Build” job has succeeded.

This is a lot of functionality straight-out-of-box, and it’s pretty much all that a lot of smaller, simpler projects need.

Ah, I love it when stuff works.

7. One last thing… optimize the config.

As your CircleCI configurations get longer and more complex for bigger projects and more functionalities, you’ll want to employ DRY (Don’t Repeat Yourself) to avoid missed changes in multiple places.

CircleCI has built-in executor and command definitions that you can make use of. It does, however, require 2.1 config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# Top of file
# Common config
executors:
my-go-executor:
docker:
- image: circleci/golang:1.11.1
working_directory: /go/src/github.com/Static-Void-Academy/circleci-hello

commands:
get-go-deps:
description: Get go modules / dependencies
steps:
- run: go get -d ./...

version: 2.1
jobs:
build:
executor: my-go-executor
steps:
- checkout
- get-go-deps # See common
- run:
name: Build the executable
command: |
echo "Other important command here as example"
go build
- persist_to_workspace:
root: .
paths: ./*
- store_artifacts:
path: ./circleci-hello
test:
executor: my-go-executor
steps:
- attach_workspace:
at: .
- checkout
- get-go-deps # See common
- run: go test
workflows:
version: 2
build_test:
jobs:
- build
- test:
requires:
- build

Congratulations - you’ve built yourself a CI workflow in just a few minutes! You can see how easy that was and how it’s much more pleasant than trying to tweak Admin settings and read docs for 3rd-party plugins.

Where do you go from here?

  • CircleCI has written a nice tutorial to help you automate deployment to various cloud platforms.
  • If your project has some more complex requirements needed to run tests and whatnot, it might be in your best interest to build custom Docker images to test in CircleCI.
  • Need nightly builds? CircleCI can do that too.
  • CircleCI has also written an excellent welcome doc covering various features. We’ve only really touched a couple key items in this article.
  • If you don’t like how long it takes to download your dependencies in each job, you may want to go back and add dependency caching.
  • If you have some kind of test reporting and it generates static files after each run, try artifacting those and accessing them in the UI…
  • As mentioned above, you can start fiddling w/ their API and doing things with it, like triggering builds or downloading artifacts w/ IFTTT…
  • If you’ve already done all of the above and have created the most robust CI/CD workflow ever, try to Terraform or Ansible your infrastructure using CircleCI before deploying to it…
  • Drink a beer (if you’re of age) because you’re already doing a whole lot more to automate your software development process than a lot of large companies out there, believe it or not. Not everyone can be a badass.

Take care and happy engineering!