Easy-Peasy CI/CD With CircleCI
CI/CD is an important topic for modern software development, and my personal favorite tool for this is CircleCI. Simple tool, great user experience, great design, and overall, easy to use.
- Easy YAML config
- Excellent documentation
- Starts building as soon as you add and push your stuff. Takes ~5–10 minutes, or 15 minutes if you’re new (that’s how long it took me to learn the first time I used it).*
*depends on how complex your requirements / workflows are. Most of my side-projects are simple builds and running of unit tests.
It’s easy. I can access specific output, from specific steps, from specific jobs, from specific workflows, for specific repositories. See example below.
All of your jobs run in Docker containers, a Linux VM, or a Mac OS machine (costs extra). If a build fails, you can “Rerun w/ SSH” and then SSH into your environment to diagnose issues. Hot damn!
You get about 1,000 free build minutes and one “container” (i.e., job runner) for free. Always. Even for private projects.
For paid plans, the new pricing charges you based on how much compute you use and a user fee. See pricing plans here.
Gone are the days of trying to setup and administer your own Jenkins or GoCD instance. Onboarding a project is as simple as “adding” it on CircleCI, writing your YAML file, and then pushing it.
Of course, if you require self-hosting then the option does exist.
No more dealing w/ 3rd-party plugins and sh*t documentation. You write all of your configs and functionality in the YAML file, and anything else you need to configure can be done per project or globally in the easy-to-use interface.
Since everything runs inside Docker containers, you can easily write your own Docker images to test with if you require specific setups/dependencies.
You can use their handy API to trigger jobs w/ specific parameters and arguments in other projects. This is great for complex, super-apps w/ inter-project dependencies.
I have no idea if CircleCI hired any UX designers, but the app is just pleasant to use.
It looks great, handles well, it’s fast and responsive… Features and settings are intuitive. Everything is where you expect it to be, even if you’ve never used it before.
Note: I understand for each of these points, there may be one or more other CI/CD apps that are similar or superior. However, I’ve yet to use a solution that has an equal balance of all benefits/features above. If you know of one, please suggest it, and I will check it out.
Also Note: No, CircleCI is not compensating me in any way for this article. I don’t think they even know I’m writing this.
You can setup CircleCI for your own project since I’ve generalized this guide. But if you don’t have one or don’t want to, you can fork or duplicate our example project on GitHub. It’s a Go Hello World program.
Every project built in CircleCI will need to have a folder
.circleci at the top of the repository. Inside that folder, you will have a
config.yml with the configuration for your project’s jobs. Go ahead and make the folder,
touch the file, then open it in your favorite text editor.
All CircleCI config files start with the version. For a while they supported their 1.0 alongside the new 2.0 but 1.0 has been discontinued.
Next, we have the concept of a job, which would be something like building source code or running unit tests. It’s the exact same as a Jenkins job, if that is your background.
“Jobs” is a parent-level dictionary in the YAML and each mapping underneath is a job you define. If you want a nice quick tutorial of YAML, Taurus has written a great YAML intro for their own project.
We will define a job called “build”. Here’s how your file should look now:
GOTCHA #1: CircleCI will automatically expect a job called “build.” If you want to name your job something else, you’ll need to define a workflow to tell CircleCI what jobs to look for. But I’ll go over that later - for now just name it “build.”
Within “build”, you need to define the executor. This is the environment your project builds in. You can choose between a Linux VM (not recommended in most cases), a Mac OS, or from Docker images. Here’s CircleCI’s list of prebuilt Docker images for testing.
The reason the Linux VM (defined as
machine: true in the config) isn’t recommended is because it’s actually a full Ubuntu 16.04 AMI in Amazon that has to start. So it can be a bit slower to boot, and on the free plan, minutes are precious.
Also, it’s Ubuntu 16.04, and apparently there’s no way of changing that. Blegh. Who uses crap made from more than 3 years ago? coughPython2cough
Our example project is a Go “Hello World” program. So we will use their prebuilt Go Docker image.
BEST PRACTICE #1: Make sure you set the exact version for the Docker image. If you just use the latest by default then sometimes you’ll find builds randomly breaking because a language upgraded (i.e., everyone and their mother dropping Java 10 support back in October).
BEST PRACTICE #2: By default, CircleCI will clone your source code into “~/project” inside the environment. However, we recommend setting the directory yourself to something you know and will remember.
In our example project’s case, because it’s a Go project, it must adhere to the Go Workspace requirements. So it’s pretty much mandatory to set the working directory to where it would be in the Gopath.
Within each job, there are steps you define. These will be individual commands being executed to achieve your goals. Many jobs start off by checking out the code, which is what we’ll do. After checking out the code, we will also define a simple run step that simply prints “Hello World!”.
Run steps are just shell execution commands.
That’s good for now. Add this change to git (preferably on master branch if possible) and push.
GOTCHA #2: The reason I suggest master branch is because yes, you can have different CircleCI configs by branch. This has bitten me in the ass before. And CircleCI will execute the different configs differently on separate branches.
All pushed? Ok. All CircleCI jobs are triggered on push*. You don’t have to fiddle with refspecs or tokens or webhooks or any of that garbage - it’s done for you. So good.
*You actually can exercise fine-grained control over what jobs run on what branches and when, etc., or whether they’re triggered at all. I won’t cover that in this starter guide.
3. Let’s get started w/ CircleCI by going to their website.
Click the Green “Start Building for Free” button and sign up using your BB or GH account. It has to be the same account your project is hosted on.
After logging in, click on the left-hand side on “Add Projects”. Then click on “Set Up Project” on the right hand side next to your desired project.
Select Linux then language required for your project. We’ve already written our own file, so we won’t need to use their template. Go ahead and click “Start Building”
After you start building, you’ll be taken to the “Workflow” page. This isn’t relevant to us for now, so go back to the left-hand side and click on “Jobs.” This will show the list of all of your projects, and all of the jobs for those projects. An example is shown below.
The reason my page view has more jobs is because I have more projects I’ve setup. Yours should have just one. Go ahead and click on it.
On this page you’ll see all of the steps the job runs as dropdown sections. Click on any step to see the output to the console.
Pretty nifty huh? My favorite thing about this is the step output separation. It doesn’t just gurgle everything to window.
Congratulations! You’ve got a working CircleCI job. But it’s useless. Let’s have it actually do something.
BEST PRACTICE #3: Generally speaking, if you have larger or more complex projects, you’ll want separate jobs to do separate things (i.e., build vs. test). Despite the fact our example project is dead-ass simple, we will do this just to demonstrate the functionality.
Open up the
config.yml file again, and replace your run command and/or add more run commands with what you need to set up your project for testing. For example, Node.js projects would require
npm i or
yarn commands to download dependencies.
Our example Go project doesn’t have any dependencies, but you would normally need to download them before doing anything so I will add that step. Then I will add another step to build the executable.
Afterwards, another common thing people do is to artifact the files that are generated. This is similar in concept to Jenkins artifacts (if you come from that background). But basically it’s making certain files available for access. I will add a step to artifact the built executable as well.
BEST PRACTICE #4: Don’t ever artifact the entire project folder unless you REALLY need every single file for whatever reason. Artifacting can be really slow and waste precious time depending on how many files there are and how big they are.
BEST PRACTICE #5: If you have a simple shell command you can just use the run: value format. But if you have a complicated step, then you can use the second run step’s format above. You can put a name towards that particular step as a clarifying short description, and you can chain multiple commands in a single step using YAML’s built-in literal block scalar style.
So now I’ve got a proper build job that will build and artifact my build files. If you push your changes and go to the next job that runs, you’ll be able to see the new steps and the file(s) you artifact!
You have a build job…but we didn’t test. Oops. Let’s fix that. Let’s copy-pasta the build job and modify our commands to test instead.
You might have noticed… we’re defining the docker and working directory for both jobs. Isn’t that redundant? And that leads to…
GOTCHA #3: Every job runs in a different, new, fresh container, even if both jobs are for the same repository, even if both jobs use the same Docker image. If you need to share files between jobs, then you need to use workspaces. See the diagram below.
Again, this example project is dead simple and doesn’t need it… but I’ll show you how to use workspaces.
We’ll modify our build job to save all the pertinent stuff in the folder to a workspace. Then our test job will use that workspace, effectively eliminating the need to clone the repo again.
persist_to_workspace takes two parameters:
root, which is either an absolute path or relative path from
paths, which takes a glob to match files to a pattern.
After persisting to a workspace in one, we will then attach the workspace to the other. Our file should look like the following after done:
Now add it and push! If you switch over to CircleCI and look then you’ll see…
build ran great. But
test didn’t run. Uh oh. :(
Remember when I mentioned above how CircleCI by default just looks for “build”? Now that you’ve defined a different job w/ a different name, you’ll need to tell CircleCI about it. This is where workflows come in.
Let’s refer to our diagram again from our friends at CircleCI.
You can see here that in their workflows, they are showing separate “Build”, “Test”, and “Deploy” jobs and how one leads to the next. We’ll now define our workflow to run our “Build” and “Test” jobs.
workflows section to your config. It’s a top-level item just like
jobs is, it will need its own
version parameter, and every child we define in workflows will be a different workflow.
We’ll define one called
build-test, and specify our
test jobs like so:
Now, if you left this alone as-is and pushed it, it would spawn both jobs simultaneously, and whichever one happened to spawn first would get your one free container.
We don’t want that - especially above, we defined the
test job to depend on the workspace persisted by the
build job. So now we’ll want to add that dependency to the workflow.
test job will not run unless our
build job succeeds. This is how you prevent unwanted failures from occurring in jobs down the workflow.
So now your whole file should look something like this:
Go ahead and push and watch the action. You can watch your “Build” job pop up and run as soon as you push, and then your “Test” job will pop up immediately after your “Build” job has succeeded.
This is a lot of functionality straight-out-of-box, and it’s pretty much all that a lot of smaller, simpler projects need.
As your CircleCI configurations get longer and more complex for bigger projects and more functionalities, you’ll want to employ DRY (Don’t Repeat Yourself) to avoid missed changes in multiple places.
CircleCI has built-in executor and command definitions that you can make use of. It does, however, require 2.1 config
Congratulations - you’ve built yourself a CI workflow in just a few minutes! You can see how easy that was and how it’s much more pleasant than trying to tweak Admin settings and read docs for 3rd-party plugins.
- CircleCI has written a nice tutorial to help you automate deployment to various cloud platforms.
- If your project has some more complex requirements needed to run tests and whatnot, it might be in your best interest to build custom Docker images to test in CircleCI.
- Need nightly builds? CircleCI can do that too.
- CircleCI has also written an excellent welcome doc covering various features. We’ve only really touched a couple key items in this article.
- If you don’t like how long it takes to download your dependencies in each job, you may want to go back and add dependency caching.
- If you have some kind of test reporting and it generates static files after each run, try artifacting those and accessing them in the UI…
- As mentioned above, you can start fiddling w/ their API and doing things with it, like triggering builds or downloading artifacts w/ IFTTT…
- If you’ve already done all of the above and have created the most robust CI/CD workflow ever, try to Terraform or Ansible your infrastructure using CircleCI before deploying to it…
- Drink a beer (if you’re of age) because you’re already doing a whole lot more to automate your software development process than a lot of large companies out there, believe it or not. Not everyone can be a badass.
Take care and happy engineering!