ngOfficeUIFabric - How We Do It - Continuous Delivery… Automate All The Things!

Monday, March 7, 2016 9:34 AM
Microsoft MVP Logo

Check out the first post in this series: ngOfficeUIFabric - How We Do It - Keeping a Clean Commit Log

Check out the first post in this series: ngOfficeUIFabric - How We Do It - Handling Pull Requests

TL;DR

We want our library, ngOfficeUIFabric, to be available to anyone using their preferred packaging framework. This includes NPM, Bower, NuGet as well as a CDN. The process of cutting a new release & pushing to these different distribution targets is tedious and includes multiple steps. If you miss one, you can really screw up things for one of your users. I wanted to put in place a process, inspired by the Angular Material project, that was as automated as possible.

How automated is it? All we have to do is bump the version number + update the change log in the master branch. Once I push it to the main ngOfficeIUFabric repo on GitHub the deployment to all four of the package distribution services mentioned above happens automatically. The best part: it’s fully automated & transparent so you can see how we did it… so let’s get to it…

Overview of the ngOfficeUIFabric CD Process

Before I jump into the weeds, let me explain the process from a high level. This sequence diagram may look complicated, but in reality is quite simple:

The orange boxes {NPMJS.org} & {NuGet.org} represent the package distribution targets we have to factor into the process with every release. Before I mentioned we also use a CDN & Bower… in the next section I’ll explain why we don’t have to concern ourselves with those two with every release.

GitHub Repo’s Supporting the Release Distributions

Our entire code hosting & distribution process is comprised of three repos in our GitHub organization ngOfficeUIFabric:

  • github.com / ng-OfficeUIFabric - This is the main repository where we keep the source for the library. There are two primary branches here:
  • master: The master branch is an exact copy of the source that is currently released. Therefore if you are currently using our library using any of the distributions and you want to see what the code looks like for the implementation, you can go to this library and see exactly that (or jump to previous versions using the tags / releases in GitHub).
  • dev: The dev branch is our staging… think of it as the nightly build with the most current stuff in it. Contributors submit PRs to this branch. Eventually master will be rebased on this branch to get the latest stuff.
  • github.com / package-ngofficeuifabric - This repo is used as the primary distribution for the library. It powers NPM & Bower package managers as well as CDNJS.
  • github.com / nuget-ngofficeuifabric - This repo powers the NuGet package manager distribution for the library.

Continuous Integration (CI) Services Facilitating the Release Process

At the present time we are using three different continuous integration (CI) services to handle builds, tests and release distribution. We may move to two in the future eliminating TravisCI, but that’s still undecided. For all services we’re using the free tiers for open source projects.

Understanding the Package Manager / Distribution Options

As I said above, we use four package manager / distribution options to satisfy everyone. Personally I’d recommend using the CDN option for performance & the NPM option as secondary if you want local copies regardless of the project you use, but we provide two more options for completeness.

CDNJS - cdnjs.com

This is a community-driven CDN. Using a CDN is preferred for performance reasons in your web application.

Getting your library into CDNJS is a piece of cake… you simply fork their repo, add your library (like we did here: 044eab7f) & submit a PR (PR#6760) and you’re in provided you meet their requirements.

Better yet, if you add a section to your package.json file, CDNJS will periodically check your GitHub repo’s tags for new versions and automatically ingest them into the CDN for you. You can read more about this process in the CDNJS docs here: autoupdate.

Once you have your library setup with CDNJS and configured for auto update, you don’t have to think about it anymore provided new versions are tagged in your GitHub repo. Therefore, you won’t see this option mentioned in the rest of this post as it’s automatic.

Bower - bower.io

Another package manager is Bower. It seems to be losing steam in popularity to NPM, but it’s still very relevant. Bower has a centralized public registry that library authors can publish to. When you want to include a library from Bower in your project, Bower pulls the code straight from the library’s repo.

Therefore, once your library is added to the registry, no need to republish new versions as they will get pulled down automatically from GitHub or whatever code hoster you are using. Therefore, you won’t see this option mentioned in the rest of this post as it’s automatic.

NPM

This is the defacto package manager today. To publish to it you need to have the code in a public location, which we do at package-ngofficeuifabric, and run the NPM CLI… that’s it.

NuGet

This is the Microsoft defacto package manager today. To publish to it you either use the web browser or use the nuget CLI.

ngOfficeUIFabric’s CD Process In Depth

Now that covered the repos involved, the targets & services we use & why, let’s look at the process in detail. I’m going to go through the diagram in a bit more detail as it was tedious to put together trying to read the code from what other projects did, so hopefully this speeds your ramp up process.

Step 1: Create a New Version

Creating a new release is the simplest part and best of all, it’s the only thing we manually do. As explained above, the master branch is what the releases look like. So, once we are ready to cut a release, we go into the dev branch and do two things:

Once that’s done in the dev branch, we commit the change with the commit message docs(release): #.#.# and push it to GitHub. Then we jump over to the master branch, rebase it to the tip of dev & push to GitHub (step 1 in the picture above).

This push starts the entire release deployment process.

Step 2: Start Release Process & Update Main Repo

The push to master in the last step triggers a webhook (#2 in the picture above) that notifies TravisCI. This is where we kick off the automated process by running a shell script.. well… sort of…

When we are ready to start the release, from my laptop, I hope a prompt and run the following shell script from the root of the locally cloned repo’s master branch:

sh build/scripts/release.sh --git-push-dryrun=false

That will run our release script which is #3 in the picture above. The script is well documented if you take a look, but let me take a minute to explain it with extra comments using this gist:

if you can’t see the code, look at the post on my site, not in a feed reader

As you can see this script does a few checks and then creates a new tag for the ng-officeuifabric repo. Once that’s done, it runs two other scripts: one that updates the package-ngofficeuifabric repo & one that updates the nuget-ngofficeuifabric repo.

Step 3: Update package-ngofficeuifabric & Publish to NPM, Bower & CDNJS

Let automation flow! We saw how #3 above was kicked off in the previous step. Recall that #3 involved running the script release.sh which calls update-package-repo.sh.

This script (which you can see below) first clones a fresh copy of the pacakge-ngofficeuifabric repo & copies the two compiled ngOfficeUIFabric libraries from the parent release.sh script as well as the changelog.md file into the repo, overwriting the previous versions. It then uses Node.js to execute a JavaScript file that updates the version numbers for the library and all dependencies in the package.json & bower.json files (I found it much easier to use JS to update the JSON than using a shell script).

Now that the repo has been updated to a new version, I then save all changes by running git add -A and commiting, tagging & pushing the changes to the remote package-ngofficeuifabric repo (#4 in the image above). This simple update takes care of updating the Bowser & CDNJS release targets, but we need to publish to NPM, which is a manual command line task.

The package-ngofficeuifabric is hooked up to CircleCI which will do the publish to NPM once the master branch is updated in GitHub. That just happened so taht will trigger a webhook to notify CircleCI (#5 above) which, using the circle.yml file here, will publish the repo to NPM.

Check it out:

if you can’t see the code, look at the post on my site, not in a feed reader

Step 4: Update nuget-ngofficeuifabric & Publish to NuGet

Last step, update NuGet. Again, from step 3 above, the release.sh script ran two scripts. One updated the package-ngofficeuifabric repo as explained in step 4. The other, update-nuget-repo.sh does the same thing, but for the nuget-ngofficeuifabric repo. The code is mostly the same, the difference is that we have to update an XML file with the new versions.

Just like in step 4, once that’s done the changes to the locally cloned repo are saved, tagged and pushed to GitHub. The repo is hooked up to AppVeyor which is used to build a new NuGet package using the nuget CLI (#7 in the image above) and publish it to the public NuGet registry. That’s controlled by the appveyor.yml file here.

Closing

And that’s it! Usually within a few minutes of starting the release process we have new releases published to NPM, Bower & NuGet. CDNJS will pickup the changes from the new tag in the package-ngofficeuifabric repo within an hour when they run their scan.

The goal here was to make things as automated as possible. Today I am using one manual command to start the publishing process.

Looking Forward

This is how we do releases today, and it works great. There are a few things we are looking to do to extend this process. Here’s what’s on my backlog, in order of priority:

  • Standardize on CircleCI: I much prefer their interface to TravisCI and they seem a bit more reliable. We already have to use two CI/CD tools in CircleCI & AppVeyor so I’d like to remove Travis from the equation. That requires a bit more testing on my fork before I do that.
  • Fully Automate the Release: I’m waiting to move to CircleCI (unless I realize that Travis is better for us) before adding one more step when we push to master on ng-officeuifabric. This step will kick off the release automatically when we update master… but I don’t want to do this in Travis only to have to reapply to CircleCI.
  • Automatic Changelog Creation: Today we manually update it, but it can automated. That’s on my list of things to do using the conventional-changelog.
  • Automatic Demo Site Creation: Lots of work to do here, but there’s no reason each release can’t trigger the update of our demo site. Got some slick goals & ideas here so once I get this done, I’ll write about it.
  • Automatic Documentation Creation: Same as the last one… lots of ideas & goals, but more work and research required. I’ll write about it when it’s done.
comments powered by Disqus