Deployment done right – Teletraan

Bálint Csergő

August 03rd, 2016

Tell me, how do you deploy your code? If you still GIT pull on your servers, you are surely doing something wrong. How will you scale that process? How will you eliminate the chance of human failure? Let me help you; I really want to. Have you heard that Pinterest open sourced its awesome deployment system called Teletraan? If not, read this post. If yes, read it still, and maybe you can learn from the way we use it in production at endticket.

What is Teletraan?

It is a deployment system that consists of 3 main pieces:

  • The deploy service is a Dropwizard-based Java web service that provides the core deploy support. It's actually an API, the very heart and brain of this deployment service.
  • The deploy board is a Django-based web UI used to perform day-to-day deployment works. Just an amazing user interface for Teletraan.
  • The deploy agent is the Python script that runs on every host and executes the deploy scripts.

Is installing it a pain? Not really; the setup is simple. But if you're using Chef as your configmanagement tool of choice, take thesesince they might prove helpful: chef-teletraan-agent, chef-teletraan.

Registering your builds

Let the following snippet speak for itself.

import requests
headers = {'Content-type': 'application/json'}
r = requests.post("%s/v1/builds" % teletraan_host, data=json.dumps({'name': teletraan_name, 'repo': name,
                                                         'branch': branch, 'commit': commit,
                                                         'artifactUrl': artifact_base_url + '/' + artifact_name,
                                                         'type': 'GitHub'}), headers=headers
                  )

I've got the system all set up.What now?

Basically, you have to make your system compatible with Teletraan. You must have an aritfact repository available to store your builds, and add deploy scripts to your project. Create a directory called "teletraan" in your project root. Add the following scripts to it:

  • POST_DOWNLOAD
  • POST_RESTART
  • PRE_DOWNLOAD
  • PRE_RESTART
  • RESTARTING

Although referred as Deploy Scripts, they can be written in any programming language as long as they are executable.

Sometimes, the same build artifact can be used to run different services based on different configurations. In this case, create different directories under the top-level teletraan with the deploy environment names, and put the corresponding deploy scripts under the proper environment directories separately. For example:

  • teletraan/serverx/RESTARTING
  • teletraan/serverx/POST_RESTART
  • teletraan/servery/RESTARTING

What do these scripts do?

The host level deploy cycle looks the following way:

UNKNOWN->PRE_DOWNLOAD->[DOWNLOADING]->POST_DOWNLOAD->[STAGING]->PRE_RESTART->RESTARTING->POST_RESTART->SERVING_BUILD

Autodeploy?

You can define various environments. In our case, every successful master build ends up on the staging cluster automatically. It is powered by Teletraan’s autodeploy feature. It works nicely. Whenever Teletraan detects a new build, it gets automatically pushed to the servers.

Manual control

We don’t autodeploy the code to the production cluster. Teletraan offers a feature called "Promoting builds". Whenever a build proves to be valid at the staging cluster (some Automated end-to-end testing; and of course, manual testing is involved in the process) the developer has the ability to promote a build to production.

Oh noes!Things went wrong. Is there a way to go back?

Yes there is a way! Teletraan gives you the ability to roll back any build which happens to be failing. And this can happen instantly. Teletraan keeps a configurable numberof builds on the server of every deployed project; an actual deploy is just a symlink being changed to the new release.

Rolling deployments, oh the automation!

Deploy scripts should always run flawlessly. But let's say they do actually fail. What happens then? You can define it. There are three policies in Teletraan:

  • Continue with the deployment.Move on to the next host as ifnothing happened.
  • Roll back everything to the previous version. Make sure everything is fine; it's more important than a hasty release.
  • Remove the failing node from production. We have enough capacity left anyway, so let's just cut off the dead branches!

This gives you all the flexibility and security you need when deploying your code to any HA environment.

Artifacts?

Teletraan just aims to be a deployment system, and nothing more. And it does its purpose amazingly well. You just have to notify it about builds. Also you just have to make sure that your tarballs are available to every node where deploy agents are running.

Lessons learned from integrating Teletraan into our workflow

It was acutally a pretty good experience even when I was fiddling with the Chef part. We use Drone as our CI server, and there was no plugin available for Drone, so thathad to be done also. Teletraan is a new kid on the block, so you might have to write some lines of code to make it apart of your existing pipeline. But I think that if you're willing to spend a day or two on integrating it, it will pay off for you.

About the author

BálintCsergő is a software engineer from Budapest, currently working as an infrastructure engineer at Hortonworks. He loves Unix systems, PHP, Python, Ruby, the Oracle database, Arduino, Java, C#, music, and beer.

comments powered by Disqus