Capistrano Deploys without Swap

I work on a Ruby on Rails application that is deployed with Capistrano 3 to Amazon Web Services.  We monitor our site performance with New Relic.  About a month ago, we noticed that our deploys were causing a delay in request processing and a drop in Apdex.

Here is an example of what we saw during a deploy.  The blue vertical lines are the deploy times and the green bar is how long a request spent waiting to be processed.


When we dug into it, we found that our servers were going into memory swap during the deploys.  When we deploy with Capistrano, a new Rails process is started to pre-compile the assets.  This pushes the memory usage over the physical memory limits.   Here is the key graphic from New Relic.  Note the swap usage just after 3:00 pm and the disk I/O at the same time.



You can see in the graph above how memory usage drops after the deploy so the solution to this was pretty straight forward: restart the servers first.

We're using Puma as our web server, so we added these lines to our deploy file.  This causes a phased-restart to be sent before the deploy, freeing memory and allowing the asset compilation to have enough memory to run without using swap.  Since Capistrano is based on Rake, its important to re-enable the phased-restart task after its run, otherwise it will only be run once.




Now our deploys run without causing requests to be queued.  What tricks do you have for zero-impact deployments?