Deployment Setup for JRuby Rails App with Puma, Mina, and Moni

Recently, I had my first real experience with deploying a (small) web application. I had to perform a pretty wide research and torture some experienced people with handling the difficulties. I gained knowledge that I consider worth sharing. Let me describe configuration of deployment stack and scripts created for the aforementioned app.

The app is a Ruby on Rails application built with JRuby.

Main characteristics:

• Nginx – web server
• Puma – application server
• Mina – deployment automation tool
• Monit – processes monitoring tool
• JRuby – Ruby implemented in Java and executed on Java Virtual Machine

Nginx was already installed and configured by a sysadmin. I'd like to focus on the Mina, Puma and Monit configuration...

Theory

... but a little theory first. Let me tell you very short and abstract story of how a HTTP server handles requests as well as everything you need to know to understand my deployment setup.

HTTP requests

HTTP requests hit the machine and the first guy they meet is the web server. Most popular are Nginx and Apache. Let's use Nginx as example for further part of the story. Based on the request's target domain and path, web server dispatches each request to proper applications hosted by the current machine. To notify Nginx about application server which should be accessible from the Internet, we have to describe it using a server directive. This directive is used for routing requests to other servers and applicaton server is only a name for another web server with specific responsibilities.

Routed requests meet another web server, specific for the application. Example application server for Rails apps is Puma. It leads requests to the Rails application. Rails holds the connection with the database server, etc. and does its job to respond on a given request.

Naive visualisation

Deployment

When we push new changes to the repository and want them to be visible on the remote server right away, it's a bad idea to log into the remote machine, manually fetch new commits and restart Puma and Rails every time.

To do this automatically, wise people developed some pretty clever software. There are many solutions available, some with many advantages and some disadvantages. We chose Mina, because it's a relatively small, swift tool and we have a small application with small requirements. Additionaly, Mina's deployment scripts are written in Ruby, which we all love and admire.

Mina uses a special directory structure to handle application deployment (from Mina's page):

/var/www/myapp.com/         # The deploy_to path
 |-  releases/              # Holds releases, one subdir per release
 |   |- 1/
 |   |- 2/
 |   |- 3/
 |   '- ...
 |-  shared/                # Holds files shared between releases
 |   |- log/                # Log files are usually stored here
 |   '- ...
 '-  current/               # A symlink to the current release in releases/

During each deploy, Mina creates a new release (one per deploy) and puts it into the releases/ directory. Release is a clone of a git repository, with a specified branch. The most recent release is symlinked as current/. The directory called shared/ holds files shared between releases, in our case:

• config/database.yml file
• config/secrets.yml file
• .env file
• log directory
• tmp directory

You don't have to create this structure manually. Mina has a mina setup command for this. I'll describe customization of this setup script later.

The deployment process consists of basic steps from cloning the git repository, to installing dependencies from a Gemfile, to running migrations and precompiling assets. All this is done in a temporary directory that is copied to releases/ and symlinked as current/ only after a successful finish. If something fails, Mina just removes this temporary directory.

After the deployment finishes, the Puma processes needs to be restarted to pick up recent code changes. To restart Puma, we decided to use a typical and common approach - a monitoring tool that watches for timestamp changes of a specified file. This monitored file is tmp/restart.txt and the easiest way to change it's timestamp is to use the standard unix tool - touch.

Lifecycle

Successful deployment is only half of the jobs that need to be done.

• What happens if a physical machine fails?
• What if our Rails app has a memory leak and crashes?
• What if some admin jobs require a system restart?
• What causes that touch on specified file to restart our application?

To solve these problems, we use a tool called Monit combined with good, old Cron.

Monit is powerful software. In our context, let's call it a monitoring tool. It listens for specified events and performs specified actions under specified conditions. For example, it can periodically check a given file's timestamp and react if an attribute changes (our case). Another use case may be to check if our process acquired too many resources and restart it and warn the admin, for example.

Monit's primary task is to keep our processes alive - start them if they're not running, restart them when necessary.

There are multiple system restart problem solutions. Our first idea was to run a script that starts Monit as an upstart job. Finally, we decided to use Cron and its reboot option to run Monit with a specified config file on system start. We chose this solution because upstart jobs have to be run with root privileges and root becomes the owner, while Cron jobs can be run with a regular user as owner, without adding sudo to process.

Configuration

Puma

Puma is

a modern, concurrent web server for ruby

The config file itself is rather standard. It specifies:

• environment name
• pidfile (file where Puma stores id of its process – PID)
• log files
• allowed number of threads
• port used to expose our app

Here it is:

# config/puma.rb

environment ENV['RAILS_ENV'] || 'production'

pidfile "/home/user/appname/shared/tmp/pids/puma.pid"
stdout_redirect "/home/user/appname/shared/tmp/log/stdout", "/home/user/appname/shared/tmp/log/stderr"

threads 0, 16

# If puma is run with -p then this binding is ignored
bind "tcp://0.0.0.0:3000"

Monit requires a monitored process to run in background. The standard way to do this with Puma is to run it as a daemon, using the daemonize config option. The problem is JRuby. JVM does not implement the Unix fork operation, which is necessary to run the process as a daemon – this is why we don't add it to the Puma config file. I'll describe an alternative solution in the Monit section of this post.

Mina

Mina is

Really fast deployer and server automation tool

Written in Ruby, available as gem. Deployment scripts are written in Ruby too. It has many predefined packages and commands, so we don't have to build a Rails configuration script from scratch – Mina knows how to perform rake db:migrate and all the other commands. Additionally, I use the mina-multistage gem - pure Mina does not let you easily deploy to multiple remote environments with one set of config files.

As I said, you can install Mina manually as a gem. I decided to add it to my project's Gemfile:

# Gemfile
group :development do
  gem "mina", "~> 0.3.7"
  gem "mina-multistage", require: false
end

Config

I divide Mina's configuration file in multiple parts by responsibility.

Requires – Mina has many built-in packages, but we don't need to require them all if we don't use them. My set of requires at the top od config file looks like this:

# config/deploy.rb

require 'mina/multistage'
require 'mina/bundler'
require 'mina/rails'
require 'mina/git'
require 'mina/rbenv'

Requiring these provides all the required tools to correctly deploy and bootstrap a Rails app.

Constants – settings of constants used by the deployment script - for example, where to put paths to the current release, where to find files shared between releases, etc.

# config/deploy.rb

# Path of current release after every deployment
set :app_path, lambda { "#{deploy_to}/#{current_path}" }

# Path of shared directory – where all files shared between deployments are
set :app_shared_path, lambda { "#{deploy_to}/#{shared_path}" }

# List of paths to all shared files – relative to above app_shared_path
set :shared_paths, ['config/secrets.yml', 'config/database.yml', 'log', '.env', 'tmp']

Setup – basically, Mina sets up the entire directory structure on a remote server when you run “mina setup”. You can customise this process – touch all the directories where you'd like to put your custom shared files, create log files, etc.

# config/deploy.rb

task :setup => :environment do
  queue! %[mkdir -p "#{app_shared_path}/log"]
  queue! %[chmod g+rx,u+rwx "#{app_shared_path}/log"]

  queue! %[mkdir -p "#{app_shared_path}/config"]
  queue! %[chmod g+rx,u+rwx "#{app_shared_path}/config"]

  # Directories for puma files
  queue! %[mkdir -p "#{app_shared_path}/tmp/sockets"]
  queue! %[chmod g+rx,u+rwx "#{app_shared_path}/tmp/sockets"]
  queue! %[mkdir -p "#{app_shared_path}/tmp/pids"]
  queue! %[chmod g+rx,u+rwx "#{app_shared_path}/tmp/pids"]
  queue! %[mkdir -p "#{app_shared_path}/tmp/log"]
  queue! %[chmod g+rx,u+rwx "#{app_shared_path}/tmp/log"]

  queue! %[touch "#{app_shared_path}/config/secrets.yml"]
  queue  %[echo "-----> Be sure to edit '#{app_shared_path}/config/secrets.yml'."]

  queue! %[touch "#{app_shared_path}/config/database.yml"]
  queue  %[echo "-----> Be sure to edit '#{app_shared_path}/config/database.yml'."]
end

Deployment – this is where the actual deployment script starts. You need to specify all steps to setup the app from scratch whenever you wish to deploy it:

• clone the git repository, fetch branches and latest commits
• symlink shared files (secrets.yml, etc.) to temporary release directory
• run bundle install
• setup/migrate database
• precompile assets
• link temporary release directory as current release directory
• clean the deployment temporary files
• restart application

# config/deploy.rb

task :deploy => :environment do
  deploy do
    # Put things that will set up an empty directory into a fully set-up
    # instance of your project.
    invoke :'git:clone'
    invoke :'deploy:link_shared_paths'
    invoke :'bundle:install'
    invoke :'rails:assets_precompile'
    invoke :'rails:db_migrate'
    invoke :'deploy:cleanup'

    to :launch do
      # Final step after deployment? Restart puma!
      queue "touch #{app_shared_path}/tmp/restart.txt"
    end
  end
end

This is all I have in my config/deploy.rb file. If you use only Mina, you'd also put the ssh domain and username, git repository and branch name here. I use the mina-multistage gem to configure multiple stages, so environment-specific things like ssh credentials and repository settings need to be put into separate config/deploy/{envname}.rb files. Configuration in config/deploy.rb is common for every environment. Here's a staging environment config example (config/deploy/staging.rb):

# config/deploy/staging.rb

set :user, 'user'
set :domain, '192.168.0.1'
set :deploy_to, '/home/user/appname'
set :repository, '[email protected]:organisation/appname.git'
set :branch, 'develop'

That's it!

For the first configuration, perform mina staging setup and Mina will create all files and the directory structure required for deploying the app. Deployment to a specific remote environment is done by mina staging deploy. Without mina-multistage, these are mina setup and mina deploy. It will ask you for an ssh password and then perform the deployment.

Monit

Monit is

a small Open Source utility for managing and monitoring Unix systems

As I mentioned before, in our context Monit is a process monitoring tool. Basically, it's used to keep processes alive, stop/start/restart them when necessary, or just perform specified actions when specified conditions occur.

Monit requires the monitored process to run in the background. We can't daemonize Puma on JVM, so another solution is required. Finally, we decided to use nohup command. It runs the specified instructions as a background process (like &) but prevents the termination of this process when you logout from the terminal session. When a process runs in the background, we need its PID to control it by sending signals. That's why we configured Puma to store its own PID in a specified pidfile.

If you run a process in background using nohup without knowing its PID, you can't control it. It's no longer accessible through command line signals (ctrl+c to stop). You need to save the PID and then, for example:

kill -9 {PID}

Usage of nohup:

nohup {commands} &

Config

First, let's setup the monitoring process itself. Please notice that it's not a bash/shell script. Monit has its own syntax.

• logs
• pidfile
• running a Monit process as daemon with a given interval - it will sleep all the time, periodically waking up and checking if any reaction is required. We can daemonize Monit, because it's not a Ruby tool limited by the lack of fork with JVM as Puma is.

# .monitrc

# Logs
set logfile /home/user/monit.log
# Pidfile
set pidfile /home/user/.monit.pid
# Run Monit as daemon, waking up every 60 seconds
set daemon 60

Next, let's setup the Puma process monitoring:

• give it some name
• specify the path to its pidfile
• specify the command to start the process
• specify the command to correctly stop the process
• specify constraints for the process to restart (we don't want to kill the host machine), for example:
  • restart if memory usage is higher than 300 MB
  • restart if cpu usage is higher than 95%
• specify the processes group name (if you want to control multiple processes)

# .monitrc

check process app-puma
  with pidfile /home/user/appname/shared/tmp/pids/puma.pid
  start program = "/usr/bin/nohup /bin/bash -c 'cd /home/user/appname/current; PATH=$PATH:/usr/local/bin:/home/user/.rbenv/bin:/home/user/.rbenv/shims RAILS_ENV=staging bundle exec puma -C config/puma/staging.rb >/home/user/appname/shared/tmp/puma.out 2>/home/user/appname/shared/tmp/puma.err </dev/null' &"
  stop program  = "/bin/bash -c 'cd /home/user/appname && if [ -f shared/tmp/pids/puma.pid ]; then cat shared/tmp/pids/puma.pid && echo 'STOP' | xargs kill -9; rm shared/tmp/pids/puma.pid; fi'"
  if totalmem > 300.0 MB for 5 cycles then restart
  if cpu usage > 95% for 5 cycles then restart
  group app

Finally, specify the conditions to manually restart the application. In our case, we want to restart Puma when the timestamp of a specified file changes. That's why we specified the touch #{app_shared_path}/tmp/restart.txt command as the last step of the deployment script:

• specify the file name and path
• specify the condition (timestamp changed)
• specify the command to execute if given condition is true

We don't just start and stop Puma to restart it - we use its hot-restart feature, triggered by sending SIGUSR2 signal to the Puma process (kill -12).

# .monitrc

check file app-restart with path /home/user/appname/shared/tmp/restart.txt
  if changed timestamp
    then exec "/bin/bash -c 'kill -12 `cat /home/user/appname/shared/tmp/pids/puma.pid`'"

That's it!

We keep this config file in /home/user/.monitrc.

Run Monit: monit -c /home/user/.monitrc.

Start Monit after system boot - Cron

Now we have the configuration files.

When we want to start the application, we just start Monit and it handles all the rest. We may want to touch the specified file sometimes, to restart Puma. The Monit process should be up and running all the time.

To start the Monit process after system boot, we use good, old Cron:

# crontab

@reboot monit -c /home/user/.monitrc

Summary

Everything above should allow you to perform your own Ruby on Rails (with JRuby) deployment process.

Every tool described has pretty good documentation.

Their homepages:

• http://nadarei.co/mina/
• http://puma.io/
• https://mmonit.com/monit/

Please add a comment if anything is not clear, you have a better solution or found errors in my code.

Final words

My main purpose was to describe all this information in an accessible and comprehensible way. I hope it will help to understand what's happening on the other side of web development to laics like me.

I'm aware that my case is very simple and my understanding of the whole process is still very general. I encourage you to consider it as the easiest step of the web applications deployment highway.