# testing puppet with beaker pt 3 - testing roles and profiles

The third installation in this series cover the use-case of testing roles and profiles.

## Master-vs-Masterless

Prior to this post all our beaker testing has been master-less i.e. using using puppet agent apply. This is perfectly adequate for most use cases when testing modules in isolation but doesn’t always work when testing an internal code base (unless you are masterless there as well then please skip to the next section).

At OpenTable we do use a central puppet master to compile our catalogs. So when testing our puppet roles we wanted to make sure that we were also testing with a master-agent configuration. It is worth mentioning here that if (like us) you are testing windows agents then you are going to need to test with master-agent approach due to the lack of a windows master.

Testing the master-agent configuration means configuring multi-node sets in beaker. There are not many examples of this but the principle is very much the same as the single-node nodeset. Here is an example:

HOSTS:
ubuntu-server-12042-x64-master:
roles:
- master
platform: ubuntu-12.04-amd64
box: ubuntu-server-12042-x64-vbox4210-nocm
box_url: http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210-nocm.box
hypervisor: vagrant
ip: '10.255.33.135'
win-2008R2-std:
roles:
- default
- agent
platform: windows-server-amd64
box: opentable/win-2008r2-standard-amd64-nocm
box_version: = 1.0.0
box_check_update: false
hypervisor: vagrant
user: vagrant
ip: '10.255.33.129'
communicator: bitvise
CONFIG:
log_level: verbose
type: git


In this example you will see that we are specifying different ‘roles’ for each host in the nodeset. What a role is in in this context is a tag for that node that allows us to reference it directly later when running commands on the host. To avoid any further confusion, from this point onwards if I am referring to the role defined in the nodeset file I will call it the ‘nodeset role’ otherwise I am referring the the puppet role provided in the manifest. There are a couple of build-in nodeset roles that Beaker already knows about: master, agent and default. The first two are pretty self explanatory but the last nodeset role - default - is the location where the tests themselves run. In you don’t specify the ‘default’ nodeset role on any of your host definitions then the tests will run on the first host that you specified in in the nodeset file (which in the case of the example above would be wrong).

You may have a more complicated configuration that you wish to test and this allows you to specify arbitrary tags which can be very useful.

We can now use these nodeset roles to configure our master and agent.

In parts 1 and 2 of this series we saw what a basic spec_acceptence file looks like. So let’s start with that:

require 'beaker-rspec/spec_helper'
require 'beaker-rspec/helpers/serverspec'
require 'winrm'

hosts.each do |host|

if host['platform'] =~ /windows/
include Serverspec::Helper::Windows
include Serverspec::Helper::WinRM
end

version = ENV['PUPPET_VERSION'] || '3.5.1'
install_puppet(:version => version)

if host['roles'].include?('master')

... # Install a master

else

... # Install an agent

end
end

RSpec.configure do |c|

c.before :suite do

hosts.each do |host|
c.host = host

if host['platform'] =~ /windows/
endpoint = "http://127.0.0.1:5985/wsman"
c.winrm = ::WinRM::WinRMWebService.new(endpoint, :ssl, :user => 'vagrant', :pass => 'vagrant', :basic_auth_only => true)
c.winrm.set_timeout 300
end
end
end
end


We can see here how we use the host[‘roles’] in order to select the appropriate code-path for configurting each nodeset role. Now let’s move onto how we configure each of those nodeset roles.

## Configuring the master

There are a lot of things that go into building a puppetmaster:

• puppetmaster packages
• hiera backends
• gems for addditional dependencies (eyaml + puppetdbquery)

Now let’s step through our new spec_acceptence file that supports this multi-node environment:

### Deploying the codebase

Stage one is getting our puppet codebase onto the master, which includes all the files, internal modules and anything else we need to get the master up and running. We do this like follows:

files = [ 'environments','facts','hiera','roles', 'profiles', 'keys', 'app_modules', 'auth.conf','autosign.conf',
'fileserver.conf', 'Gemfile','hiera.yaml','Puppetfile'
]

files.each do |file|
scp_to master, File.expand_path(File.join(File.dirname(__FILE__), '..', file)), "/etc/puppet/#{file}"
end

# scp dist modules folder (this excludes stuff like spec and test folders)
dist_modules = Dir["#{dist_modules_root}/*/"].map { |a| File.basename(a) }
dist_modules.each do |module_name|
dist_module_dir = "#{dist_modules_root}/#{module_name}"
copy_module_to(master, :source => dist_module_dir, :module_name => module_name)
end


Here we are selecting all the files that we want and calling the scp_to method which will scp any file or directory to the host of choice, in this case our master.

### The puppetmaster:


...

on master, "apt-get install -y rubygems git"
on master, "apt-get install -y puppet-common=#{version}-1puppetlabs1 puppetmaster-common=#{version}-1puppetlabs1 puppetmaster=#{version}-1puppetlabs1 "
on master, "echo '*' > /etc/puppet/autosign.conf"

...



So we have already installed puppet at a previous stage in our script. At this point we are performing all the steps required to install the puppetmaster: git, rubygems (if on an older distro) and the puppetmaster packages. We also making sure that we auto-signing if configured to save us some pain later on. This step should really be configured as another beaker method that we can just call but for now it is still manual. It is at this point that we have first introduced the “on master” this does what you think it might, it executes the command you pass it onto the host with the nodeset role on ‘master’.

### Set the puppet.conf file:

...

config = {
'main' => {
'server'   => master_name,
'certname' => master_name,
'logdir'   => '/var/log/puppet',
'vardir'   => '/var/lib/puppet',
'ssldir'   => '/var/lib/puppet/ssl',
'rundir'   => '/var/run/puppet'
},
'agent' => {
'environment' => 'vagrant'
}
}

configure_puppet(master, config)

...


Here we are configuring out puppet.conf file, making sure that it includes any customization we might need. This uses a configure_puppet method that we have added to beaker to allow us to do this customization and in this case it is taking the hash to modify the puppet.conf file on the master host.

### Install the required ruby gems:

...

on master, "gem install bundler"
on master, "gem install hiera-eyaml"
on master, "cd /etc/puppet && bundle install --without development"

...


The average production-ready puppetmaster also requires a number of gems to function such as hiera-eyaml, deep_merge any many others depending upon what backends and other custom puppet extensions you have implemented. Here we are installing all our dependencies from the Gemfile we have already put onto the host.

### Installing modules:

...

on master, "cd /etc/puppet && bundle exec librarian-puppet install"

...


The last major step is installing any external modules you have. You may be using librarian-puppet or r10k to do this. In our case it is the former so we go ahead and make sure that our modules directory is full of all the modules we require.

### Networking:

...

master_name = "#{master}.test.local"
on master, "echo '10.255.33.135   #{master_name}' >> /etc/hosts"
on master, "hostname #{master_name}"
on master, "/etc/init.d/puppetmaster restart"

...


This last step is a small hack that you will probably require if you are running on vagrant. It just configures the host file to make sure that it’s hostname if configured correctly from certificate signing to work as expected. This might not be required in your environment and I would try it without first but it’s worth noting anyway.

## Configuring the agent

So if you’ve got to this point well done - most of the hard work is done. Configuring the agent(s) is pretty straightforward in comparison to a puppetmaster and some of the steps are similiar:

### Set the puppet.conf file:

if host['roles'].include?('master')
...
else
agent = host
master = only_host_with_role(hosts, 'master')
agent_name = agent.to_s.downcase
master_fqdn = "#{master}.test.local"
agent_fqdn = "#{agent_name}.test.local"

if agent['platform'] =~ /windows/
config = {
'main' => {
'server'   => master_fqdn,
'certname' => agent_name,
'logdir'   => 'C:\\ProgramData\\PuppetLabs\\puppet\\var\\log',
'vardir'   => 'C:\\ProgramData\\PuppetLabs\\puppet\\var\\lib',
'ssldir'   => 'C:\\ProgramData\\PuppetLabs\\puppet\\var\\lib\\ssl',
'rundir'   => 'C:\\ProgramData\\PuppetLabs\\puppet\\var\\run'
},
'agent' => {
'environment' => 'vagrant'
}
}
else
config = {
'main' => {
'server'   => master_fqdn,
'certname' => agent_fqdn,
'logdir'   => '/var/log/puppet',
'vardir'   => '/var/lib/puppet',
'ssldir'   => '/var/lib/puppet/ssl',
'rundir'   => '/var/run/puppet'
},
'agent' => {
'environment' => 'vagrant'
}
}
end
...

configure_puppet(agent, config)
end

...


Again here we are again using the configure_puppet method, this time change the puppe.conf file on the agent.

As you can see we are catering for both windows and linux hosts here. We are also making sure that the certname and server are defined properly and match what we set-up for the master so that auto-signing works correctly.

## Testing the Role

At this point we have now does all our prerequisites and we can spin up two machines to test against - 1 master and 1 agent. But when we are testing a role what is it that we actually want to test and why is this not covered in earlier less-expensive puppet-rspec unit tests?

Well there are 3 key things that we wanted to test:

1. Idempotence This is pretty straight forward to test. Beaker provides a method run_agent_on that will run the puppet agent on a given host. This means we can test idempotency like this:

run_agent_on(agent, :catch_failures => true)
expect(run_agent_on(agent, :catch_failures => true).exit_code).to be_zero


2. Interaction of multiple modules and profiles This is the big motivator - we want to test that and make sure that the combinations of profiles that we are applying work together and do not either break the catalog or operate in a non-idempotent way. We are also gaining the ability to test that updates in modules (many of which are external from the puppet forge) do not break our roles in any way.

3. Postivie/Negative testing - do we clean up after ourselves if we remove something. This is not something that is often considered very often, particularly in a world where machines are torn down and re-build so often but there still exists a use-case where this is not always possible and we want to make sure that our roles and manifests are not littering our machines unnecessarily.

Below is a full example of one of our linux profiles:

require 'spec_helper_acceptance'

describe 'linux_base_profile', :if => fact('osfamily').eql?('Debian') do
context 'linux base profile' do
it 'should should run successfully' do

agent = only_host_with_role(hosts, 'agent')
master = only_host_with_role(hosts, 'master')

pp = "node \"#{agent}\" { include profiles::linux::base }"
on master, "echo '#{pp}' >> /etc/puppet/manifests/site.pp"

run_agent_on(agent, :catch_failures => true)
expect(run_agent_on(agent, :catch_failures => true).exit_code).to be_zero
end

context 'installation of ops tools' do

describe package('sysstat') do
it { should be_installed }
end

describe package('iotop') do
it { should be_installed }
end

describe package('ngrep') do
it { should be_installed }
end

describe package('lsof') do
it { should be_installed }
end

describe package('unzip') do
it { should be_installed }
end
end

context 'managing puppet version' do

describe file('/etc/apt/sources.list.d/puppetlabs.list') do
it { should be_file }
it { should be_owned_by 'root' }
it { should be_mode 644 }
end

describe package('puppet') do
it { should be_installed.by('apt').with_version('3.6.1-1puppetlabs1') }
end
end

context 'manage sshd configuration' do

describe process("sshd") do
it { should be_running }
end

describe port(22) do
it { should be_listening }
end

describe file('/etc/ssh/sshd_config') do
its(:content) { should match /PermitRootLogin no/ }
its(:content) { should match /PasswordAuthentication yes/ }
its(:content) { should match /UseDNS no/ }
end
end
end
end


We have a lot of roles and profiles that we would like to test. As you might imagine this could get quite verbose and repetitive pretty quickly. We are currently building up shared_contexts for each of our profiles which we can then wrap up into roles to easily reflect our roles/profiles structure in the main codebase.

## Summary

We are just at the very beginning of this journey with Beaker. As well as testing all our modules we are looking to scale our to test the roles and profiles in our whole code base. These examples here are how we are doing it at the moment for our mixed-platform environment. We will continue to expand upon it and build it into our pipeline. At this moment we are looking to expand beyond vagrant and run these against AWS instances but perhaps that is for the next post …