winops 2015

Last week I was kindly invited to participate in the first WinOps (http://winops.org/) conference in London.

I have been skirting around the edges of the microsoft world for some time now and when I saw this conference appear I thought it would be a good opportunity to chat with those individuals whom are fully immersed in windows but may have a different devops journey to my own.

I will be honest, I was fully expecting to walk into a room full of suited business types looking to buy the devops and discuss ROI. Thankfully that trope was only in my own head (aren’t they always!) and I what I really discovered was a group of engineers very like myself but from very different backgrounds.

Most of the attendees were from large enterprises with diverse environments (not all were 100% windows). This meant that they were coming to the discussion with problems like: big legacy systems, enterprise tooling and licensing agreements - something for which I have up to this point managed to only view from a safe distance. They were also coming from a background of being new to devops and some were still struggling with continuous delivery pipelines. This reminded me that for all the advances that have been made in the devops discussions there are always going to be people who are new so we can’t always make assumptions of knowledge - something we often do when in a bit of an echo chamber.

I won’t go through all the talks - they were all fantastic and a great overview of the current state of the universe for windows (thanks Stephen Thair and John Rakowski)

I will point out one however, the talk from JustEat’s Peter Mounce, who discusses their infrastructure and the journey from hand-build machines in their own datacenter to windows servers in AWS that are regularly destroyed. This was a great talk to give to this audience because it demonstrated that even in the world of windows, where the perception is that everything is a little bit more difficult, that things like this are actually possible. I am sure that we (OpenTable) will be looking more and more at this as we to move to the path of trying to destroy more of our infrastructure on a regular basis. (note: I am not going to say immutable here - more on that some other time)

After these though provoking talks and some more interesting discussion over lunch it was time for me to step up and try and sound like I know what I am talking about.

Firstly I was invited to be a member of the panel discussion Infrastructure as Code. On this panel were representatives of DSC,Chef,Puppet and Ansible. I was there to represent the non-vendor view (my preference towards puppet was made clear). Most of the questions veered towards to more basic IaC principles: Why manage as text files (code)?, Thoughts on SCCM?, Where do I start? I think the most personally interesting part of this panel was that all of the panel members seemed to agree - something that was unexpected on my part.

Next, I was unexpectedly asked to also fill in some space on the CI/CD for Windows panel. This was again about some core principles: deployments and packaging. While this was basic, it’s been a few years since my job has been focused on this so it was good to get back to basics and also being forced into justifying my opinions. We had a great chat about packaging on windows and the options available, as well as how to manage database deployments and orchestration. It’s great to know that there has been a long of work done here in the windows space and it’s all a lot better than when I was going it 2-3 years ago.

Over all I found this to be a great little conference with a diverse range of people with lots of challenges. Most of the companies represented might be at the start of their devops journey but there are a lot of passionate people focussed on helping improve this space. Now that microsoft is also embracing this, most of these companies and engineers feel much more comfortable in moving forward. I think it’s going to be an exciting few years ahead for those in the windows world.

Note: you might also want to read this great post from Hannah Foxwell who beat me to it in reviewing: http://hannahfoxwell.net/2015/09/25/review-of-winops-2015/

Another talk on windows

I gave another talk this week to the folks over at Ve Interactive.

It’s been a while since I last did a talk and I enjoyed it alot. Lots of interesting discussions and questions.

Slides:

Video:

[Quick Update] - 23-Mar-2015

Ok, so you’ve seen the date of this post. I’m not exactly as regular with these posts as I would like but it seems that I do in fact still have things to write about.

#### Foreman One of the big things that I have been working on recently has been in puppetizing the foreman infrastructure used at OpenTable. Thankfully this is quite a mature community and there has already been a lot of work done however one limitation that I found was that I wanted to actually use puppet to configure foreman after I had installed it.

For anyone that has actually spent some time with foreman you will know that entity model is a little complex and it can take a long time to configure with settings specific to your organization. Having a somewhat ageing foreman installation meant that we were suffering from the problems of any aging, snow-flaked system, namely it was flakey and no one wanted to make changes to it. This is when I decided that this was a good opportunity to extend the already existing puppet module to support this entity model with custom types/providers.

This was my first foray into writing types and providers so it took a little longer than might be expected and took several iterations to get the code working and more importantly, performant. Writing types and providers is still one of those areas that I don’t think is a well documented process and is not for the unexperienced. For the majority of my time I had [this]() wonder blog open, in addition to [the book]() and the [puppet documentation]() on the subject. Only with the arsenal of resources available was I able to tackle the challenge. Thankfully a couple of weeks of hacking later and I managed to have types for all the major foreman components: operatingsystems, configuration templates, provisioning profiles etc.

Take a look at my work and see what you think …

puppet-community

Puppet community is a github organization that has been developing for some time now. The idea behind the project/community is provide a shared space for puppet modules to live and be developed. It’s key benefits are really just starting to develop but are quite useful - shared build scripts, shared testing infrastructure and shared development and PR management. The hope is that with the experience that is found in the community the quality of the modules in the community as a whole will increase.

Being a separate non-corporate, non-individual entity means that when a primary author moves organizations and/or does not have the time to work on a module any more they have people who will continue to support it. This is huge, as anyone who has any more than about 5 modules on the forge will tell you - managing PRs and releasing updates can become a big job pretty fast. I know there are 8 modules authored by me personally and over 10 authored by myself and my OpenTable colleagues - this was becoming a full time job.

As you might be able to assume by this point, I am a huge fan of the project and I have been moving both my own personal modules as well as a number of OpenTable modules into the community. I won’t discuss this aspect any further, because there will be a much larger post of the subject - but be assured, it’s very exciting and a good move for all the users of all the modules I have written.

puppet notes

Finally anther small thing I have been working on is externalizing the notes I have been keeping about puppet over the last couple of years. These notes over everything from weird edge-cases you might run into, certain style guides and by far the largest part of the notes is on different code patterns I have been and my comments on them. As you might expect there is lots of content in there on several different approaches to the same problem - I have been doing my best to document these, including when I think they can be used and trying to pick a favourite. I’d love for you to take a look and let me know your thoughts.

* * *

Hopefully by now you are not too bored and have found some interesting things. In which case you should absolutely reach out to be at @liamjbennett and let me know your thoughts.

Until next time …..

[Quick Update] - 12-Mar-2015

It has recently been commented by a few individuals that I am a poor communicator of some of the interesting things that I am usually hacking on. I guess they are right and I thought that rather than wait and do a big-bang style blog post of the usual multi-thousand word variety that I would start trying to write these little posts in about 30 minutes and see how that works out. By the time your reading this I will have likely already written several and managed to build up a least a little motivation in writing again.

* * *

With all that in mind, what have I actually been working on recently?

The slow road towards linux on the desktop

Like almost all of my peers, it seems that for a number of years now I have been spending 40+ hours a week curled up with my macbook only to be dissatisfied with the operating system that it comes with. I have already seen a number people start to make the transition to Linux or BSD on the desktop and I think that 2015 might be the year for me (if you’re reading this after 2015 - don’t laugh too hard because I really meant it). With that target in mind I have been spending some time reducing my tooling to only those things that are cross-platform, preferring the command-line and making a note of all those things that I might have to keep OS X around for. So far in the past few months I have changed my password manager (1password to lastpass) my editor (atom to tmux+vim) and made a good list of things I like on OS X (RoyalTSX and VMware fusion) none of which seems to be blocking my transition. I am sure that this little project might turn into one of those big-bang blogs - but don’t hold your breath ;)

A few more updates with testing puppet on windows

So I wrote (this) post a while back and I have continue to work on the project and getting the changes that need to made into the master branch of beaker for everyone to use easily. This is almost done now so over the next few weeks you should be able to really start testing those modules - with our packer images of course.

Updates to those packer images

From the download numbers it seems like a number of your like using those windows images that I created so I am very happy about that. If you go and take another look at the page on atlas you should see that there a number of new versions there now that you might want to play with. We unfortunately had to temporally close-source the packer templates behind those images while we dealt with an issue with the bitvise installation, although expect this to be open again very soon. In addition to that I am very excited with the things going on over at the packer-community project. It seems like the plugins are now there so that we will eventually be able to remove the bitvise dependency completely, which is exciting.

* * *

Ok, so that’s enough for this first 30-minute update, but don’t worry I have plenty more updates to come in the following days so stay tuned for that.

puppetconf 2014 - part 1

The first installation of my review of the 2014 Puppet conference.

This series of posts will cover the talks, workshops and the contributors summit.

http://tech.opentable.co.uk/blog/2014/10/06/puppetconf-2014-part-1/

puppetconf 2014 - part 2

The second installment of my review of the 2014 Puppet conference:

http://tech.opentable.co.uk/blog/2014/10/06/puppetconf-2014-part-2/

puppetconf 2014 - part 3

The third installment of my review of the 2014 Puppet conference:

http://tech.opentable.co.uk/blog/2014/10/06/puppetconf-2014-part-3/

testing puppet with beaker pt 2 - the windows story

The second installment in the series on the Beaker testing tool.

This post covers the work that was done to enhance the tool for testing windows systems.

http://tech.opentable.co.uk/blog/2014/09/01/testing-puppet-with-beaker-pt-dot-2-the-windows-story/

testing puppet with beaker pt 3 - testing roles and profiles

The third installation in this series cover the use-case of testing roles and profiles.

http://tech.opentable.co.uk/blog/2014/09/01/testing-puppet-with-beaker-pt-dot-3-testing-roles/

About

Contact Details

Email: liamjbennett@gmail.com
Twitter: @liamjbennett
Github: github.com/liamjbennett
LinkedIn: liam-bennett-77415821

Employment

Feb 2014 - Present: OpenTable International Limited (London) - Senior Infrastructure Engineer

A Senior infrastructure engineer at OpenTable means supporting the datacenter operations and development teams to build new infrastructure and to automate the build and managment of existing legacy infrastructure.

During my time at OpenTable I have built a logging infrastructure from the ground-up (on AWS using ELK and Apache Kafka that can support 1B message a day), replaced multiple monitoring systems based on nagios with a single solution based upon sensu, applied config management to a wide range of systems both on windows and linux and helped to develop new tools to support standardized deployments on mesos.

A significant portion of this role has involed me liaising with and training development and operations teams both in London and in San Francisco. It also required communicating with all layers of management to ensure the successfull rollout of the projects.

A lot of this work have given me the opportunity to work and release a lot of code open source and build communities around that. I am now reasonably well known in the puppet community for my work with Windows and have had the opportunity to blog, give talks and podcast on the this and other subjects. Community evangelism is not something that is often though of as significant in an operations-based role but I have found it to be both enjoyable and beneficial on many occasions.

From April 2015 to date I have been leading my team in its projects and decision making. We reamin fairly autonomous within the organization, indenitying problems and providing solutions whereever we see value.

June 2010 - Feb 2014: Mimecast Ltd (London) - DevTools Engineer

As a build and release engineer for Mimecast I worked with all teams within the business and directly with the CTO to introduce Continuous Delivery into the business. That continued to be my main focus as the engineering team grew from 20+ to 100+. As the maturity of the build and release process advanced my role changed to a more DevTools focused position.

I think a devops mentality was critical to a role like this and I had to make sure that I remain balanced in both my development and operations skill-sets. This means that I was involved in all aspects of software development lifecycle. I advised on component based architectures to help speed up the build/delivery process, I introduced and educated on new testing tools, and I implemented and improved the deployment and release processes. I was also the primary administrator for all pre-production environments both Linux (CentOS) and Windows (all versions) and made extensive use of configuration management (puppet) and monitoring tools (Opsview, Nagios, Munin) to achieve this. I have performed many large migration projects and was heavily involved in Mimecast’s ISO 28000:2007 certification.

It was very much platform and language agnostic role and I go to work day-day with a wide variety of tools and platforms including Java, C# and Ruby on both Linux, Windows and all major mobile platforms. I continue to believe that it is hugely important to remain flexible and to be able to pick up new languages and skills as quickly and efficiently as possible.

Feb 2009 - June 2010: Saga Holidays Ltd (Folkestone, Kent) - System Consultant

As a systems consultant for Saga Holidays my work was extremely varied, from Java development work to advising business stakeholders about upcoming ideas and technologies. As a developer I worked on Java systems, both client and server side. I was originally recruited for my swing development skills but extended my knowledge beyond this into unit testing, webservices and databases.

In addition, I found that I spent a considerable amount of time focusing on build and releasing issues. I moved the team from performing manual builds to a semi-automated process and I continued to work on this as they advanced further in the continuous integration maturity model. This has extended my skill set to cover the use of tools such as Jenkins/Hudson, Sonar and scripting technologies such as Ant, Ivy and Maven. This was a difficult process (due in part the then legacy codebase) and required code refactoring, education of other impacted teams and migration to new toolsets (including new SCM).

Saga gave me experience in bringing CI to a large organization and the various challenges both technical and cultural that are involved in that.

Sept 2006 - Aug 2007: Nexor Ltd (Nottingham) - Placement Student

I had various roles during my year with Nexor as I rotated around a number of departments where I learnt Perl, C++ and a number of Testing and Development skills (including unix development and the use of vmware). I also gained experience in network administration and the implementation and maintenance of military standard email software.

This development experience, in addition to my time in testing, has given me experience of the full software development life-cycle as well as a consideration of everything that is required for ISO:9000 status. During this time I was trained by a external party in the practice of good customer service and I maintain that this is one of my strengths from the year.

Education

University of Kent at Canterbury

Bsc Computer Science with Year in the Industry (2:1)

Norton Knatchbull Secondary School

A-Levels (3 Bs, 1 C)

GCSE (9 A-C)

testing puppet with beaker

The first instalment of a series of posts covering the Puppet testing tool Beaker (https://github.com/puppetlabs/beaker)

This first post is an initial introduction.

http://tech.opentable.co.uk/blog/2014/04/04/testing-puppet-with-beaker/

profiles, roles, stacks and clouds

One of the more recent patterns in structuring out puppet manifests is: roles and profiles [1] [2]. The summary is this: profiles are a collection of classes/modules and roles are a collection of profiles. Having started to re-factor our own infratructure with this pattern recently there was one question that cam immediately to mind .. what’s next?

Roles and profiles is all about abstraction. With roles all I have to care about is what “type” of node I am building:

class role::www inherits role {
  include profile::tomcat
}

But what is the next abstraction? How far can we abstract?

###Stack Can we group together those nodes and roles? What I would like to see is the “stack” ..

define stack::webstack(
  $node_webserver,
  $node_appserver,
  $node_dbserver
) {
     node $node_webserver {
       include role::webserver
     }

     node $node_appserver {
       include role::appserver
     }

     node $node_dbserver {
       include role::dbserver
     }
 }

This does use the node declarations rather than the ENC but it can allow us to scale out small indeependant infrastucture quickly.

###Cloud Puppet also is now moving beyound the server to network swtiches and embedded hardware. Can we also extend out thinking beyond the 3-tier web stack? Can we use puppet to manage independent regions or independent clouds? I would like to see something like the following …

define cloud::region(
  $stack
  $network
) {
  include $stack

  network_route { $network:
    ensure    => 'present',
    gateway   => '10.0.2.2',
    interface => 'eth0',
    netmask   => '255.255.255.0',
    network   => '172.17.67.0'
  }
}

Do I have any code to show? Not yet. What I have is lots and lots of questions ..



If you want to discuss this more then reach out to me on twitter @liamjbennett

###References

Puppet on Windows - Part 2

In part 1 of this series of posts I discussed some of the challenges when writing modules for puppet on windows: documentation, packging, ISOs and reboots. In part 2 I will discuss how I learnt about this, about the modules that I have written, the learning outcomes and about contributing to the forge.

There are many ways that you can contribute and extend puppet: classes, defintions, facts, types and providers. Of the modules that I have written I wanted to specifically look at the following:

###Writing facter facts (win_facts) Writing facter facts is actually a good place to start with puppet. What is puppet not telling you about the system? Puppetlabs have actually provided a very useful tool that tells you lots of great information about the system but in some areas, either in general (platform-speific) cases or in your specific edge cases it might not give you that information you were hoping for.

When I started working with Windows the only facts that classied specifically for windows were:

:kernel => windows
:operatingsystem => windows

While this is useful if your mananging windows machines in an existing linux environment and want to seperately classify them, if you’ve got multiple versions of windows then your going to need a little bit more. This is why I started to write win_facts which provided the following additional facts:

:operatingsystemversion  => "Windows Server 2008 R2"
:windows_productkey      => "XXX-XXX-XXX-XXX-XXX"
:windows_sid                  => "S-1-S"
:windows_systemtype      => "x64"

These facts use two techniques that I want to draw attention to: The registry read and the command wrapping:

...
require 'win32/registry'

 Win32::Registry::HKEY_LOCAL_MACHINE.open('Software\Microsoft\Windows NT\CurrentVersion') do |reg|
    reg.each do |name,type,data|
      if name.eql?("ProductName")
        operatingsystemversion = data
      end
    end
 end
 ...

(see: https://raw.github.com/liamjbennett/puppet-win_facts/master/lib/facter/operatingsystemversion.rb)

As dicsussed in Part 1. The windows registry should be any windows developers friend and contains a weath of information. This is just one example - there are centainly plenty more than this. Puppet recently introduced the concept of External facts [1] - drop a file in the C:\ProgramData\PuppetLabs\facter\facts.d\ directory and facter will read it. This is good for many reasons but windows already has a great list of files containing facts - the windows registry - so let’s read from it and find some more.

The alternative is the command wrapper:

...
systemtype = Facter::Util::Resolution.exec('wmic ComputerSystem get SystemType | FindStr /i x')
systemtype.gsub!(/-based PC/,'')
systemtype.gsub!(/\s/,'')
systemtype.downcase!
....

(see: https://raw.github.com/liamjbennett/puppet-win_facts/master/lib/facter/windows_systemtype.rb)

The registry is, for most people, a very complex nest of hidden information. Sometimes it doesn’t contain all the information you need and sometimes it’s difficult to find or process. In these cases it’s always going to be an option to exec out and wrap an existing windows command. Windows has a wealth of small utilities [2] avaliable that can be used to pull out information, wmic is just one example of these.

Make sure that if your are going to exec that it’s as simple as possible. No long multi-line commands, no pages of powershell. If your logic is suffiently complex then facter is probably not appropriate to execute it inline. Instead write a utility (C#, Powershell, or even ruby) and build, test and deploy that seperately. You can then execute that as a fact or execute it out-of-band and have it write a external fact. Either way, getting the complex logic out of the main facter code and making it simpiler for everyone wishing to use it or maintain it.

And that points me to another important thing - do test your facts! Many people, myself included are guilty of not doing this but it’s pretty simple to do using standard rspec [3].

Most windows libraries, utilities and applications vary widely from one version of windows to another (and sometimes from one edition or service pack to another) so I make plenty of use of these facts in my modules to take this into account and you should to.

###Writing classes for comman dependencies (dotnet) There is category of classes that will be used everywhere, they are not common code (that could be built into the PuppetX namespace) but they are common between manifests. Such classes include java, dotnet, powershell, ruby, gcc. I call these common dependencies.

Many people will implement common dependencies in different ways but the community really needs to converge on a standard for most of them - this hasn’t happened yet. The first thing I would suggest is to always write as a definition first and only if it is totally impossible to install mutliple versions on the same machine (either legitimately or though some form of hack) should you make it a class. This is what I did with the dotnet module.

The .NET framework has some unique characteristics - firstly its installed differently on servers than it is on clients and secondly major versions (2,3,4) can be installed side-by-side while minor versions overwrite (4 and 4.5). I still made it a defintion because it can be installed side-by-side.

For modules like java and powershell there are even custom providers [4] that use those commands and will require seperate modules to make sure they are installed.

What I learnt about writing this module was actually a very simple lesson - don’t assume how your users will want to provide the packages. I assumed the exe files used would be stored on a network share (this was my personal use-case) but what if your users want to download over http, use chocolatey or some other custom package manager. I want not have wanted to (or had time to) implement all the possibilities but this is really about good design and about amking your module as extensible as possible. I am still refactoring now to take this into account - it’s easier if you design it in from the beginging.

These modules are probably the best modules - they are pretty quick to write, simple to test and get added everywhere so add huge value.

###Writing defintions, types and providers and knowning what’s appropriate (windows_firewall) The windows firewall module was written to manage the built-in firewall provided by Microsoft. This is one situation, similiar to what I discussed with the facter facts above, where I wanted to provide a wrapper to the existing netsh commands. So that I what I did, wrote a manifest for the service and a defintion for the exceptions - this can be seen here (https://raw.github.com/liamjbennett/puppet-windows_firewall/master/manifests/exception.pp).

This works quite well and as an initial version meets the use-cases but it’s far from elegant. This module raised the simple question: When should a defined type be refactored into a native type?

There is also a good rule of thumb for this: If the effort spent munging variables and arguments is greater than the effort spent managing resources or calling exec then that should be a native type. What this meant for the windows_firewall module is that while the main class manifest should remain as it is, the defined type for exception should be made into a native type. For me this is currently a work-in-progress and can be seen in the “type refactoring” branch on Github.

There are lots of benifits to writing types and providers. They are pure ruby, they can be tested as plain ruby, but more importantly they have the benefits of a turing-complete language and not being limited by the Puppet DSL. It does however also provide a great framework for managing and validating arguments and making sure that your type and providers remain idenpotent. For more information on how to write custom types and providers then go and read this great book on the subject (Puppet Types and Providers)

###Tackling the big applications (msoffice) For most windows infrastrcuture there come a point where you need to manage the much larger applications - you need to tackle those multi-GB applications normally distubuted via ISO. Taking on Microsoft, IBM, Oracle et al with Puppet is no small feat and is probably worth a much larger discussion but hopefully this will let you realise that such things are possible.

My first start at this was the Microsoft Office suite of applications. Most of these types of applications support silent installation to make it easy for administrators to deploy via group policy. We don’t want to distrubute by group policy but we can use puppet to template these answer files and exec that silent installation.

Templating and Exec-ing that setup.exe is really as complicated as it gets with regards to pupept itself. You find that the remaining complexity comes from the applications itself.

But using those exec resources too is another smell. If your using more than one exec in your manifest and/or you keep calling out to the same command line tool then this is another suggest that you need to be writing more types and providers.

For larger more complicated applications you will find yourself using many of the tools in the puppet toolbox - custom types/providers, custom facts and perhaps custom functions. If you find yourself doing this try and make those thing as genric as possible and split them out into smaller modules. I am current working on a module of MSExchange [4] and while prototyping this with ulgy exec statements and numerous hacks - it is leading down a path to create other modules for some custom functins and facts.

The point of this is to prove that all of those huge applications or suites of applications can still be installed and managed by puppet. It takes a great deal of initial effort, experience and on-oging work but it is worth it for us all in the long run. These sorts of applications come with long installation guides and come with certifications - we don’t all need to read this and take the exams if we can make puppet do all the heavy lifting for us.

###References

puppet on Windows - part 1

I have recently spent some time working with puppet, in particular working on writing modules for Windows and I wanted to share some of my thoughts and experiences that I learnt along the way.

###Finding useful information When I got starting writing my first module the thing that I realised was that at the time there was very little information out there of people using puppet on Windows. Thankfully in the past few months this has changed and I wanted to point you in the direction of some very useful resources:

###Your 3 new friends: Powershell, MSDN and the Registry This really depends what background your coming from. If your a developer or operations engineer from a Windows background and your just starting to use puppet, well then you probably know this already. If however your a coming from a Linux background and your looking to expand your puppet infrastructure to cover you Windows environments (and this was me) then you’ve got some learning to do.

The first thing is get to know Powershell [1][2][3] - it’s either bash for Windows or cmd on steroids depending on your point of view. Either way your going to find it very useful when trying to bend Windows to your will. Not that you need to be a master of Powershell to be productive with puppet on Windows but go and read, get familiar with how it works and get ready to google plenty. Powershell (recent versions on recent versions of Windows) allows you do do almost everything that you will want to configure from the command line. It is a feature rich language that most Windows operations people will be familiar with and best of all that means that there is plenty of code out there that makes good examples.

Next up is the big daddy of them all - the MSDN/TechNet documentation [3]. This is a huge archive that contains documentation on every built in library, utility and registry setting that comes with Windows. The more you work in the Windows world the more you will find yourself staring at its pages - that’s if you can find the one your looking for. A lot time was spent while writing my first Windows modules tracking down useful documentation - installation prerequisites, silent install options, config files options, registry tweaks. Digging all of this out can be time consuming but in the end totally worth it. Do your module users a favour and put links to these documents either in your README or in your wiki - it will save the pain for us all.

The Windows registry is the final weapon in your toolbox. If you’ve ever even looked at a Windows machine before then this is not news to you but your going to learn this more than you have ever done so before. Firstly it will help you in making sure your configurations are idempotent but also most Windows applications these days have both documented and undocumented registry tweaks so it would be good to expose some of them in your modules.

###The 3 classic problems: Packaging, ISOs and reboots When discussing windows with other puppet users the three topics that always come up are windows packaging, installing large applications that are distributed via ISOs and how best to manage system changes that require reboots.

####Packaging Packaging is a challenge on all operating systems and despite some very successful tooling like fpm [5] it still continues to have it’s problems. Windows is no exception to this. Windows has many methods for getting software onto the box: the exe, the msi, the 7zip executable, the zip file and probably a few more that I haven’t even come across yet. Each is also authored with very flexible tooling which means that you will face issues like packages that will declare their dependencies while others won’t and packages that install the source while other wants you to do it all yourself. Even the msi package which is by far the industry standard has it’s faults. The reality is that all of these methods need to be supported because there are applications out there that continue to use them so any configuration management system needs to deal with that as gracefully possible.

Puppet’s Package resource is very well suited to create this abstraction. In version 3.x it’s as simple as:

package { 'Firefox Setup 21.0':
    provider => windows,
    ensure   => installed,
    source   => 'C:\\Files\\Firefox Setup 21.0.exe',
    install_options => ['/S']
}

While this is great for all you 3.x users, there a still a large number of users out there (myself and github included) who are working with a 2.7 code tree. Not all is lost however. Firstly if your only using msi packages then there is nothing for you to do as puppet 2.7 already has you covered with the msi provider:

package { 'mysql':
  ensure          => installed,
  provider        => 'msi',
  source          => 'N:/packages/mysql-5.5.16-winx64.msi',
  install_options => { 'INSTALLDIR' => 'C:\mysql-5.5' },
}

Alternatively if you still need to support those pesky exe files then Reid Vandewiele (@reidmv) has you covered with his 2.7 backport of the new windows package provider [6]:

windows_package { 'Firefox Setup 21.0':
    ensure   => installed,
    source   => 'C:\\Files\\Firefox Setup 21.0.exe',
    install_options => ['/S']
}


But that’s not all - there is one more contender in this game of packaging - and it goes by the name of chocolatey [7]. Chocolatey is the self-styled apt-get for Windows - so all of your Linux admins can rejoice - and it’s also built on the well supported nuget dependency management system - so all of you .NET developers can join in this party. It has a huge community of users so most of the applications and MS re-distributables that you will want are available. Best of all the author is a one Mr Rob Reynolds (@ferventcoder) is now one of the members of the puppet windows team at puppetlabs so we can expect excellent chocolatey support within puppet going forward. Rich Siegal (@rismoney) has already released a puppet provider for chocolatey [8] which means that packages can be installed like this:

package { 'Firefox':
    ensure          => installed,
    provider        => 'chocolatey',
    install_options => ['/S']
}

The use of chocolatey really depends on your corporate situation. If your already supporting a private nuget repository then absolutely use chocolatey as it will save you some pain. If your not then it your choices are either configure a nuget repository, use Nexus Pro as a nuget repository, or stick to the old network share approach and the windows package provider. I am not here to say which method you should use - but you should use one.

####Managing those ISOs Many windows applications, at least “Enterprise-ready” ones and all of those distributed by Microsoft itself are packaged as an .iso file. This is both a throwback of days of boxed software and burned CDs but it also serves the purpose of being a single container for what is often a multi-gigabyte installation. The problem is one of size: you don’t want to be downloading 4GB iso onto your clients in order to install your applications. This leaves you with two options: Network share or mounted network drive.

#####Network share When dealing with large files this is going to be your obvious first approach. Set-up a network share (either windows or samba) and decompress those iso files into a local directory on the share. From there you can do something as simple as this:

exec { 'install-iso-app':
      command   => "\"\\\\fileshare\\folder\\SETUP.EXE\" /settings \"C:\\config.ini\"",
      provider  => windows,
      logoutput => true,
      require   => File["C:\\office_config.ini"]
}

The benefit of this approach is that you probably already have this sort of share set-up already. If you’ve got reasonably large windows environment this probably already existing on your Windows Update (WSUS) server. By installing directly over the network (without mounting) your not exposing this network share to your end clients at all. After all you may have lots of other software on this common windows share and you may not want all of your clients/users to be able to see that.

The disadvantage of this approach is that firstly, not all applications support this approach and secondly you will have to make sure that the network server is added to your local intranet security settings. Luckily the later can be fixed using the puppetlabs-registry module [9].

#####Mounted Network drive/folder The other approach to this problem is to mount a network share either as a drive or a folder. Simon Dean has written a net_share module [10] for this.

net_share {'PuppetTest':
    ensure        => present,
    path          => 'c:\puppet_test',
    remark        => 'PuppetTest',
    maximumusers  => unlimited,
    cache         => none,
    permissions   => ["${hostname}\\PuppetTest,full", "${hostname}\\PuppetTest2,full"],
}

This approach has the benefit that packages are installed as if they were from a local drive - many applications were written making this assumption. The disadvantage as stated above is that you can unnecessarily expose other applications to your users that you may not want to. There may be ways to mitigate this by mounting then dismounting the drive pre and post-install or mounting to a hidden folder but both approaches seem like a work-around.

My thoughts on the best of these methods to use really varies depending upon the size and complexity of the applications that I am trying to work with. Up to this point I have mostly been working with the network share approach but this required making registry entries and that seems a little nasty. I think that a better approach would be to write an provider just for iso files and use that as a vehicle to work through some of these use-cases.

I will discuss my experiences of using these approaches in a future post where I discuss in more detail the modules that I have written but for now I suggest you go and look at all these available methods and choose the one that best suits your existing environment and security needs.

###Reboots Up until very recently this a big problem for windows users. May packages require post-install reboots and for more complex applications that install their own dependencies they may require multi-reboots during an installation or upgrade run. This poses the problem of puppet should deal with this - does it skip the reboot and try to run through the rest of the catalog or does it perform the reboot in-line and perform the puppet run again? Well thankfully puppetlabs has come to the rescue with their reboot module [11]. This allows reboots to be added into the manifest like this:

 package { 'Microsoft .NET Framework 4.5':
   ensure          => installed,
   source          => '\\server\share\dotnetfx45_full_x86_x64.exe',
   install_options => ['/Passive', '/NoRestart'],
   provider        => windows,
 }
 reboot { 'after':
   subscribe       => Package['Microsoft .NET Framework 4.5'],
 }

Actually I like having reboots within the manifest - it helps prove my idempotence. This module is still being worked on by puppetlabs and at the time of writing there were issues for puppet users running ruby 1.8.x [12] but it’s a great solution to the problem.

###Summary Windows is like any other new platform - it has it’s quirks and it’s own sets of challenges but it also has an existing vibrant community which will be able to help with these sorts of issues and hopefully allow you to write some fantastic new modules for the forge. In part 2 of this post I will discuss the modules that I have written, my experiences of writing them and how they have evolved as both puppet and my experience has evolved.

####References

thoughts on scaling cucumber

I have been spending a lot of time lately working with Ruby and in particular a reasonably large suite of cucumber tests. Cucumber has been adapted by several teams within Mimecast to varying degrees of success and maturity, mostly dependent on each individual teams’ experience with Ruby.

My recent work has identified two problems scaling the test suite: cross-team testing and multi-platform. I want to discuss some of these problems and how we are going about solving them.

I guess the fist of these needs a little bit of background. Mimecast has grown significantly since I first joined them 3 years ago and while I joined them as a start-up I guess you would now consider them a medium-sized enterprise. In the beginning all developers worked on the entire codebase at different times but now the codebase and feature-set is so large that functional teams have formed around various products and/or various aspects of the Mimecast platform. What this means for testing is that many of the engineers are now focussing on testing features within their own little world view: database, front-end, corporate website etc. We call this the horizontal.

About six months ago we realised that there was an obvious problem with this approach: that the actual day-to-day workflows of the customer were being missed because of the team-siloed view of things (I will talk more about this in a future post) – we call this the vertical. What we wanted to do was to cover these workflows while making the most use of the work that each team had already done.

Let’s take a simple example: sending an email. There are a number of ways that the Mimecast platform can receive an email: from a windows client, from a mobile client, from a web client or even from the telnet command-line itself. As the most critical aspect of the business we want to make sure that all of these scenarios work as expected. This is are number one smoke test.

###The badly written test problem The first problem that we had in this case was each teams set of tests were written with only them in mind and a common pattern we discovered was the long 20+ line step definition. We also started to see really long features and long complex example tables. These code smells (and let’s not forget that this is code) are common [1] [2] [3] [4] and pointed us immediately to the need for refactoring and abstraction.

###The novice Ruby problem Then we ran into another problem, something common to novice Ruby users: the non-OO Ruby script pattern. There was abstraction in the test code, they had defined common functions and split them up into related collections of Ruby files but that is as far as it had gone: no-classes, no-modules and plenty on inbuilt assumptions. Now here is the fun bit – we knew this code was messy, but it worked and so our priority was first to abstract it into a format that was sharable across all our projects: gems.

What this also meant this that everyone got to see what was available, we found out that we had implemented 3 different ways to access the database, 2 different methods for logging and many other smaller common pieces of functionality. We were being open about the technical debt [5] rather than hiding it away in some corner of the codebase. It also had the added benefit that more people were looking at this code and trying to use it in different ways, which meant that it always going to improve as a result. I think it is very important to be open about your technical debt – it’s nothing to be ashamed of and hidden away, it happens to all projects and can given occur without you doing anything. It’s important to be honest and blameless about where it lives and try and integrate it’s pay-back into your standard project planning process. There is a lot of good resources out there on technical debt and how to manage it so I won’t discuss that much here. There aren’t so many on the cultural aspect of managing technical debt so I might write more about that in the future.

Using gems was tricky, we has a lot of them – about 10 or 15. Authoring them was the easy part, after one failed attempt to use them with bundler as-is we realised that the code was not stable enough and so we decided that the best approach was the vendor all our dependencies. The reason we do this is not as horrible as it first sounds (some people [6] even agree with it). We don’t fully vendor our gems in each project and we never check-in our vendor directory into source control. The only reason we vendor is for developer convenience. What it allows them to do is to debug the code problems, add logging and possible even work out a possible bug fix, all within the safety of their own environment knowing that whatever changes they make will be blown away when they next vendor.

###The Sharing problem The next problem we discussed was sharing step definitions. I am sure that this topic has come up with any team that has used cucumber for any decent length of time and I have one answer to this: Don’t! The path that many of our developers went down was to do exactly what we were doing with the rest of the code: to package it up into gems and share it around. This sounds good in practice, but in reality it causes more problems that it solves. This leaves you with 3 sorts of projects: one containing feature files, ones containing steps, and ones containing plain Ruby code. Aside from that fact that it means that a developer will be committing to 3 separate projects at any one time it also causes the projects containing the steps to start to build up with code again, something we were trying very hard to avoid. It also means that as a project grows over time it is more likely that the steps with conflict between shared projects or become so abstract as to be painful to debug. We considered switching away from Cucumber to something like Spinach [7] to avoid this problem but in the end we realised at it wasn’t the best option for us at the time. Actually the end solution was much more simple: keep the steps with the features but have a strict rule on the amount of code that should be contained within an individual step: at the moment this is 10 lines, I guess this is pretty arbitrary and perhaps not the best way to do it but it works for us and we are enforcing it within our code review process.

There is also a cultural problem here: individual teams were still writing libraries for their own needs first, forgetting or ignoring the needs of the downstream consumers in other teams. I think what makes this a difficult problem is that each team has it’s own test developer, incentivised to enhance the tests of their teams own product(s) not the wider testing effort. They will write code for the functions that they need and if other teams can use it then that is an additional bonus but not a priority. Each team has it’s own needs and it’s own way of doing things. Team-A will write a function to send a email via the command-line and Team-B will write a function to send an email via the web-ui and both of these are incompatible at the API-level. There are two solutions to this problem: put in an architect and force the teams to standardize or alternatively create an abstraction layer above the differences. Actually the answer is both as it’s the most flexible without causing too much cultural strain.

###The cross-platform problem The final problem isn’t really a problem at all - well not yet. It’s the issue of dealing with cucumber in a cross-platform way. I can only briefly discuss this problem and without a good solution at this point as we have yet to resolve it ourselves. What we have is a very cross platform set of projects: we have Java projects (on Linux) and C# projects (on Windows) and Objective-C projects (on Mac and iOS). Each set of projects is running tests with it’s own gherkin-based tool (Cucumber [8] and SpecFlow [9]) this is all well and good, we are arguably using the best tool for the job in each case while culturally still speaking the same language. Where we find difficulties is in sharing code: both because of language differences and platform differences. We are currently looking to move away from SpecFlow to cucumber which should solve one problem but the other problem is likely to involve some nasty ssh-based solution. I shall follow up with our outcomes on this.

Both myself and my current organisation are continuing to learn and mature in our experience with cucumber and with Ruby and how best to deal with these projects when we have it used so widely and by so many. Hopefully this will lead to further discussions and I hope to follow up this blog with further updates in the future.

####References

Why I am Blogging Again

It has been far too long since I last found myself writing any tl;dr content. Like the vast majority, this is at least my third attempt, my third tool and who knows if it will stick although I am now using the octopress/github setup. I am now writing along side my the rest of my code, so I think that it will be gaining traction that way.

So what I am writing about? Well I am working on so many different things these days: both in work and personally so I thought I would start with some of that. I working in both Ruby and Java as both developer and ops and I am in the process of open sourcing as much of my work as possible because I want you all to be able to read it and comment on it.

I am also going to be commenting on my day-to-day work and my observations of culture. As such it is worth stating that the opinions I have here on this blog are my opinions alone and do not represent the views or values of my current (or for that matter my previous) employer.


And with that all taken into consideration lets do this thing …