puppet

Component Pattern

This is part of a series of posts on puppet patterns. If you can spot any flaws or think of better solutions let me know!

This pattern is similar to the PuppetLabs best practices of roles and profiles. If you’re not using roles and profiles it’s a good idea to look into them, there’s a post by Craig Dunn which explains it really well. The basic idea is reuse via abstraction. There’s an acronym in development DRY which stands for Don’t Repeat Yourself (the antitheses being WET, Write Everything Twice, so smart!!) so we try to “dry out” our code by reducing all the times we repeat ourselves. So we use profiles to abstract classes and roles to abstract profiles:

node webserver01 {
  include role::webserver
}

node webserver02 {
  include role::webserver
}

class role::webserver {
  include profile::tomcat
  # other profiles
}

class profile::tomcat {
  # include classes and/or add parameterised classes
}

Instead of defining all the classes we need with all the parameters etc on every node, we wrap all related classes together into profiles and then group the profiles into roles. Yes we could define both nodes as a comma separated list, but pretend they’re in different files! This pattern also works well if you’re using an ENC, each node only has one role so that’s a lot less configuration/management in the ENC

Roles and profiles look just like normal modules:

puppet
  - modules
    - role
      - manifests
        - webserver.pp
    - profiles
      - manifests
        - tomcat.pp

I have found one downside though. Even in a relatively non-trivial environment structure, by the time you’ve created roles to describe every different type of machine in all of your environments, your role names either become too long and cumbersome, or they become too vague and unclear.

The component pattern is designed to be an abstraction on top of profiles that focuses on the deployed components on the nodes instead of the role of the node as a whole:

node webserver01, webserver02 {
  include component::company_webapp
}

node apiserver01, apiserver02 {
  include component::search_api
  include component::rest_api
}

node dbserver01 {
  include component::main_db
}

node testserver01 {
  include component::company_webapp
  include component::search_api
  include component::rest_api
  include component::main_db
}

class component::company_webapp {
  include profiles::tomcat
  include profiles::company_webapp
  # other profiles
}

class component::search_api {
  include profiles::tomcat
  include profiles::search_api
  # other profiles
}

class component::main_db {
  include profiles::mysql
  include profiles::main_db
  # other profiles
}

Each component has everything it needs to run in isolation, but you can also put all your components on one node if you want too, even if they share resources e.g. the webapp and both APIs are probably all going to run on the same JVM behind the same instance of tomcat, but because of the way we structure our components, profiles and classes, any combination should be possible (assuming your apps play nicely together)

Yes, it’s a bit more code in the node definitions and there is more potential for repetition, but I think the node definitions are a lot more transparent and the component classes are reusable all over the place allowing you to structure your environments however you want.

Thera are some wins for ENCs too. You can create new environments and new layouts, or add components to existing nodes without having to create new roles. You can also give more context to your ENC because you’re now defining nodes by the components that are running on them rather than the role. I came up with this pattern when I was working out how to structure a release orchestration tool I’m tinkering with. I can now define components in the tool’s database and assign them to nodes in a layout in any combination and then easily translate that layout into ENC definitions.

So component classes only include profiles and have everything they need to run your component. However they are also designed to run side by side with any other components allowing any combination you want.

Virtual Parameter Pattern

This is part of a series of posts on puppet patterns. If you can spot any flaws or think of better solutions let me know!

The aim of this pattern is to reduce coupling between modules, or rather flip the coupling in the right direction. Imagine you have a module that can collate a number of parameters from other modules, for example mcollective has facts like facter, any module can add a fact to mcollective’s list so that it can be queried by a client (the version of the code you’ve deployed for example).

The first way to solve this is to have an “mcollective::facts” class which has all the facts preset as variables. This isn’t good for two obvious reasons, the list of facts is preset, and the mcollective module now contains references to other modules whether through references to variables in specific classes, calling functions in another module, or knowledge of how that module works (hiera keys etc)

class mcollective::facts {
  fact_one = $other_module::foo::var
  fact_two = function_from_another_module('var1')
  fact_tree = hiera('module_specific_key')

  file { 'mcollective facts':
    # write facts
  }
}

The second way is to have an “mcollective::fact” defined type that updates a file line (or whatever needs doing to set the parameter). The classes in other modules that want to set facts can now just define an instance of the fact class and pass the key and value in as parameters. This is good because the coupling is now reversed so that the mcollective module isn’t dependent on all the modules that want to use it. But, it’s now almost too tightly coupled, you can’t just have random file_line (or concat, or whatever) resources floating around the catalogue, there are dependencies that need to be defined, for example the file has to exist before you can set a line in it, and the service might need to be restarted once you’re set a new fact. So you now have to define before/require and notify/subscribe relationships between mcollective classes inside the classes that are setting facts. Your classes now have to have knowledge of how the mcollective module works which means you have to set it up properly, but it also means that if anything changes in the mcollectice module it could potentially break all the modules using it.

define mcollective::fact (
  $fact_value,
) {
  file_line { "mcollective fact $name":
    # write fact
  }
}

class foo {
  # class stuff

  mcollective::fact { 'foo-fact':
    fact_value => 'baz',
    require    => File['mcollective facts'],
    notify     => Service['mcollective'],
  }
}

The solution is to use virtual resources. We keep the “mcollective::fact” class but we make the file_line resource virtual. This way we can define facts in whichever classes we want, but all the logic is encapsulated inside the mcollective module. Resource collectors can grab all the virtual parameters and handle the dependencies. Use tags to make sure we only grab the file_line resources we actually want.

define mcollective::fact (
  $fact_value,
) {
  @file_line { "mcollective fact $name":
    # write fact
    tag => 'mcollective_fact'
  }
}

class mcollective::facts {
  # setup facts file and directories etc
  
  File['mcollective facts'] -> File_line <| tag == 'mcollective_fact' |> ~> Service['mcollective']
}

class foo {
  # class stuff

  mcollective::fact { 'foo-fact':
    fact_value => 'baz',
  }
}

Now the foo class only needs to know about one resource, and the mcollective module doesn’t need to know how it’s being used (if at all). If we want to change how mcollective manages it’s facts it doesn’t matter, we can change everything behind the scenes without having to change any other modules.

This pattern can be used in many other situations, it doesn’t just have to be facts, files and file_lines

Unit Testing Puppet with rspec-puppet: Mocking Classes and Functions

I’ve created a sample repo on GitHub that has a mini tutorial and examples of how to mock functions, classes, and defined types. This way you you can isolate the specific thing that you want to test (a class, a defined type, a custom function) without inheriting complexity from dependencies.

It’s still in its early stages so bugs and weird things are likely to crop up, but it’s a start!

I wont repeat anything here, this is just a placeholder if anyone finds this page first

Enjoy