Migrate ServerSpec to InSpec - Part 3

From Bonus Bits
Jump to: navigation, search

Purpose

This article gives the steps to migrate from ServerSpec to InSpec Integration tests in a Chef Cookbook. The steps are split into several small parts. In Part 3 we see how to take local InSpec tests and put them in their own repository to gain a shared InSpec Profile. With a shared InSpec Profile we can have countless Cookbooks using the same set of InSpec Profiles without duplicating code.

Shared InSpec Profile Project

Create New Project

First, we need to create a new project that will be our shared InSpec Profile.

  1. Create a Git repo.
    1. I created a repository named example_shared_inspec on the bonusbits Github organization
  2. Clone Locally
    git clone https://github.com/bonusbits/example_shared_inspec.git
    


Add Project Files

Most of these we can copy from the cookbook.

.gitignore

.vagrant
*~
*#
.#*
\#*#
.*.sw[a-z]
*.un~

# Bundler
Gemfile.lock
bin/*
.bundle/*

# test kitchen
.kitchen/
.kitchen.local.yml

# Chef
Berksfile.lock
.zero-knife.rb
Policyfile.lock.json
.idea

.rubocop.yml

AllCops:
  Exclude:
    - 'berks-cookbooks/**/*'
    - '.idea/**/*'
    - '.git/**/*'
    - '.kitchen/**/*'
    - '*.lock'
    - 'vendor/**/*'

Documentation:
  Enabled: false

Metrics/LineLength:
  Max: 200

Metrics/BlockLength:
  Enabled: false

Metrics/MethodLength:
  Max: 60
  CountComments: false
  Enabled: true

Style/Encoding:
  Enabled: false

Style/SignalException:
  EnforcedStyle: only_raise

Style/EmptyLiteral:
  Enabled: false

CyclomaticComplexity:
  Enabled: false

AlignParameters:
  Enabled: false

Encoding:
  Enabled: false

Style/FrozenStringLiteralComment:
  Enabled: false

Style/TrailingCommaInLiteral:
  Enabled: false

Rakefile

We'll strip down the Rakefile to only have Rubocop style testing task.

# Rubocop
namespace :style do
  require 'rubocop/rake_task'
  desc 'RuboCop'
  RuboCop::RakeTask.new(:ruby) do |task|
    task.options = ['--display-cop-names']
  end
end

desc 'Rubocop'
task default: %w(style:ruby)

desc 'Circle CI Tasks'
task circleci: %w(style:ruby)

Gemfile

We'll also strip down the Gemfile to only have the min needed for Rubocop.

source 'https://rubygems.org'

gem 'rake', '~> 12.0.0'

group :style do
  gem 'rubocop', '~> 0.47.1'
end

circle.yml

Next, we'll use a different format for the CircleCi configuration. We'll be using the new Native Docker features of CircleCi v2.0 beta. So, instead of creating an entire VM and install Ruby, bundler plus rubocop. We'll just call a Docker image that has these already included. I decided to use Chef's chefdk image for this example.

Gnome-sticky-notes-applet Currently, to use the CircleCI v2.0 API open beta, you must sign up through their website and get approved access. It was pretty quick and painless for me.

version: 2
jobs:
  build:
    working_directory: ~/circulate
    docker:
      - image: chef/chefdk:1.3.32
    steps:
      - checkout
      - run:
          name: Run CircleCI Rake Task
          command: /opt/chefdk/embedded/bin/rake circleci --trace
notify:
  webhooks:
    - url: https://webhooks.gitter.im/e/00000000000000000

inspec.yml

One of the minimum requirements for a InSpec Profile is to have an inspec.yml in the root. It contains basic project information.

name: example_shared_inspec
title: InSpec Example Nginx Cookbook
maintainer: first.last@domain.com
copyright: Bonus Bits
license: MIT license
summary: Run InSpec Tests for Nginx and PHP FPM
version: 1.0.0
supports:
- os-family: linux

controls

Copy the test/integration/default/inspec folder to the root of our new project and rename the folder to controls

cp -R ../example_serverspec_to_inspec/test/integration/default/inspec .
mv inspec controls


Run Rubocop

Now let's run rubocop before we commit the project to Github.

rake

OR

rubocop .

Example

# rake
Running RuboCop...
Inspecting 4 files
....

4 files inspected, no offenses detected

Commit to Github

Finally, let's commit our new Shared InSpec Profile to Github.

git push


Cookbook Project

Now that we have the tests in its own project it's time to remove the tests from the cookbook and point the kitchen configuration to the git repo.

Remove Local Tests

For this example, we're going to completely remove the local InSpec tests.

rm -rf test

Update .kitchen.yml

Next, we need to add a verifier section to our default test suite that points to our Shared InSpec Profile git URL.

Gnome-sticky-notes-applet The inspec_tests value is an Array of tests. We are just adding the one Shared InSpec Profile we created. Although, we can call multiple InSpec Profiles. Including a mix of local and remote tests. We can even call tests from a Chef Compliance server.

---
verifier:
  name: inspec
  format: <%= ENV['CI'] ? 'junit' : 'cli' %>
  <% if ENV['CI'] %>
  output: "test-reports/%{platform}_%{suite}_inspec.xml"
  <% end %>

driver:
  name: docker
  use_sudo: false
  privileged: true
  forward: 80
  driver_config:
    ssl_verify_mode: ":verify_none"

provisioner:
  name: chef_zero
  always_update_cookbooks: true
  require_chef_omnibus: 12.19.36

platforms:
  - name: amazon-docker
    driver_config:
      image: amazonlinux:latest
      platform: rhel

suites:
  - name: default
    run_list:
      - recipe[example_serverspec_to_inspec::default]
    verifier:
      inspec_tests:
        - name: inspec_tests
          git: https://github.com/bonusbits/example_shared_inspec.git
    attributes:

Revision Cookbook

We'll go ahead and bump the cookbook version to 1.2.0.

name 'example_serverspec_to_inspec'
maintainer 'First Last'
maintainer_email 'first.last@domain.com'
license 'MIT'
description 'Demonstrate Migration Integration Tests from ServerSpec to InSpec'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.2.0'
chef_version '~> 12.5' if respond_to?(:chef_version)
issues_url 'https://github.com/bonusbits/example_serverspec_to_inspec/issues'
source_url 'https://github.com/bonusbits/example_serverspec_to_inspec'

supports 'amazon'

Run InSpec Tests

Now that we have a refactored our cookbook to use the Shared InSpec Profile; the last step is to run the integration tests.

rake integration

OR

kitchen test

Example

# kitchen test
-----> Starting Kitchen (v1.16.0)
-----> Cleaning up any prior instances of <default-amazon-docker>
-----> Destroying <default-amazon-docker>...
       Finished destroying <default-amazon-docker> (0m0.00s).
-----> Testing <default-amazon-docker>
-----> Creating <default-amazon-docker>...
       Sending build context to Docker daemon  481.8kB
       Step 1/16 : FROM amazonlinux:latest
        ---> 766ebb052d4f
       Step 2/16 : ENV container docker
        ---> Using cache
        ---> 9306967c7449
       Step 3/16 : RUN yum clean all
        ---> Using cache
        ---> 6729a55d63b3
       Step 4/16 : RUN yum install -y sudo openssh-server openssh-clients which curl
        ---> Using cache
        ---> d91055dade53
       Step 5/16 : RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''
        ---> Using cache
        ---> c647407017ec
       Step 6/16 : RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ''
        ---> Using cache
        ---> dde4401a6122
       Step 7/16 : RUN if ! getent passwd kitchen; then                 useradd -d /home/kitchen -m -s /bin/bash -p '*' kitchen;               fi
        ---> Using cache
        ---> 43bc21da1946
       Step 8/16 : RUN echo "kitchen ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
        ---> Using cache
        ---> c13e2988ada5
       Step 9/16 : RUN echo "Defaults !requiretty" >> /etc/sudoers
        ---> Using cache
        ---> 4fabb287c4c5
       Step 10/16 : RUN mkdir -p /home/kitchen/.ssh
        ---> Using cache
        ---> fa81cf0825ed
       Step 11/16 : RUN chown -R kitchen /home/kitchen/.ssh
        ---> Using cache
        ---> 6742e97caba3
       Step 12/16 : RUN chmod 0700 /home/kitchen/.ssh
        ---> Using cache
        ---> 790ecdc0c8a1
       Step 13/16 : RUN touch /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 682afe7e98ab
       Step 14/16 : RUN chown kitchen /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> fa4279bfb34f
       Step 15/16 : RUN chmod 0600 /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 4ebe38dc87f3
       Step 16/16 : RUN echo ssh-rsa\ AAAAB3NzaC1yc2EAAAADAQABAAABAQDh0jPG\+f\+JLWB9ckvab0iAhiae/ICRQ0JbIOrtwEgGsLisWLPDGE81JxXR0UuwUKuoJ30F2Mih1YgcoE7qYAouBUSseNgDs3E5O8V1IVjSGGdXXkFJf56jU7dABGbavXH3tq0l1NnBtx6krJNO5xhlJU5QhgZlbu\+1r8/g1AkEIVbWYLjaPMciicmEQ\+j\+IUNTjuMBy4rWaE\+37MmE8QgD5XL/NDkEyhZ1u2vegKebvCEcX2IhNNUnyPkVXz2lqeCyrQX/MczeguX0rtxQbJyQ3WU1/O/u5YS\+na6Td16hJt/fGQzTZkUl\+8MYqS8xFNn\+hdlZ7Fd\+QgEvs/mTYc5B\ kitchen_docker_key >> /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 4ad91eb3a6dd
       Successfully built 4ad91eb3a6dd
       d63e6ac532f06119d9e347538fc12203f158e440aae21f26586b8f2b86f44864
       0.0.0.0:32773
       [SSH] Established
       Finished creating <default-amazon-docker> (0m0.91s).
-----> Converging <default-amazon-docker>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 5.6.4...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
-----> Installing Chef Omnibus (12.19.36)
       Downloading https://omnitruck.chef.io/install.sh to file /tmp/install.sh
       Trying curl...
       Download complete.
       el 6 x86_64
       Getting information for chef stable 12.19.36 for el...
       downloading https://omnitruck.chef.io/stable/chef/metadata?v=12.19.36&p=el&pv=6&m=x86_64
         to file /tmp/install.sh.31/metadata.txt
       trying curl...
       sha1	323e88bd64b166a823087d0f50a949506d0fa6b5
       sha256	89e8e6e9aebe95bb31e9514052a8926f61d82067092ca3bc976b0bd223710c81
       url	https://packages.chef.io/files/stable/chef/12.19.36/el/6/chef-12.19.36-1.el6.x86_64.rpm
       version	12.19.36
       downloaded metadata file looks valid...
       downloading https://packages.chef.io/files/stable/chef/12.19.36/el/6/chef-12.19.36-1.el6.x86_64.rpm
         to file /tmp/install.sh.31/chef-12.19.36-1.el6.x86_64.rpm
       trying curl...
       Comparing checksum with sha256sum...
       Installing chef 12.19.36
       installing with rpm...
       warning: /tmp/install.sh.31/chef-12.19.36-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
       Preparing...                          ################################# [100%]
       Updating / installing...
          1:chef-12.19.36-1.el6              ################################# [100%]
       Thank you for installing Chef!
       Transferring files to <default-amazon-docker>
       [2017-04-24T18:31:22+00:00] INFO: Forking chef instance to converge...
       Starting Chef Client, version 12.19.36
       [2017-04-24T18:31:22+00:00] INFO: *** Chef 12.19.36 ***
       [2017-04-24T18:31:22+00:00] INFO: Platform: x86_64-linux
       [2017-04-24T18:31:22+00:00] INFO: Chef-client pid: 151
       Creating a new client identity for default-amazon-docker using the validator key.
       [2017-04-24T18:31:23+00:00] INFO: Client key /tmp/kitchen/client.pem is not present - registering
       [2017-04-24T18:31:23+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/default-amazon-docker
       [2017-04-24T18:31:23+00:00] INFO: Setting the run_list to ["recipe[example_serverspec_to_inspec::default]"] from CLI options
       [2017-04-24T18:31:23+00:00] INFO: Run List is [recipe[example_serverspec_to_inspec::default]]
       [2017-04-24T18:31:23+00:00] INFO: Run List expands to [example_serverspec_to_inspec::default]
       [2017-04-24T18:31:23+00:00] INFO: Starting Chef Run for default-amazon-docker
       [2017-04-24T18:31:23+00:00] INFO: Running start handlers
       [2017-04-24T18:31:23+00:00] INFO: Start handlers complete.
       [2017-04-24T18:31:23+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found:
       resolving cookbooks for run list: ["example_serverspec_to_inspec::default"]
       [2017-04-24T18:31:23+00:00] INFO: Loading cookbooks [example_serverspec_to_inspec@1.2.0]
       Synchronizing Cookbooks:
       [2017-04-24T18:31:23+00:00] INFO: Storing updated cookbooks/example_serverspec_to_inspec/recipes/default.rb in the cache.
       [2017-04-24T18:31:23+00:00] INFO: Storing updated cookbooks/example_serverspec_to_inspec/metadata.json in the cache.
       [2017-04-24T18:31:23+00:00] INFO: Storing updated cookbooks/example_serverspec_to_inspec/README.md in the cache.
       [2017-04-24T18:31:23+00:00] INFO: Storing updated cookbooks/example_serverspec_to_inspec/templates/default/sysconfig.network.erb in the cache.
         - example_serverspec_to_inspec (1.2.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Converging 5 resources
       Recipe: example_serverspec_to_inspec::default
         * template[/etc/sysconfig/network] action create[2017-04-24T18:31:23+00:00] INFO: Processing template[/etc/sysconfig/network] action create (example_serverspec_to_inspec::default line 7)
       [2017-04-24T18:31:23+00:00] INFO: template[/etc/sysconfig/network] created file /etc/sysconfig/network

           - create new file /etc/sysconfig/network[2017-04-24T18:31:23+00:00] INFO: template[/etc/sysconfig/network] updated file contents /etc/sysconfig/network

           - update content in file /etc/sysconfig/network from none to 9d2f64
           --- /etc/sysconfig/network	2017-04-24 18:31:23.651808504 +0000
           +++ /etc/sysconfig/.chef-network20170424-151-1mw5fcb	2017-04-24 18:31:23.651808504 +0000
           @@ -1 +1,4 @@
           +NETWORKING=yes
           +HOSTNAME=localhost.localdomain
           +NOZEROCONF=yes[2017-04-24T18:31:23+00:00] INFO: template[/etc/sysconfig/network] owner changed to 0
       [2017-04-24T18:31:23+00:00] INFO: template[/etc/sysconfig/network] group changed to 0
       [2017-04-24T18:31:23+00:00] INFO: template[/etc/sysconfig/network] mode changed to 644

           - change mode from '' to '0644'
           - change owner from '' to 'root'
           - change group from '' to 'root'
         * yum_package[php70-fpm, nginx] action install[2017-04-24T18:31:23+00:00] INFO: Processing yum_package[php70-fpm, nginx] action install (example_serverspec_to_inspec::default line 15)
       [2017-04-24T18:31:28+00:00] INFO: yum_package[php70-fpm, nginx] installing php70-fpm-7.0.16-1.21.amzn1 from amzn-main repository nginx-1.10.2-1.30.amzn1 from amzn-main repository
       [2017-04-24T18:31:32+00:00] INFO: yum_package[php70-fpm, nginx] installed ["php70-fpm", "nginx"] at ["7.0.16-1.21.amzn1", "1.10.2-1.30.amzn1"]

           - install version 7.0.16-1.21.amzn1 of package php70-fpm
           - install version 1.10.2-1.30.amzn1 of package nginx
         * ruby_block[Set PHP FPM Ownership] action run[2017-04-24T18:31:32+00:00] INFO: Processing ruby_block[Set PHP FPM Ownership] action run (example_serverspec_to_inspec::default line 17)
       [2017-04-24T18:31:32+00:00] WARN: Open3: Standard Out ()
       [2017-04-24T18:31:32+00:00] WARN: Open3: Status (pid 429 exit 0)
       [2017-04-24T18:31:32+00:00] INFO: ruby_block[Set PHP FPM Ownership] called

           - execute the ruby block Set PHP FPM Ownership
         * service[php-fpm-7.0] action enable[2017-04-24T18:31:32+00:00] INFO: Processing service[php-fpm-7.0] action enable (example_serverspec_to_inspec::default line 32)
       [2017-04-24T18:31:32+00:00] INFO: service[php-fpm-7.0] enabled

           - enable service service[php-fpm-7.0]
         * service[php-fpm-7.0] action start[2017-04-24T18:31:32+00:00] INFO: Processing service[php-fpm-7.0] action start (example_serverspec_to_inspec::default line 32)
       [2017-04-24T18:31:32+00:00] INFO: service[php-fpm-7.0] started

           - start service service[php-fpm-7.0]
         * service[nginx] action enable[2017-04-24T18:31:32+00:00] INFO: Processing service[nginx] action enable (example_serverspec_to_inspec::default line 37)
       [2017-04-24T18:31:32+00:00] INFO: service[nginx] enabled

           - enable service service[nginx]
         * service[nginx] action start[2017-04-24T18:31:32+00:00] INFO: Processing service[nginx] action start (example_serverspec_to_inspec::default line 37)
       [2017-04-24T18:31:32+00:00] INFO: service[nginx] started

           - start service service[nginx]
       [2017-04-24T18:31:32+00:00] INFO: Chef Run complete in 9.275659912 seconds

       Running handlers:
       [2017-04-24T18:31:32+00:00] INFO: Running report handlers
       Running handlers complete
       [2017-04-24T18:31:32+00:00] INFO: Report handlers complete
       Chef Client finished, 7/7 resources updated in 10 seconds
       Finished converging <default-amazon-docker> (0m27.61s).
-----> Setting up <default-amazon-docker>...
       Finished setting up <default-amazon-docker> (0m0.00s).
-----> Verifying <default-amazon-docker>...
       Loaded example_shared_inspec

Profile: InSpec Example Nginx Cookbook (example_shared_inspec)
Version: 1.0.0
Target:  ssh://kitchen@localhost:32773


  Nginx nginx
       installed
       service
  Php FPM
       php-fpm installed
       php-fpm service
       nginx owns /var/log/php-fpm
       nginx owns /var/lib/php/7.0

Test Summary: 6 successful, 0 failures, 0 skipped
       Finished verifying <default-amazon-docker> (0m1.10s).
-----> Destroying <default-amazon-docker>...
       PID                 USER                TIME                COMMAND
       5410                root                0:00                /usr/sbin/sshd -D -o UseDNS=no -o UsePAM=no -o PasswordAuthentication=yes -o UsePrivilegeSeparation=no -o PidFile=/tmp/sshd.pid
       5435                root                0:00                sshd: kitchen@pts/0
       5921                root                0:00                {php-fpm-7.0} php-fpm: master process (/etc/php-fpm.conf)
       5922                499                 0:00                {php-fpm-7.0} php-fpm: pool www
       5923                499                 0:00                {php-fpm-7.0} php-fpm: pool www
       5924                499                 0:00                {php-fpm-7.0} php-fpm: pool www
       5925                499                 0:00                {php-fpm-7.0} php-fpm: pool www
       5926                499                 0:00                {php-fpm-7.0} php-fpm: pool www
       5997                root                0:00                nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
       5998                499                 0:00                nginx: worker process
       5999                499                 0:00                nginx: worker process
       6001                499                 0:00                nginx: worker process
       6002                499                 0:00                nginx: worker process
       6004                root                0:00                sshd: kitchen@notty
       d63e6ac532f06119d9e347538fc12203f158e440aae21f26586b8f2b86f44864
       d63e6ac532f06119d9e347538fc12203f158e440aae21f26586b8f2b86f44864
       Finished destroying <default-amazon-docker> (0m1.14s).
       Finished testing <default-amazon-docker> (0m30.77s).
-----> Kitchen is finished. (0m32.29s)


Reference Github Link


Walkthrough Video


Related Articles


Sources