Chef Ideas

If we (Chef) could take the time to invest in addressing some long-standing issues with Test Kitchen, it will make using our kit much better. I'm going to highlight two issues here, however the "idea" is to invest some more product cycles into making Test Kitchen easier to use and more robust:

Here are the two issues that are currently affecting more than one of my customers:

# Managing Chef versions during kitchen testing if the AMI already has Chef baked in

kitchen provisioner install strategy options are: always/once/skip

https://docs.chef.io/workstation/config_yml_kitchen/#new-provisioner-settings

This is super annoying when we have Chef14 (or really any version that’s not latest stable) baked into an AMI and then we want to have a kitchen suite that always tests against latest stable Chef.

The current workaround is to set the product_name: chef-workstation, which gets usually (but not always) has the latest stable bundled in. It’s kludgy and is a pain to explain yet-another workaround to our Chef developers.

If I’m following the code correctly, kitchen/provisioner/chef_base gts all Mixlib::Install-y around here:

https://github.com/test-kitchen/test-kitchen/blob/6112653db9ff44cc7bdb5380203ae7095ba9e71f/lib/kitchen/provisioner/chef_base.rb#L506

Contrast this with chef-client-updater cookbook examples:

https://github.com/chef-cookbooks/chef_client_updater#examples

Whereby something like:

version ‘16’ => installs latest 16.x.y

version ‘12.13’ => installs latest 12.13.x

version ‘12.15.16’ => installs exactly this version

This also uses mixlib-install

https://github.com/chef-cookbooks/chef_client_updater#resources

So, tl;dr -- why can’t ktichen provisioner install strategies behave like chef-client-updater install strategies?

I have tried workarounds whereby a test cookbook would do the update using chef-client-updater, but the return code situation on windows is still not working as hoped (at least from what I had observed in my testing). And even IF this workaround did work, its yet-another workaround we have to train our devs on.

# concurrent kitchen verify

Using a cookbook with 4 or more suite as an easy case to reproduce, finish a successful ‘kitchen converge -c’ then try ‘kitchen verify -c’ the concurrent/parallel verify fails. Our current workaround is to run kitchen verify serially. This is yet-another workaround we have to train our devs on.

167 is probably the relevant one still open

https://github.com/inspec/kitchen-inspec/issues/167

https://github.com/test-kitchen/test-kitchen/issues/1568

https://github.com/test-kitchen/test-kitchen/issues/1307

https://github.com/inspec/kitchen-inspec/issues/119

c-rock says:

”

Its a bit tricky to implement parallel execution. The challenge is not so much in inspec itself, the challenge is that we use RSpec undercover and its creating a global state. I see two doable options:

instead of using the inspec’s ruby classes, we exec to the inspec bin with the arguments

we install inspec on each machine and (not preferred, since its way slower)

Essentially, we need to make sure that the execution context is not shared