If we (Chef) could take the time to invest in addressing some long-standing issues with Test Kitchen, it will make using our kit much better. I'm going to highlight two issues here, however the "idea" is to invest some more product cycles into making Test Kitchen easier to use and more robust:
Here are the two issues that are currently affecting more than one of my customers:
# Managing Chef versions during kitchen testing if the AMI already has Chef baked in
kitchen provisioner install strategy options are: always/once/skip
This is super annoying when we have Chef14 (or really any version that’s not latest stable) baked into an AMI and then we want to have a kitchen suite that always tests against latest stable Chef.
The current workaround is to set the product_name: chef-workstation, which gets usually (but not always) has the latest stable bundled in. It’s kludgy and is a pain to explain yet-another workaround to our Chef developers.
If I’m following the code correctly, kitchen/provisioner/chef_base gts all Mixlib::Install-y around here:
Contrast this with chef-client-updater cookbook examples:
Whereby something like:
version ‘16’ => installs latest 16.x.y
version ‘12.13’ => installs latest 12.13.x
version ‘12.15.16’ => installs exactly this version
This also uses mixlib-install
So, tl;dr -- why can’t ktichen provisioner install strategies behave like chef-client-updater install strategies?
I have tried workarounds whereby a test cookbook would do the update using chef-client-updater, but the return code situation on windows is still not working as hoped (at least from what I had observed in my testing). And even IF this workaround did work, its yet-another workaround we have to train our devs on.
# concurrent kitchen verify
Using a cookbook with 4 or more suite as an easy case to reproduce, finish a successful ‘kitchen converge -c’ then try ‘kitchen verify -c’ the concurrent/parallel verify fails. Our current workaround is to run kitchen verify serially. This is yet-another workaround we have to train our devs on.
167 is probably the relevant one still open
Its a bit tricky to implement parallel execution. The challenge is not so much in inspec itself, the challenge is that we use RSpec undercover and its creating a global state. I see two doable options:
instead of using the inspec’s ruby classes, we exec to the inspec bin with the arguments
we install inspec on each machine and (not preferred, since its way slower)
Essentially, we need to make sure that the execution context is not shared