INFO: Chef client is running, will wait for it to finish and then run.

tldr; So I wasted a few hours today because I had changed a setting, (or chef had changed) the location of its lock file (chef-client-running.pid). I found that manually removing the file, and restarting chef was a work-around.





How very inconvenient:

"INFO: Chef client  is running, will wait for it to finish and then run."

dum..dee. dum. dumm....

"INFO: Chef client  is running, will wait for it to finish and then run."

what? thats like 10 minutes.. what's it doing? 

So previously I've seen some bad things happen when executions should take a few seconds, actually take a while. involving recursive operations and restoring from backups... ;-)

As its the weekend, and this is getting boring now. "killall chef-client" or something appropriately destructive followed.

So given that this is definitely not running 

[root@server-124 chef]# ps -ef | egrep "(ruby|chef)"

root      3941 29729  0 06:01 pts/0    00:00:00 egrep (ruby|chef)

 why am I still seeing this bullshit?

 # chef-client --once
[2013-03-30T05:43:09-07:00] INFO: Chef client  is running, will wait for it to finish and then run.

So on a normal running service, you would expect to see the pid here;

[root@server-123 chef]# cat /var/run/chef/client.pid
3201

but on this machine, nothing;

[root@server-124 chef]# ls /var/run/chef/client.pid
ls: /var/run/chef/client.pid: No such file or directory

so lets see what the criteria for this "Chef client is running" message;

[root@server-124 chef]# cd /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.0/lib/chef/
[root@server-124 chef]# grep -ir "will wait for it to finish" * 

run_lock.rb:        Chef::Log.info("Chef client #{runpid} is running, will wait for it to finish and then run.")

ah, so that gives up the necessary infos... so I've not set a lockfile in the /etc/chef/client.rb and thus its defaulting to this one;

    # Create a new instance of RunLock
    # === Arguments
    # * config::: This will generally be the `Chef::Config`, but any Hash-like
    #   object with Symbol keys will work. See 'Parameters' section.
    # === Parameters/Config
    # * :lockfile::: if set, this will be used as the full path to the lockfile.
    # * :file_cache_path::: if `:lockfile` is not set, the lock file will be
    #   named "chef-client-running.pid" and be placed in the directory given by
    #   `:file_cache_path`
    def initialize(config)
      @runlock_file = config[:lockfile] || "#{config[:file_cache_path]}/chef-client-running.pid"
      @runlock = nil
    end

here;

/var/cache/chef/chef-client-running.pid

delete that, and its working again.

now I have to track down what caused the situation with the zombied lockfile...