Showing posts with label chef. Show all posts
Showing posts with label chef. Show all posts

Wednesday, June 3, 2020

Using Chef Event Handlers for Resources

Chef Provides a comprehensive list of events and different ways to handle them. These are decently documented at https://docs.chef.io/handlers.
In a recent use case, I had to ensure that a folder has only the files I wanted it to have, any more files or directories should get deleted whenever the chef-client runs. Similarly, if a file is missing, it should be re-created. In addition, I must know the names of the files that were created/deleted and the name and the content of the file that was updated.

Problem 1 - How to find which files you don't want?
There's no easy way chef provides this, so a bit of Ruby code to the rescue.
# The path where all my source files are stored.
# For me the source is the cookbook itself.
source_path = "#{__dir__}/../files/default/myfiles"

# Path on Target; where I need all my files
# Set some temporary variables
target_files = []

# Deleted Files
deleted_files = []

# updated_files
updated_files = {}

# The code to populate our source files dynamically
Dir.chdir(source_path) do
  source_files = Dir.glob("*")
end

# Ensure Target Directory Exists
directory target_path
# List of Files on Target
begin
  Dir.chdir(target_path) do
    target_files = Dir.glob("*")
  end
rescue
  Chef::Log.info "Directory may not exists, skipping as non-fatal"
end

files_to_delete = target_files - source_files
Chef::Log.info "Total Source Files: #{source_files.length}"
Chef::Log.info "Total Target Files: #{target_files.length}"
Chef::Log.info "Files to delete: #{files_to_delete}"
Problem 2 - How to find which files were created/deleted/modified?
This is where the Chef handler events kick in. I prefer using the 'on event' style.
# Add the Resource Handler
Chef.event_handler do
 on :resource_updated do |resource, action, update|
   begin
     if resource.is_a?(Chef::Resource::File) or resource.is_a?(Chef::Resource::CookbookFile) or resource.is_a?(Chef::Resource::Directory)
       if action == :delete
         deleted_files << resource.path
       elsif action == :create
         updated_files.store(resource.path, resource.diff)
       end
     end
   rescue
     # do nothing
   end
  end
end
The code above will get triggered everytime a resource is updated (any resource). Chef provides different events which you can trap, here I was interested to know only about the updates. I was also only interested in if the resource was a File or a Directory.

Bringing the last bits together.
Once I knew which files to delete, I could just delete them and the event above would keep a track of which files were deleted. Similarly for file creation and modification.
files_to_delete.each do | f |
  if File.directory?("#{target_path}/#{f}")
    directory "#{target_path}/#{f}" do
      recursive true
      action :delete
    end
  else
    file "#{target_path}/#{f}" do
      action :delete
    end
  end
end
source_files.each do | f |
  cookbook_file "#{target_path}/#{f}" do
    source "#{f}"
  end
end

# Finally, log a message with all files deleted.
# Notice the use of lazy block to evaluate the variable at converge time.
if !deleted_files.empty?
  log 'deleted_files' do
    message lazy { "Deleted Files: #{deleted_files}" }
  end
end

if !updated_files.empty?
  log 'updated_files' do
    message lazy { "Updated Files: #{updated_files}" }
  end
end

ruby_block 'updated_status' do
  block do
    node.normal['deleted_files'] =  deleted_files
    node.normal['updated_files'] =  updated_files
    node.normal['last_run_time'] =  Time.now
  end
end

Wednesday, May 25, 2016

CHEF - Sequel - RuntimeError: unknown privilege type SYSDBA

I explained in my previous post how to install Sequel Ruby gem to interact with Oracle databases in CHEF. However, there's little documentation on Sequel's website about connecting to Oracle databases.

I had two issues, (1) I couldn't figure out the connection string pattern for the database and (2) Once I did, I had issues connecting to the database using elevated privileges (e.g. connecting as SYSDBA).

Issue 1
The correct format for Oracle database is, you must quote the whole thing.
DB = Sequel.connect('oracle://user:password@hostname:port/service_name')
e.g.
DB = Sequel.connect('oracle://scott:tiger@oel1.mydomain.com:1523/orcl.mydomain.com')

Issue 2
If you want to connect with SYS user, the above syntax needs to be changed to

DB = Sequel.connect('oracle://sys:mypassword@oel1.mydomain.com:1523/orcl.mydomain.com', :privilege => :SYSDBA)


CHEF - Using an Oracle Database

There are a lot of cookbooks that let you run queries/operations against open source databases (MySQL/PostgreSQL etc). I wanted to use something to deal with Oracle database. I found an old blog on Oracle's website that shows few examples of this integration using oci8. But oci8 works specifically for Oracle and while it solved my problem, I couldn't use it for other databases. A friend recommended me Sequel which I found really useful. But using Oracle with Sequel still requires you to install and configure oci8. Here's the process:

Example below is for configuring on Mac OS. See footnotes for Linux.

Step 1
Install Ruby gem oci8. To install oci8, you must have at a minimum Oracle Instant client and SDK. Download them from Oracle's website.

Step 2
Copy the zip files to the homebrew library folder. If you are not using homebrew, you should!
ls /Library/Caches/Homebrew
instantclient-sdk-macos.x64-11.2.0.4.0.zip
instantclient-basic-macos.x64-11.2.0.4.0.zip

Step 3
Install the packages using homebrew.

brew install InstantClientTap/instantclient/instantclient-basic
==> Tapping instantclienttap/instantclient
Cloning into '/usr/local/Library/Taps/instantclienttap/homebrew-instantclient'...
Tapped 4 formulae (43 files, 37.9K)
==> Installing instantclient-basic from instantclienttap/instantclient
==> Downloading http://download.oracle.com/otn/mac/instantclient/11204/instantclient-basic-macos.x64-11.2.0.4.0.zip
Already downloaded: /Library/Caches/Homebrew/instantclient-basic-macos.x64-11.2.0.4.0.zip
==> /usr/bin/install_name_tool -id /usr/local/lib/libclntsh.dylib.11.1 /usr/local/Cellar/instantclient-basic/11.2.0.4.0/lib/libclntsh.dylib
.... [output clipped]
🍺  /usr/local/Cellar/instantclient-basic/11.2.0.4.0: 8 files, 181.0M, built in 3 seconds
/Library/Caches/Homebrew >

brew install InstantClientTap/instantclient/instantclient-sdk
==> Installing instantclient-sdk from instantclienttap/instantclient
==> Downloading http://download.oracle.com/otn/mac/instantclient/11204/instantclient-sdk-macos.x64-11.2.0.4.0.zip
Already downloaded: /Library/Caches/Homebrew/instantclient-sdk-macos.x64-11.2.0.4.0.zip
🍺  /usr/local/Cellar/instantclient-sdk/11.2.0.4.0: 40 files, 1.9M, built in 0 seconds
/Library/Caches/Homebrew >

Step 4
Install the oci8 Ruby gem, in this case I want to use it with chef so I am using chef's syntax to install the gem, it will work for just ruby too (without preceding the command with chef)

chef gem install ruby-oci8
Building native extensions.  This could take a while...
Successfully installed ruby-oci8-2.2.2
1 gem installed

Step 5
Install the Sequel Ruby gem.

chef gem install sequel
Fetching: sequel-4.34.0.gem (100%)
Successfully installed sequel-4.34.0
1 gem installed

That's it! You should now be able to use Sequel gem to query Oracle databases.

Important note for Linux:
If you want the CHEF recipes to work on your target systems (say Linux), these packages must be installed on them. The problem is that if you attempt to install these gems using chef_gem package in a recipe, the oci8 gem will fail to execute unless you set the LD_LIBRARY_PATH which at the time of writing I couldn't figure out how to. So as a workaround, I installed the Oracle instant client packages (rpm) manually, set the LD_LIBRARY_PATH to where libclntsh.so file was. In my case it was export LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib and then installed the gems using the CHEF recipe. Make sure you are in the same session or the LD_LIBRARY_PATH is set at the profile level.



Monday, May 23, 2016

CHEF - ArgumentError: could not find a temporary directory

When running chef-client as non root user, you may encounter the following error:

==============================
  Error Syncing Cookbooks:
  =============================
  Unexpected Error:
  -----------------
  ArgumentError: could not find a temporary directory
 
  Platform:
  ---------
  x86_64-linux


In the stacktrace, the error shows up as:
ArgumentError: could not find a temporary directory
/opt/chef/embedded/lib/ruby/2.1.0/tmpdir.rb:34:in `tmpdir'
/opt/chef/embedded/lib/ruby/2.1.0/tmpdir.rb:105:in `tmpdir'
/opt/chef/embedded/lib/ruby/2.1.0/tmpdir.rb:137:in `create'
/opt/chef/embedded/lib/ruby/2.1.0/tempfile.rb:136:in `initialize'
/opt/chef/embedded/lib/ruby/2.1.0/tempfile.rb:320:in `new'
/opt/chef/embedded/lib/ruby/2.1.0/tempfile.rb:320:in `open'


Solution:
This is not a CHEF issue rather a ruby one. Newer versions of ruby do not allow any directory to be used as a temporary directory that has (777) permissions set on it unless setuid of 1 is set on the directory too. As root user do the following:

chmod 1777 /tmp
ls -ld /tmp  drwxrwxrwt. /tmp

notice the "t" in the output.

CHEF - Running chef-client as non-root user

Chef works best when run as root user but there are scenarios when you would want to run a chef-client as non root user. To do that, do the following steps:

Assumption:
It is assumed that you already have bootstrapped the node and/or installed the chef-client.  Also, it assumes that you have access to the client.pem key needed to talk to the CHEF server. For this example, we'll assume the non root user is "oracle".

Step 1:
As root, copy the entire content of /etc/chef folder to "oracle" user's home directory and change permissions.
cp -rf  /etc/chef /home/oracle/.chef
chown -R oracle:oinstall /home/oracle/.chef

Step 2:
Open the /home/oracle/.che/client.rb file and add the following lines:
cache_path "/home/oracle/.chef"
client_key "/home/oracle/.chef/client.pem"
 

If you don't specify cache_path, you will get the error:
Unable to access cache at /var/chef. Switching cache to /home/oracle/.chef
whenever you run the chef-client as oracle user.

If you don't specify the client_key location, chef-client will look at /etc/chef/client.pem file, if the file is not readable (by default it won't), the client will error out with

Private Key Not Found:
----------------------
Your private key could not be loaded. If the key file exists, ensure that it is
readable by chef-client.

Relevant Config Settings:
-------------------------
validation_key "/etc/chef/validation.pem"


Step 3:
Now, run the chef-client with the following arguments:
chef-client -o recipe[] -c /home/oracle/.chef/client.rb



Monday, May 9, 2016

CHEF - Uninstalling CHEF Server

Once installed, it can be tricky to cleanup a CHEF server install. The guys at CHEF have updated their docs (https://docs.chef.io/uninstall.html) with instructions on how to deinstall CHEF server but I found that it does not cleans up the system entirely.

Note: Running these steps will completely wipe out all configurations you have, so proceed with caution.

I followed the following steps to remove CHEF server on my RHEL machine.

1. As mentioned in the CHEF docs, run
chef-server-ctl uninstall
This step will first stop all the processes and then run a cleanup.

2. If you do a grep on the running processes, you'll notice the EPM daemon would still be running. Kill it.
ps -ef|grep chef
/opt/opscode/embedded/service/opscode-chef-mover/erts-6.4/bin/epmd -daemon
kill -9 "process id of the epm daemon"

3. Now remove the CHEF packages. You can use whatever package manager you use, I use yum.
yum remove chef-*

Even if you did not install the CHEF package using yum, this will remove the package.

4. Cleanup all the folders.
rm -rf /opt/opscode*
rm -rf /opt/chef-manage
rm -rf /var/opt/opscode*
rm -rf /etc/chef

5. Restart the host.
init 6


Tuesday, February 2, 2016

CHEF – Installing a Standalone CHEF Server


Hosted CHEF is a common practice and is a good way to start learning CHEF. But at times you don’t have access to the internet and hence that makes it difficult to keep working. I am getting my hands across CHEF these days and wanted to setup my own (standalone) CHEF server. I have the following setup:
A MAC machine that is my desktop – I’ll be installing the CHEF DK here and this will be my workstation.
A Linux VirtualBox server (mychef.mydomain.com) – This will be the CHEF server.
A Linux VirtualBox server (oel1.mydomain.com) – This will be the CHEF client.
This post will cover installing the standalone CHEF server. From here download the CHEF server RPM. In my case it was chef-server-core-12.3.1-1.el6.x86_64.rpm. From here download the CHEF manage rpm (chef-manage-2.1.1-1.el6.x86_64.rpm). And finally from here download the CHEF reporting rpm (opscode-reporting-1.5.6-1.el6.x86_64.rpm).  Place all the RPMs in a location that is accessible from the VirtualBox servers. I have setup a shared folder on my MAC and I share that across all my VirtualBox servers over NFS. In this example, all rpms are located at /oracle/stage.
Step 1: Install CHEF Core
From the command prompt on the CHEF server (mychef), execute the following:
[root@mychef ~]rpm -iv /oracle/stage/chef-server-core-12.3.1-1.el6.x86_64.rpm
Preparing packages for installation...
chef-server-core-12.3.1-1.el6
[root@mychef ~]#
Step 2: Install CHEF Manage
[root@mychef ~]# rpm -iv /oracle/stage/chef-manage-2.1.1-1.el6.x86_64.rpm
Preparing packages for installation...
chef-manage-2.1.1-1.el6
Thank you for installing the Chef Management Console add-on!
The next step in the process is to run:
chef-manage-ctl reconfigure
[root@mychef ~]#
Don’t run the reconfigure command just yet.
Step 3: Install CHEF Reporting
root@mychef ~]# rpm -iv /oracle/stage/opscode-reporting-1.5.6-1.el6.x86_64.rpm
Preparing packages for installation...
opscode-reporting-1.5.6-1.el6
Thank you for installing the Opscode reporting addon!
[root@mychef ~]#
Step 4: Configure CHEF Core
[root@mychef ~]# chef-server-ctl reconfigure
Starting Chef Client, version 12.5.1
resolving cookbooks for run list: ["private-chef::default"]
Synchronizing Cookbooks:
  - enterprise (0.5.1)
  - private-chef (0.1.0)
  - apt (2.7.0)
  - yum (3.6.0)
  - openssl (4.4.0)
  - runit (1.6.0)
.
.
. (lines truncated)
Running handlers:
Running handlers complete
Deprecated features used!
  Cannot specify both default and name_property together on property path of resource yum_globalconfig. Only one (name_property) will be obeyed. In Chef 13, this will become an error. at 1 location:
    - /opt/opscode/embedded/cookbooks/cache/cookbooks/yum/resources/globalconfig.rb:76:in `class_from_file'
Chef Client finished, 369/483 resources updated in 02 minutes 50 seconds
Chef Server Reconfigured!
[root@mychef ~]#
Step 5: Configure CHEF Manage
root@mychef ~]# chef-manage-ctl reconfigure
Starting Chef Client, version 12.4.1
resolving cookbooks for run list: ["omnibus-chef-manage::default"]
Synchronizing Cookbooks:
  - omnibus-chef-manage
  - chef-server-ingredient
  - enterprise
  - private_chef_addon
  - runit
  - unicorn
  - packagecloud
  - build-essential
  - yum
  - yum-epel
Compiling Cookbooks...
.
.
. (lines truncated)
Running handlers:
Running handlers complete
Chef Client finished, 63/89 resources updated in 46.284372132 seconds
chef-manage Reconfigured!
Step 6: Configure CHEF Reporting
[root@mychef ~]# opscode-reporting-ctl reconfigure
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["opscode-reporting::default"]
Synchronizing Cookbooks:
  - enterprise (0.3.0)
  - opscode-reporting (0.2.0)
  - runit (1.3.0)
  - build-essential (2.2.3)
  - yum (3.6.1)
Compiling Cookbooks...
Recipe: opscode-reporting::default
.
.
Running handlers complete
Deprecated features used!
  Cannot specify both default and name_property together on property path of resource yum_globalconfig. Only one (name_property) will be obeyed. In Chef 13, this will become an error. at 1 location:
    - /opt/opscode-reporting/embedded/cookbooks/cache/cookbooks/yum/resources/globalconfig.rb:76:in `class_from_file'
Chef Client finished, 37/55 resources updated in 20 seconds
opscode-reporting Reconfigured!
Step 7: Check if services are up:
[root@mychef ~]# chef-server-ctl status
run: bookshelf: (pid 6592) 504s; run: log: (pid 5479) 550s
run: nginx: (pid 8334) 80s; run: log: (pid 6212) 511s
run: oc_bifrost: (pid 6353) 509s; run: log: (pid 4821) 620s
run: oc_id: (pid 6408) 508s; run: log: (pid 4893) 610s
run: opscode-erchef: (pid 6680) 502s; run: log: (pid 5648) 544s
run: opscode-expander: (pid 6504) 505s; run: log: (pid 5131) 595s
run: opscode-expander-reindexer: (pid 6550) 504s; run: log: (pid 5334) 556s
run: opscode-reporting: (pid 8347) 79s; run: log: (pid 8307) 82s
run: opscode-solr4: (pid 6460) 506s; run: log: (pid 5086) 598s
run: postgresql: (pid 6330) 509s; run: log: (pid 4307) 636s
run: rabbitmq: (pid 6237) 510s; run: log: (pid 2898) 658s\
run: redis_lb: (pid 5908) 526s; run: log: (pid 5901) 526s
[root@mychef ~]# chef-manage-ctl status
run: events: (pid 7353) 175s; run: log: (pid 7363) 173s
run: redis: (pid 7504) 164s; run: log: (pid 7316) 179s
run: web: (pid 7514) 163s; run: log: (pid 7438) 169s
run: worker: (pid 7455) 166s; run: log: (pid 7485) 164s
Step 8: Create an Administrator for CHEF
[root@mychef opscode]# chef-server-ctl user-create admin admin admin admin@foo.com welcome1 --filename /opt/opscode/admin.pem
Step 9: Create an Organization for CHEF
[root@mychef opscode]# chef-server-ctl org-create adminorg "admin org" --association_user admin
Step 10: Login to CHEF UI
From a browser, open https://mychef.mydomain.com (add the certificate exceptions to your browser)

-->
Enter the user credentials you created in Step 8 and Sign in.
 
--> Your CHEF server configurations are now complete.