CFEngine configuration should be stored under version control, such as Concurrent Versions System (CVS) or Subversion. Version control repositories log changes over time, allow rollbacks, and can employ commitinfo or taginfo scripts to manage access. Most importantly, tags or branches can be used to test changes against development class systems before the change is released to production. This page details configuring a CVS repository to manage systems in several different realms (development, load, production) using tags.
If using Subversion, use branches instead of tags. Disadvantages of tags include their invisibility in CVS (need cvs status -v to show), lack of change logging, and difficulty training sysadmins in tags over editing files in different trees. Another option: use Perforce.
CVS Modules
CFEngine configuration files should be placed into a single module, such as cfengine/inputs under the CVS repository. Most other configuration files can be stored under a masterfiles module, with subdirectories for global configuration (masterfiles/etc), operating system specific needs (masterfiles/os), and different system roles (masterfiles/roles).
CFEngine modules, due to the : in their names, should be placed into a special top-level CVS module, such as cfengine-modules. This way, Windows systems will be able to checkout most of the repository, except for the cfengine-modules module.
$ ls
CVS CVSROOT cfengine cfengine-modules masterfiles
$ ls cfengine
CVS inputs test
$ ls cfengine-modules
CVS README module:authkey module:groups module:users test-cf-module
$ ls masterfiles
CVS app etc os role
$ ls masterfiles/app
CVS cfengine sendmail
$ ls masterfiles/etc
CVS authkey certs groups motd resolver rpm-gpg-keys ssh users yum
$ ls masterfiles/os
CVS macosx openbsd redhat solaris windows
$ ls masterfiles/role
CVS apache named nfs ntp sendmail squid tomcat
Create aliases in the CVSROOT/module file, or other top-level modules as needed. Breaking areas down into different modules may be required if different groups will have different access rights to various areas. However, too many modules will complicate staging from CVS.
$ cat CVSROOT/modules
redhat-spec masterfiles/os/redhat/SPECS
Examples
Review CFEngine Examples to see how CFEngine configuration and other data use the repository layout proposed here.
Alternate masterfiles Layout
Another possible masterfiles layout: store all configuration under directories named after the classes. This way, configuration files related to the redhat class would be under masterfiles/redhat, and those for role_ntp_client under masterfiles/role_ntp_client. I have not tried this layout in practice. One advantage of $masterfiles/$class: configuration would be trivial to lookup under the masterfiles area, rather than copying the arbitrary directory from a copy statement: ${masterfiles}/somewhat/random/dir/ntp.conf.
Regardless of the layout, I strongly recommend configuration files be stored by the duty or role of the file, and never by hostname. Under CFEngine, these duties will correspond to a class (or possibly variable) name.
Tag Managements
Use CVS tags, so that committed changes do not apply to any systems until the appropriate tag is updated. Consider tags for different groups of systems, such as DEV for development systems, LOAD for load testing hosts, and PROD for production systems. This way, a change can first be tested on development systems:
$ cvs ci -m 'Disable SSH1 protocol support.' cf.app_openssh
$ cvs tag -fF DEV cf.app_openssh
And once development systems pass any tests, the tag updated on cf.app_openssh for production systems:
$ cvs tag -fF PROD cf.app_openssh
A change to the critical update.conf file can also be tested first on development systems, to avoid breaking everything. Testing cfservd.conf changes will also be possible, by using a realm_dev CFEngine server, before tagging the change for any production CFEngine servers. Under this system, tags should closely follow the head revision, and no branches will be used, to prevent a divergence of configuration between the different tags. The different tags should be exported from CVS into different staging directories, as shown below.
Display Tags
Run cvs status -v to display the tags associated with a file. Set status -v in the ~/.cvsrc configuration file to set -v by default.
$ cvs status -v update.conf
===================================================================
File: update.conf Status: Up-to-date
…
Existing Tags:
stable (revision: 3.14)
Remove Tags on Deleted Files
Tags remain on files removed from the repository. After removing a file with cvs remove, also remove any tags on the file with cvs tag -d tagname file.
Showing Differences
Use cvs diff to display differences between tags and the special HEAD revision. For example, to compare differences between the PROD tag on the CFEngine policy files and any updates in the repository, run:
$ cvs update -Ad
$ cvs diff -r PROD cfengine/inputs
If possible, use the diff -u option set by default in the ~/.cvsrc file for easier to read diff output.
Exporting from CVS
Use stage-from-cvs to export CVS tags into different directories on the master CFEngine server. stage-from-cvs is easier to bootstrap than the usual method of manually creating the staging directories and manually checking out the repository, then running cvs update under the staging areas. stage-from-cvs also avoids repository metadata being present in the staging areas via the use of cvs export. Different modules and tags must be exported, leading to multiple stage-from-cvs runs to maintain the various trees and branches. Use CFEngine to stage from CVS periodically:
classes:
any::
role_cfengine_master = ( cfengine01 )
role_cfengine_slave = ( cfengine02 cfengine03 )
role_cfengine = ( role_cfengine_master role_cfengine_slave )
control:
any::
cf_stage_cmd = ( /var/cfengine/scripts/stage-from-cvs -d /cvs -r )
cf_stage_dir = ( /var/cfengine/stage )
shellcommands:
role_cfengine_master::
"%{cf_stage_cmd} DEV cfengine/inputs %{cf_stage_dir}/DEV/inputs"
"%{cf_stage_cmd} PROD cfengine/inputs %{cf_stage_dir}/PROD/inputs"
"%{cf_stage_cmd} DEV masterfiles %{cf_stage_dir}/DEV/masterfiles"
"%{cf_stage_cmd} PROD masterfiles %{cf_stage_dir}/PROD/masterfiles"
"%{cf_stage_cmd} DEV cfengine-modules %{cf_stage_dir}/DEV/cfengine-modules"
"%{cf_stage_cmd} PROD cfengine-modules %{cf_stage_dir}/PROD/cfengine-modules"
The master CFEngine server should run more frequently than client systems, or stage with a custom schedule five minutes before clients run, to update the staged areas before clients run:
control:
any::
schedule = ( Min00_05 Min30_35 )
role_cfengine_master::
schedule = ( Min55_00 Min25_30 )
Slave CFEngine Servers
Slave CFEngine servers should mirror the staging area on the master server, and also backup the CVS repository, in the event the master system fails. Ensure cfservd permits these areas to be copied in the cfservd.conf configuration file.
control:
any::
policyhost_master = ( cfengine01.example.org )
copy:
# mirror staging area from master cfengine server onto slave cfengine servers
role_cfengine_slave::
${cf_stage_dir}/
dest=${cf_stage_dir}
backup=false recurse=inf pruge=true
server=${policyhost_master}
mode=0444
type=checksum encrypt=true
# backup CVS area on all cfengine hosts
role_cfengine::
/cvs/
dest=/var/cfengine/backup/cvs/
backup=false recurse=inf pruge=true
server=${policyhost_master}
mode=0440
type=checksum encrypt=true
ifelapsed=59
Additional stage-from-cvs Runs
To prevent CFEngine configuration file errors from preventing fixes from being staged, either run the stages from some other scheduler, such as cron(8), or include a script for administrators to update the staging areas with. Also consider other means of staging data to client systems, such as package updates, or via Secure Shell (SSH), in the event update.conf breaks.
Alternatives
Instead of automatic staging by script (or on commit), consider also manual staging. This would allow an administrator to sign the release with a private key, and increase the separation between the data in the repository and the data published on the production CFEngine servers.
Limit CVSROOT/history Growth
Check the CVSROOT/config file LogHistory option, which may log each export run by stage-from-cvs, thus creating a huge CVSROOT/history file over time. Either do not log exports, or only log write options:
# Set `LogHistory' to `all' or `TOEFWUPCGMAR' to log all transactions to the
# history file, or a subset as needed (ie `TMAR' logs all write operations)
LogHistory=TMAR
Realm Classes
Create CFEngine classes that correspond to the tags used above. For example, if using DEV, LOAD, and PROD tags, also create realm_dev, realm_load, and realm_prod classes. Additionally, create a realm_norealm to classify systems that do not explicitly belong to one of the existing realms. This way, client systems will connect to the staging area appropriate to their realm: production systems to /var/cfengine/stage/PROD/inputs, and development systems to /var/cfengine/stage/DEV/inputs. Systems under the special no_realm class should connect to the PROD branch (presumably the most stable), then issue alerts or other messages until an admin can assign the host to an appropriate realm.
Both update.conf and cfagent.conf must define these realms and associated variables to control where client systems look for configuration data.
classes:
any::
realm_prod = ( IPRange(192.0.2.0/24) )
realm_dev = ( dev_example_org )
realm_norealm = ( any -realm_prod -realm_dev )
control:
realm_prod|realm_norealm::
cf_inputs = ( /var/cfengine/stage/PROD/inputs )
masterfiles = ( /var/cfengine/stage/PROD/masterfiles )
realm_dev::
cf_inputs = ( /var/cfengine/stage/DEV/inputs )
masterfiles = ( /var/cfengine/stage/DEV/masterfiles )
copy:
any::
# mirror cfengine inputs directory from cfengine server onto clients
${cf_inputs}/
dest=${cf_inputs}
backup=false recurse=1 pruge=true
server=${policyhost_master}
mode=0444
type=checksum encrypt=true
alerts:
realm_norealm::
"error: realm not set: addr=${ipaddress}, host=${fqhost}" ifelapsed=1439
Use CVS Keywords
When possible, insert the $Id$ or $Header$ keyword into every file under the repository. These keywords identify the last person to commit the file, and other useful metadata. The $Header$ keyword shows the full path to the file, and helps associate a deployed configuration file to the source file under version control. Both Subversion and Perforce support RCS style keywords, though vary in supported options and their contents.
# $Header: /cvs/conf/bind9/named.conf,v 2.71 2005/06/24 22:08:19 user Exp $
# Note: this file managed under CFEngine
#
# BIND 9 configuration for a master DNS server.
acl clients {
…
If an admin needs to change the named.conf, the $Header$ keyword should point them to the conf/bind9/named.conf file under CVS.
Also, when editing a file using an editfiles statement, include the $Id$ of the CFEngine configuration file:
editfiles:
any::
# TCP Wrappers: deny by default, allow as needed by class
{
/etc/hosts.deny
AutoCreate
Backup "false"
IfElapsed 59
EmptyEntireFilePlease
AppendIfNoSuchLine "# $Id$"
AppendIfNoSuchLine "# Note: this file managed under CFEngine"
AppendIfNoSuchLine "ALL: ALL"
}
{
/etc/hosts.allow
AutoCreate
Backup "false"
IfElapsed 59
EmptyEntireFilePlease
AppendIfNoSuchLine "# $Id$"
AppendIfNoSuchLine "# Note: this file managed under CFEngine"
BeginGroupIfDefined "role_mail_server"
Append "sendmail: ALL"
EndGroup
…
}
The resulting /etc/hosts.allow will then display the CFEngine file with the above editfiles block, for example on a role_mail_server class system:
# $Id: cf.tcp_wrappers,v 1.61 2005/11/20 19:51:45 user Exp $
# Note: this file managed under CFEngine
sendmail: ALL
editfiles alternative: use InsertFile to assemble the configuration file, rather than Append statements.
If the $Header$ keyword produces wrapped lines, consider also the $RCSfile$ and similar keywords over multiple lines. I strongly recommend against the $Log$ keyword, as it duplicates log information into the configuration needlessly, and may cause massive file growth over time.
Reporting
Use a commit handler to e-mail changes to a mailing list, or publish changes via a web page. Consider cvs2cl to create change logs.
Web services must not run on the master CFEngine server, as they pose an unnecessary security risk. Web services such as CVSWeb and CVSHistory should instead be run on a different system. This system would duplicate the CVS repository from the master server via a CFEngine copy statement.
Alternatives
Instead of cvs tag to maintain different branches, some sites may want to use two different CVS servers, one for development, and the other for production. This way, changes would simply be committed to the appropriate server, and no tags changed. Changes made to development would be imported into the production server at release time. The production server could have more stringent rules for commits, such as requiring a change request number to be logged in the commit message, or an EMERGENCY flag used otherwise.
