agentless monitoring of linux servers from ossec

we’ve been poking around ossec for a while. it’s nice and all but… i never liked the idea of opening additional ports on the production server or letting it ‘call home’ with alerts to the central monitoring machine. i prefer ‘polling’ model where central server communicates with monitoring nodes. ideally all should be done via ssh to avoid opening additional ports. ideally there should be no need to install additional software on the monitored machines. with ossec’s agent-less mode and a bit of custom scripts it’s actually possible.

disclaimer – [pseudo]security measures described below can be circumvented. proper file system integrity check should be done offline otherwise you risk not detecting threats that modify running kernel to hide own existence and file system modifications.

ossec provides this script as a simple example of agent-less monitoring of a linux server file system; a good starting point but the script:

• does not provide ability to filter out some files or directories from the check. for instance i’d like to inspect /etc/ except /etc/lvm/* and /etc/mtab – since those two change every time i make lvm-level-snapshot. that happens during every nightly backup. way too many warnings.
• occasionally times out on a high latency/low bandwidth connection between europe and asia. i did not manage to find actual cause of the problem although i suspect expect. replacing expect-based remote execution mechanism with ssh resolved the intermittent problems when script was receiving ‘cut-in-half’ responses.

so here’s mine improved version of the ossec agent-less script for monitoring remote linux servers via ssh. it should be saved in the ~ossec/agentless/ssh_integrity_check_linux2

#!/bin/bash
function handle {
        echo  "ERROR: an error has occured"
        exit 1
}

set -e
trap handle ERR

if [ "$1" == "test" ] && [ "$2" == "test" ] ; then
        exit 0
fi

echo "INFO: Starting."
echo "STORE: now"

params='
 echo "INFO: starting file system checks";
 echo "----- starting file system checks -----";
 IFS=$'"'"'\n'"'"';
 for i in `find '$2' -type f| egrep -v "'$3'"` ; do
        echo -n `md5sum "$i"| cut -d " " -f 1`;
        echo -n `sha1sum "$i"| cut -d " " -f 1`;
        stat --printf "%s:%a:%u:%g" "$i";
        echo "$i" ;
 done;
'
ssh $1 $params
echo "INFO: Finished."
exit 0

this script will calculate and report back to the ossec server checksums, permissions and sizes of files in the selected directories, it’ll also exclude specified file system branches from the check. to make ossec monitor selected host with this script [and notify about any changes in the remote file system] add in the ossec.conf:

  <agentless>
    <type>ssh_integrity_check_linux2</type>
    <frequency>1800</frequency>
    <host>root@someHost</host>
    <state>periodic_diff</state>
    <!-- arguments passed to the script. 
     * first - implicit, passed by ossec: root@someHost
     * second - set of paths that should be monitored. all together enclosed in quotes.
     * third - regular expression telling which files/directories should be excluded. leave empty "" if all should be included.
     -->
    <arguments>"/etc/ /boot /bin /sbin /usr/sbin /usr/local/bin /usr/bin" "/etc/lvm|/etc/mtab"</arguments>
  </agentless>

you’ll need to configure the ssh public/private key authentication so that ossec’s user can log-in as root@someHost using its private key.

you can extend that script and add monitoring of change of the output from some other checks – for instance just after done; you can add:

 echo "INFO: lsmod";
 echo "----- lsmod -----";
 lsmod|sort|grep -v dm_snapshot|awk "{print \$1\" \"\$2}";
 echo "INFO: debsums";
 echo "----- debsums -----";
 debsums -s 2>&1|grep -v "missing file /usr/share/doc/";
 # rkhunter, unhide and other tools might follow