HotSaNIC
HTML overview to System and Network Information Center
by Bernd Pissny ( hotsanic@bernisys.prima.de / http://www.berni sys.prima.de )


HotSaNIC version history

0.4.0 - 20020108
new features
extensions
bugfixes
All features of the work version 0.3.6 are included in 0.4.0 plus:
  • data-traffic has been speed-up a bit
  • dupe-control checks for PIDs now
  • project's contact address has changed to: hotsanic@bernisys.prima.de
IMPORTANT NOTICE FOR UPGRADERS:
The database in the traffic module has changed!
- You will need to run the update script in the module's subdir in order to adapt the databases. Otherwhise you will get strangely high traffic values or maybe in some cases a zero-line.


0.3.6 - 20011217
new features
extensions
bugfixes
  • The main settings now contain a switch to select the format of the generated images (.gif or .png). If you already have built own modules, please take care that they use the new feature ;)
  • All graphs now have create-timestamps in their titles
  • Bugfix in dnet module: OGR stats will no longer accident'ly be logged as DES.
  • Bugfix in ping module: makeindex.pl does no longer generate dead links.
  • The ping module now does a statistic analysis of its results to suppress annoying peaks.
  • New module: data-networks which uses iptables-accounting to let you know where your traffic goes to/comes from
  • data-traffic now supports Open/Free BSD ( Thanks to Matt Burke! )
  • data-traffic now supports SNMP (both GetBulk and SNMPwalk method)
  • data-traffic's initial scales are set down to 1000 bytes/s for any kind of device (better readable graph at low traffic)
  • data-traffic's diagram script has been upgraded to evaluate settings instead of executimg the settings-file
  • data-traffic now can show a bits/s legend instead of bytes/s only
  • data-network now can show a bits/s legend instead of bytes/s only
  • data-part now supports Open/Free BSD ( Thanks to Matt Burke! ) - the resolution is now 1024 bytes.
  • data-dnet now supports configurable logfile names ;)
  • HotSaNIC.pm (a perl library containing common functions) now exists to eliminate multiple definitions and redundant functions in different modules
  • all initializing stuff has been moved from "rrdgraph" to "rrdtimer"
  • rrdtimer now has a signal-handler for the TERM signal to properly shut down the daemon
  • the whole tool now should be CVS-proof ;)
  • some "setup" scripts now exist to create initial configurations easier. They are called automatically by the main "setup.pl" script.
  • "compile" now supports option "r" for recompiling (might be necessary after upgrading)
  • a new module "data-diskio" exisits which reports I/O stats of the pyhsically installed harddrives
  • data-worms has been rewritten completely because it wasn't sampling correctly at all. Should be ok by now.
  • data-system now has BSD-support ( Thanks to Matt Burke! )
  • data-system now supports sampling CPU stats for multi-CPU machines (images are being generated, but the index.html will not yet be created - But i'm working on it...) - Thanks Aide Florent for the ideas and some support!
  • data-system main CPU stats may be generated in percentage instead of absolute values
  • all modules can generate a browser auto-refresh tag now.
  • all thumbnail images are moved to the corrosponding subdirs on the webpage to re-gain more overview.
  • HotSaNIC.pm now contains a parser function for the settings. All modules have been adapted to use this function instead of parsing the lines on their own (read-data.pl and makeindex.pl are affected).
  • added a directory "tools" where a few helpers could be located.
  • tools: makeperlheaders (call if ping module complains about missing syscall.ph)
  • stop function in "rrdgraph" script is now BSD-proof :) ( Thanx to Peter Reich! )
  • tools: warnings.pl generates warnings if a defined threshold matches (for example 20KB/s on eth0 over the last 5 minutes)
  • data-traffic: diagram script converted to perl now
  • data-system: shows user-stats now
  • all modules: main options are being transfered to the read-data scripts by rrdtimer as commandline-arguments (saves parsing of mein settings over and over for some scripts)
  • setup.pl: option "y" tells the script that it shall use all available modules, settings are being backed-up instead of creating a settings.new file and leaving the original untouched
  • rrdtimer: debugging mode also available in daemon-mode (configured in main settings)
  • all modules: dupe control added. If a module is already running, don't start it again until the old instance exits (control-file: "running.dat").
  • data-system: now with NetBSD support
  • data-system: showing processes on FreeBSD
plus: many many little bugfixes i forgot to mention here ;)


0.3.5 - 20010923
bugfixes
new module
Bugfixes:
  • Missing directories in the module-subdirs are now being created as they should be. Sorry for the inconvenience!
  • I corrected some misbehavior with directories containing spaces. (correction may not affect every module yet - remember that the project is still in ALPHA state)
enhanced logging - timestamps are now present in some modules and logfiles get rotated when they reach the configured size. (at the moment only one backup is kept - cyclic backups will be implemented soon.)
A new module "data-worms"exists which counts the accesses caused by worm-activities in your apache-logfile.


0.3.4 - 20010918
rrdtimer
I have extended "rrdtimer" a lot!
  • The script now has many sub-routines which will make debugging much easier - well, at least i hope so ;).
  • The main "makeindex.pl" now has been integrated into "rrdtimer" to get the whole thing more compact. To generate the html index-files just type "rrdtimer i" - that's all. I left the "makeindex.pl" in this directory for compatibility, but this is just a shellscript that calls "rrdtimer i". ;)
  • The way you call "rrdtimer" has changed a bit - just type "rrdtimer" on the console to get some help.
Soon "rrdtimer" will be extended to replace "diagrams" and "convert.sh" as well.


0.3.3 - 20010916
bugfixes:
rrdtimer
I found a proper name for my tool: From now on it shall be called "HotSaNIC" which stands for "HTML overview to System and Network Information Center".
On execution "rrdtimer" checked for existing daemons using the wrong program-name (rrdgraph instead of rrdtimer).
"rrdtimer" doesn't any longer just execute "read-data", but also checks for uncompiled "read-data.pl" scripts, too - while "read-data" has the highest priority.


0.3.2 - 20010915
bugfixes:
rrdtimer
sensors:makeindex
I accidently left a debug-line "rrdtimer" script which produces unnecessary output to logfiles.
The "makeindex.pl" in the sensors-module had to be fixed to generate html indexes for each sensor as well.
I checked every script for mix-ups of integer and string comparison. There shouldn't be any errors by now.


0.3.1 - 20010911
bugfixes:
rrdtimer
ping:makeindex
ATTENTION!
There is a rather serious BUG in versions < 0.3.1 !
Please upgrade to version 0.3.1.

Symptom: If you restart the daemon it may happen that no modules are queried and thus your databases will not be updated.
Cause: This is because i mistakenly used "le" to compare two integer values instead of using "<=" under PERL. Sinc e i initially compare "120" with a timer-value which begins with "1000", PERL states that "120" is GREATER than "1000.*" - which is absolutely correct in a string context. Thanks to Bodo, Peter and Robert for the hint that made up my mind. :)
Cure: You may prevent this "overflow" by editing the "rrdtimer" script and simply exchanging all occurences of " le " by " <= ".
Fix: I have fixed this in version 0.3.1 and to be on the safe side i initialise "$last" with "$now-100" and "$lastscan" with "$now-1000" instead of zero. That should do it for a really really long time now ;).

The "makeindex.pl" in the ping-module had to be fixed a bit. Now it generates index-files properly again. Some bugs may occure since i will change the script's syntaxes and variables a bit to make it possible in future to create a library with global functions which are commonly used in the modules' scripts.


0.3.0 - 20010909
almost all ;)
Added a module that pings a list of hosts, stores the results in databases and creates nice graphs out of it ;)
The sensors module has been (almost ...) completely rewritten to give you full support to all sensor-types. BUT! You have to know how these sensors are represented in the /proc filesystem and how they have to be treated properly. The module just helps to gather the data. WHICH it shall gather ... well, that's about up to you. But if you have some idea what /proc is and where sensors are located it should be no trouble at all to configure the module. Hint: some sensors need proper scaling! This is (poorly) described in the settings file. You should read the lm_sensors howto to get the idea of what's happening and put the correct scaling-values in the settigs. Remember: Just use plain floating point numbers, not fractions like in lm_sensors!
The partition-plugin has been altered to support NFS volumes (and other) as well. To achieve this the "settings" file had to be adapted and the names of the databases have changed. Simply rename the old databases to keep your old recordings.
I also changed the whole behaviour a bit:
ALL CONFIGURATION OPTIONS for MODULES have been moved into the module-directories !!!
This makes it a bit easier to install or remove modules without fiddling with the main config, WHICH STILL RESIDES IN THE MAIN DIRECTORY!
The main "makeindex" script has been converted to perl to let it run faster (and it sure is MUCH faster now!)
The "compile" script now can uncompile all scripts using the "u" option
I began to convert some of the modules' "makeindex" scripts to perl to make them faster


0.2.6 - 20010805 (obsolete!)
modules
Added a module that gives you very primitive lm_sensors support (just the gl518sm sensors for now) It will sure take some time to get these to work properly...
The swap-space readings in the system-module are extended by a "max usage" line which plots over the whole thing to give you a hint about the "real" usage. This maybe helps a bit in structuring your system.



0.2.5 - 20010710
modules
I changed the index-generation scripts so they generate valid html 3.2 ...



0.2.4 - 20010607
diagrams
clearall
The "clearall" script has now an option to remove precompiled binaries created by "compile"

"diagrams" now checks if it has been called before and if maybe it's stalled somehow and kills it's evil brother ;)



0.2.3 - 20010605
rrdtimer
modules
Previously "rrdtimer" called all existing modules parallel, but this behaviour may cause problems on some systems, so i changed the default behaviour to call all modules in a sequence. The parallel mode is still available and can be activated by calling "rrdtimer p"

An additional module called "networks" has been created. It uses ipchains to measure the traffic to dedicated networks and evaluates these traffic samples pretty much like the "traffic" module - except min/max plots which would be quite a thing to implement properly. Maybe this will come some day as an option, but i don't think so right now.
A speedup has been added:

Try to run the "compile" script to compile all modules via "perlcc". This may give you a major speedup ( about 2-3x ) because the perl modules don't have to be compiled over and over again before they are executed ( this happens each 10 seconds ) - but be aware that errors may occur if some important perl-libraries are missing! In that case you have got to live with it and reconstruct the "read-data" scripts in each module directory.

I changed the global behaviour a bit. "rrdtimer" now calls the diagram scripts itself - so cron-entries aren't needed anymore.

The modules now generate timestamps in the "rrdtimer.err" logfile.

The name of the PID-file is now free configurable in the settings file.


0.2.2 - 20010514
rrdtimer
databases
system
Debugging mode in "rrdtimer" now only shows 4 digits after decimal point for better readability.

All databases have been moved to a subdirectory "rrd/" in each module-subdirectory. So please move the .rrd files into each module's subdir to the corrosponding <module-dir>/rrd/ subdir.

Additional logging-functionality for system-module. Now the loadavg values for the 5min and 15min average are logged as well. The process-stats show an additional value "disc" representing processes being swapped to disc at this time.

IMPORTANT NOTICE FOR UPGRADERS:
Be sure to run "update.pl" after you installed the new data-system module! The system.rrd database has to be split to add the new logging-channels!
Another possibility is to just let the new scripts run and remove system.rrd, but all previously sampled data will be lost by doing so!


0.2.1 - 20010510
traffic, dnet, part
convert
settings
Some bugs were fixed in the traffic, dnet and part modules:
  • the "diagrams" script in the traffic-module produced some errors due to a wrong variable-assignment.
  • the "diagrams" script in the part module produced errors because "rrdtool" was called with some parameters mixed up.
  • the convert.sh script only converted the graphics from system and dnet. Now it handles all graphics.
I played a bit with the settings-file and some of the scripts. Now there is a configuration-entry to set where logfiles will be stored.
Then i altered the behaviour of the DNET-module. Now you can configure networks and their names to receive better readable statistics for dynamic IPs. This is done in the "data-dnet" subdir using a file called "networks.cf"
Some speed-ups were achieved by using hashes to store multiline values and by changing file-access behaviours a bit. Saved about 10% execution-time :)



0.2.0 - 20010509
rrdtimer
dnet
Well, this is something revolutionary ;)
I had the time to read the PERL-FAQs and found some interesting aspects like daemonizing (i.e. fork into background) and other stuff. So I decided to completely rewrite the main script. This is no longer bash, but plain PERL! Included are some features like a debug-mode with different debug-levels to get your modules straight. Just experiment with it if you like. :)
Some changes were made to the dnet module as well. I've rewritten the routines into PERL now and i hope they will do what they are supposed to. I don't quite know, so please inform me if anything goes wrong there.


0.1.9 - 20010506
dnet
I had some spare-time to spend, so while I was chatting on ICQ, I decided to give myself a little push and have almost completely re-written the core of the DNET module in PERL. Speed-up seems to be a factor of about 10, though this was not really needed since the module gets called only once a day anyway. But major speed-up has been achieved in building the initial databases which will reach back a couple of months ... these are about 400 (for each host !) database updates sequentially and this took quite a while back in BASH ;)
I also discovered and removed a little bug in this module. When a host has not sent any block the last day, then the database would have taken the values of the day before. This looks rather impressive in the statistics, but is of course dramatically wrong :)
Ok, these stats are just looking fine right now.


0.1.8 - 20010421
diagrams
traffic
some changes were made in the "diagrams" scripts. Now the diagrams are created in "interlace" gif mode, which will improve the load-behaviour of the webpage.
I also tuned in some additional "rrdtool"-features - now the byte-oriented graphs have a legend on the Y-axis and 1K is calculated by 1024 and not 1000 as before.
The traffic module now can cope with asymmetric interfaces as well, the MAX option in the settings-file has a new syntax to set in and out speed independant (of course as before in bits/s)


0.1.7 - 20010408
traffic
further little speed-up was archieved by not using so many temp-files in the traffic-plugin.


0.1.6 - 20010405
read-data
settings
A little speed-up was archieved by doing some changes in the "read-data" scripts for the system and the partition plug-in. Some awful shell-calls were replaced by PERL calls which do not write in sample-files any more, but directly into the PERL-array :).
Also some new setting-file entries exist now to tell the program which interfaces/devices are to sample and which stat-graphs are to be bulit.
However, this applies only for the sample-part of the traffic and the partition section for now... but i'm working on it ... ;)


0.1.5 - 20001219
makeindex
Corrected a little bug in the "makeindex" files. Weekly stats indexpage will now be built correctly.
The "makeindex" script now removes all temporary files it creates. It also doesn't nag about existing directories, it just silently creates them if needed.


0.1.4 - 20001215
makerrd
Corrected a missing settings-call in the "makerrd" scripts which caused the tool to create no databases at all.
Thanks to Markus Bajohr!


0.1.3 - 20001209
part
I just corrected a bug in the part-plugin which caused the script to create numerous databases with no further use, but a lot of load while building the graphs ...
Thanks to Michael Neumann!


pre 0.1.3 I report changes and bugs since version 0.1.2 - all earlier versions were just like the beginning of all life ;)

o.K. let's see. What can "rrdgraph" do right now ?

The system-plugin ( This was my first plugin. Actually it hasn't been a plugin up to 0.0.1 ) will collect data about:
  • processes (sleeping, running, stopped, zombie)
  • CPU time (idle, nice, user, system)
  • loadaverage
  • memory (free, buffered, shared, cached)
  • swapfile (free, used)
The traffic-plugin will collect data about all your local network interfaces except loopback.

The partitions-plugin collects data about your partition-usage.
Some bugs have to be fixed here!
For example network-mounted directories will be saved under just a part of their name. This may cause serious problems if there are two similar paths... I haven't checked yet, but I guess it will.
For now I will override this bug by filtering out all network-mounted filesystems. Later i'll try to convert the "/" in the device-pathnames to "_" and hopefully all will be fine in future...