A so long awaited v.9.0 is here! :-))
The main "global" change in dim_STAT v.9.0 is a move to the latest MySQL 5.5 GA version. This move was driven by willing to use a true transactional storage engine within MySQL (which is InnoDB) to avoid any database files repairing in case of systems hang or unexpected power-off.. - Now you're free to choose which storage engine you prefer to use for your data and select MyISAM or InnoDB on your database creation. As well at any moment you're able to convert any database to InnoDB or MyISAM according your needs. As such a process may be quite long enough with big databases, there is a command line tool dim_STAT-Admin coming since v.9.0 - you're able now from a command line involve DBA actions:
- Create a new Database
- Convert existing Database to another Storage Engine
- Backup a whole Database
- Export STAT Collect(s)
- Import STAT Collect(s)
- Recycle STAT Collect(s)
dim_STAT v.9.0 is shipped with a "remastered" MySQL v.5.5.15 Community Edition (GPL) - it's different from the "official" MySQL 5.5 by being more compacted (all debug and test stuff is removed) and compiled in way to have less dependencies to work just out-of-the-box if possible (for ex: on Linux you'll not need AIO library, etc.). The configuration parameters are mostly present within "my.conf" file, all settings are just minimal to not use too much RAM or disk space. On the firs start InnoDB data and log files will be created, but you may adjust the settings later.. Well, my goal was to ship a compact and just working MySQL with dim_STAT, but feel free to replace it by any other MySQL version! ;-)
Another new stuff coming with v.9.0 is an experimental dim_STAT for MacOSX ! :-)) I've preferred to share it rather just enjoy to use it alone on my Mac ;-)) However, I did not yet fully tested it as I don't have (yet) another Mac ;-)) and don't want to spend any time to install MacOSX in VM.. - if somebody will want to use it on Mac I'll have first news very quickly ;-)) if no - another reason to not spend too much time on MacOSX distro ;-))
Then there is a huge list of "internal" improvements which will say you nothing, but will greatly improve your usual work with dim_STAT ;-)
But main "visible" new features are:
- simplified INSTALL process (and I hope you'll feel it since the first installation ;-))
- MySQL regular expressions can be used now within Multi-Line stats to enforce name criteria..
- name filter is added into all Multi-Line pages to be able to reduce the list of names when it's too long and simplify selection on ly of the names you need..
- value distribution graph is now working not only with Java Applet, but with PNG too..
- for any stats while drawing some data you may also (or instead) draw a sum of all selected values (ex: Usr% and Sys% CPU Usage as CPU Busy%)..
- absolutely amazing Multi-Host Bookmarks ! - gives you a way to analyze several hosts in parallel in more advanced way: you're analyzing stats within a Single-Host mode first, applying all criteria you need (ex. seeking only for "oracle" and "sqlplus" processes CPU Usage, and then saving the obtained graph as Multi-Host Bookmark, and after with a one click analyzing the same graph but with many hosts within Multi-Host mode ;-)) (NOTE: don't forget that only one stat value is allowed to be presented on the Multi-Host graphs, if you're drawing more than one value on your graph - the Multi-Host Bookmark option will not be presented..)
- dim_STAT-CLI is now able also to draw Multi-Host graphs if several Collect IDs are given in argument (-ID id1,id2,id3,..)
- Export is now have a time interval option in case you need to export a particular part of your Collect rather the whole collected data..
- STAT-service is now also reporting when the connection was lost with a server or a stat command closed it's pipe (possible abnormal finish), as well such messages are appearing in errors on the Web interface..
- by default any non-system stats are disabled (commented) now within STAT-service - I was surprised how many users are mostly clicking on all available stats and making troubles on their systems (for ex. involving DTrace for non-existing stuff, etc.) - since v.9.0 you have to enable it manually by editing /etc/STATsrv/access file (which is supposing that user is aware of the changes ;-))
- HAR2 for Solaris 10 was updated to version 2.1 (thanks to ISV team! ;-))
- LioSTAT_v10 (iostat v.10) was introduced to Linux stats (seems to work better with recent Linux distros)..
- mysqlSTAT was updated/improved
- innodbMETRICS was introduced (still experimental) - allows to monitor InnoDB METRICS table within MySQL 5.6
- oraASMIO was introduced to monitor I/O activity via ASM on Oracle RDBMS (including Exadata ;-))
- Add-On management was yet more simplified and improved, and since now you're also able to re-edit any of your Add-On descriptions directly via Web interface!! ;-))
- etc..
Well, there are many other still visible, but minor features, but I'll avoid to list them and leave you to discover them yourself ;-)) There are also many other features I've planned to ship with v.9.0, but it was urgent for me to ship the new version as soon as possible, so my main priority was focuses on the most "critical" features related to MySQL 5.5 adoption and general usability (like Multi-Host Bookmarks and dim_STAT-Admin). My goal is matched now, event it took me a major time of my week-ends and vacations during last 6 months ;-)) but there was an urgence because since next week I'll become a "true" Oracle employee, and I've to suspend any new dim_STAT development since now to avoid any potential conflicts regarding software proprietary (or until any changes will come in situation).. - shipping v.9.0 today on the remaining conditions gives me a satisfaction point for over 14 years of development ;-)) and it was hard as well to suspend my developments without shipping v.9.0 before.. ;-))
So far, there is still missing Solaris 10 x64 version (did not have an access to a needed server on time, so sol10-x86 version will come later ;-)) and Solaris 8 /SPARC is missed too as I don't have an old enough server available around me, so cannot guarantee the dim_STAT v.9.0 will come later for it and don't know if there any need.. - a simple PC is now faster than a very old SPARC, so a simple Linux box makes more sense now days ;-))
However, I hope to still be able to continue support of dim_STAT (and fix bugs if any), just only new features will be suspended until a new order ;-))
Hope it's the last 8.x version release and the next one will be really the v.9.0 :-)
Curiously there were still many things to finish before to say yes, we may start a new version tree :-))
First of all I was surprised to discover some minor bugs and prefer to fix them still within v8 rather to bring them to the v9 again :-)) Then there are some new and very powerful improvements which cannot wait so long :-)) And finally I've also wanted to bring some order into distribution/packages version-ing and resolve historical ambiguity of names :-)
So far, this release is jumped to the v.8.5 not because you're missed v.8.4, but just because within a STAT-service packages I've started to tag new updates with "ready for 8.4+" which bring ambiguity when you're still using the v.8.3 :-)) So, since this release you may find a VERSION file included withing every STAT-service package, dim_STAT distribution, or already installed dim_STAT server. This file contains the version number of dim_STAT as well the OS/platform information. Hope it'll simplify many things from now :-))
As well, the list of all main changes coming with release:
Fixes :
- Database Password fixed (curiously nobody reported it until now - it's fine to know that most of users are not paranoid :-))
- CPU Count was not exact in some conditions within Multi-Host graphs
- Empty graphs are skipped within Report generations
- Updated documentation :-)
Improvements :
- New Analyze option: Extended Multi-Host Analyze - it combines now the classic multi-host interface + all single-host bookmarks! - you're able now to mix on a single page your multi-host graphs and per host graphs coming from bookmarks! :-)
- Bookmarks links now also available for the Multi-Host too - at any time get a more detailed view per each host available via Bookmark graphs! :-)
- Surprisingly there was still a room for performance improvement on WebX - new code is running even faster than before and particularly when generating a long list of graphs! (which is also dramatically reducing a generation time of Reports :-))
- STAT-service is now accepting the syntax with command_name[:port] - it gives a possibility to run several STAT-service daemons in parallel on different ports, and according the port number assign a different final command for the same "stat name" (for ex. see how to collect data in parallel from 3 different Oracle instances running on the same server: name[:port] feature )
- Graphs accidentally involved with a too large list of items now will be auto-reduced to the lower number of items and include a short error label within a legend
EasySTAT additional options :
- Additional LOG option may be used in cases the EasySTAT output should be redirected to a file rather to be printed to the standard output (when it should to be not mixed with output from other programs/daemons running in parallel)
New Add-Ons :
- UDPstat (Solaris: UDP traffic stats)
- innodbMUTEX (MySQL: detailed InnoDB MUTEX monitoring (available since InnoDB plugin 1.0.4))
Just a minor update-1 for 8.3 version - cannot wait until v.9 :-)
Here is a list of changes coming with update:
Fixes :
- dim_STAT-Server script on start did not restat active collects (special thanks to Didier DAHIER to find it!)
- dim_STAT-Server script is now killing all clients too when involved with "stop" action
- Standard Bookmark kit was not installed by default within a template database (special thanks to
Peter Hausdorf to find it!)
- Linux iostat binaries updated for STAT-service to fix broken output in some cases (special thanks to
Melanie Ashworth-March to find it!)
Improvements :
- Added UNIX signal traps to WebX
- Replaced crypt() function to reduce WebX binary dependencies (now again compatible back to Solaris 2.6)
- Added active collect(s) information (extending active connections)
- etc..
Database schema :
- Bookmark name size was increased to 32 characters now!
EasySTAT additional options :
- COMPRESS and TIMER options are now integrated within EasySTAT.sh and enabled by default
- Run forever if Hours == 0
- Supporting integration within /etc/inittab for automated stats collecting
- Special actions are involved on signal trap (see doc)
- pid file is created within a working directory to simplify EasySTAT process identification
New Add-Ons :
- mysqlLOAD (compact MySQL stats oriented multi-host monitoring)
- pgsqlLOAD (compact PostgreSQL stats oriented multi-host monitoring)
This is a maintenance update release, just to fix all trailing issues
before a big move to the v.9 :-))
Here is a list of the main changes:
Fixes :
- security issue on STAT-service!! (special thanks to Marc Koderer @LHS to find it!)
- execution dim_STAT CLI from the web services
- fixed LANG=C and LC_ALL=C via the main STAT-service script
Improvements :
- Optional Admin password may be given on a database creation and then it'll be required for any admin action!
- Database error messages are now handled separately from LOG messages
- Java WebDRAW Applet updated
- add color LEDs near hostnames to signal running or stopped hosts
- add #Records column to have an idea about a number of database rows per collect
- Admin LOG management is working more friendly now with a long lists of messages :-)
- STAT-service daemon may listen exclusively on a given IP address now [IP:port]
- Whole database backup is added to the Admin menu
- etc..
Database schema :
- iostat disk names are moved to varchar(64)
- all float values are moved to float(25,5) to have an easily human reading :-)
- Add-On name limit is 14 characters now
Report Tool :
- wiki-like syntax is added to simplify writing!
- look & feel updated
- some minor fixes here & there :-)
EasySTAT additional options :
- COMPRESS=gzip (to automatically compress each output file)
- TIMER=yes (automatically adding timestamps and sync tags within output files)
New Add-Ons :
- HAR v2 (CPU chip counters, ex. mips, cache miss, etc. read more... )
- Solaris CPU-Set STAT (mpstat -a)
- Linux MPSTAT v2
- MySQL STAT
- InnoDB STAT
- InnoDB I/O STAT (adoption of Neel's DTrace script )
- PgSQL STAT (PostgreSQL stats)
- ZoneLOAD: zoneid => zonename
Major performance update release (code name "Speedy" :-))
Before to add some completely new features into dim_STAT, I wanted to make a point on the current development and re-look/re-design dim_STAT internals in way to break existed limitation, increase an execution speed and improve a general usability... We operating now with more and more servers running in parallel, with more and more huge amount of data to analyze, and more and more long periods of activity.. Most of solutions for incoming problems are already covered in this release.. And hope you'll be happy to discover by yourself the newly optimized dim_STAT version running x4 to x20 times faster then ever! The amount of work hidden in this release is huge.. And I address my special thanks to my friends and team-mates Matthieu and Alain, who found and noted numerous "speed-less" cases, tested each step by step improvement, and enjoyed finally the end result.. I hope all of you will enjoy the final result too :-))
Performance:
- Fixed/Forced the right query plan in all MySQL queries!
- Re-designed SQL queries for speed!
- Break 4G limit for database files, current file size limit is 2TB now!
- Re-designed WebX internals to speed-up the name tree management!
- Huge speed-up on PNG convertion!
- On-demand "More>>>" link is used now for a heavy distinct list of names...
Improved Multi-Host:
- ready to operate hundreds of servers
- added per host name grouping feature
Other features:
- host list redesigned
- host name aliasing: host[/IP][:port]
- compact/vertical/horizontal STATs List on new collect
- added MIN/MAX grouping whenever it makes sense
- Improved Add-On interface
- Database name is included in every graph URL now
- MySQL is flushing all data every 5 minutes now
- CLI added -Data option
- Added separated Solaris10+ oriented STAT-service package for SPARC
- Added IOpatt, CoreSTAT, Siostat Add-Ons
- Pre-integrated IObench and db_STRESS
- few bug fixed..
Documentation:
- Updated! special thanks to Willem van Schaik for his numerous corrections and his hard editing work!!!
- Extended.. :-)
Minor update (polished) release, but with some significant changes anyway :))
First of all, Start New Collect(s) interface is jumped to the next evolution step: there was too much options presented on the same time, as well on other side - people don't have a reflex to enable needed Add-Ons and then they are surprised to not see needed stats... So, now ALL Add-Ons are enabled by default within any newly created database. And then, only enabled stats in STAT-service are proposed via New Collect interface! (means old STAT-service versions are not accepted anymore).
Well, you need to try once to see what I mean :))
STAT-service: since v.8.1 STAT-service daemon source code is distributed under GPL. So anyone may compile it on any other UNIX systems and collect data from servers not running Solaris or Linux. As example, pilot package for HP/UX is shipped.
Special Solaris 10 extensions: (SPARC and x86), new stats gathering load per:
- Zone ID
- Project ID
- Pool ID
- Task ID
Some Very New features:
- dim_STAT-CLI module: you're able to generate stats graphs directly from command line!
- dim_STAT-Recycle: recycle your database space automatically via cron!
- Auto-Sync-to-Time: all running collects are automatically syncing every hour to the current time to avoid any possible time scale shift between different stat command!
As well some other improvement:
- host name pattern mask (some people collecting hosts by hundreds!)
- improved BatchLOAD
- fixed random zeroed graphs
- some javascript code to select/unselect/resize
- split to several pages option during Report generation
- newer htmldoc pre-integrated
- etc.
This release was initially prepared as minor update, but all new development touched so many thing and mostly bring changes in fundamental design - so I decided make a point in new features before Christmas and ship finalized work as new 8.0 version (Santa Claus edition :))
Completely NEW features:
- new STAT-service kit now self-publishing its list of enabled STATS!
- web interface cooperate with STAT-service and protect you from wrong/disabled STAT data requests!
- color LEDs signaling each host state before you start any new STAT collect
- only enabled in STAT-service commands are proposed to collect
- new rewritten all-distro 'vmstat' for Linux
- new netLOAD for Linux
- new psSTAT for Linux (similar to Solaris)
- new ProcLOAD for Linux
- new UserLOAD for Linux
- new EasySTAT kit for Linux
- etc...
Analyze:
- group data by last N letters in attribute name (disk, process, etc.)
- special selection via name pattern
- Load LOG messages from plain text file
- Truncate too long LOG messages (optional, +popups)
- even more improved Report Tool!
STAT-service:
- jvmSTAT is fully based now on the official jdk 1.5
- new/modified STATs for Linux
- MEMSTAT for Solaris is back
- new netLOAD version for Solaris
- etc.
Documentation:
- Updated!
- Special Linux updates!
- etc.
This release bring a lot of new features and more improved design. All this new staff demanded a lot of in-depth modifications and sometime partial code rewriting, etc. It took a more long time I expected initially, but I like the result very much! :))
Completely NEW features:
- You have a choice now to use Java Applets or PNG images for graphical output!
- You may at any time restart any stopped collect now!
- dim_STAT distribution for Solaris/x86 is here!
- EasySTAT tool added into STAT-service Solaris package!
- etc...
Administration:
- Simplified install, no more /apps needed!
- Main dim_STAT-Server script manages start and stop actions, repair database, restart collects, etc.
- Recycle solution added to administration tasks
- etc.
STAT-service:
- EasySTAT added
- ProcLOAD added
- UserLOAD added
- pkg for Solaris/x86
- etc.
Documentation:
- Updated!
- Report Tool chapter!
- Special Linux chapter!
- etc.
- Bookmarks:
- Bookmarks-links instead of STAT-links!
- Option to restore Standard Kit of Bookmarks
- BatchLOAD Tool! (easily adaptable for any load from output files!)
- there is even already done first convertor from GUDS to BatchLOAD! (by Antoine.Percher@France)
- Collecting Improvements:
- ioSTAT disk names auto-compressing (MPxIO)
- new version of netLOAD +nocanput
- STAT-service auto-eject by timeout (in case of halt or network problem)
- Pre-integrated Add-On STAT(s): Linux, Oracle, JVM
- LOG Messages:
- Dynamic LOG view with active message position bar
- Auto-insert error msg in case of any SQL-error
- LogMASK option everywhere
- LOG Admin
- Re-looked Interface:
- Show number of active connections for each database on the server
- Many settings saved by cookies
- Applet compressed view (never out-pass)
- More Applet colors :))
- Option to force on time scale graph alignment
- Info bullets: Start collect, Bookmark-links, Report
- Match Patterns may contains logical "&&" and/or "||", like: "*toto*||*titi*"
- General Install/Config:
- MySQL installation doesn't conflict anymore with any already installed MySQL instance
- Distribution for Solaris/SPARC moved to Apache + upgraded 3.23 MySQL
- Distribution for Linux re-compiled statically, so run on Any Linux version now!
- Solx86 STAT-service kit included
- Improved Java2GIF
- New Documentation!
- Created mail-list dim_STAT@sun.com (internal & external users!)...
- MultiHost: Multi LOG + Multi graph
- Multi LOG
- Multi Bookmarks
- Relooked interface...
- DB space info
- Start collect DEBUG option
- Real buttons
- Table text output file
- Linux version!
- T3stat included (based on code of Takao.Fujiwara@Japan.sun.com)
- netLOAD is no more shell wrapper, but well optimized binary (based on code of Yasunari.Saito@japan.sun.com)
- USIII stats included (via harUS3 collect (har program))
- Keeping selection of last collect via cookies
- Any collect selection may be preloaded by its name
- Active-ONLY option added to iostat collects
- Real/Continuous option for Graphs (presents hole/zero in case there is sometime no data for surveyed object (inactive process, for example))
- Max/Min value options extended by Number/Total to clerify result representation (for example if you are looking for most used disk during last hour - you should use "Max-Total" Busy%, but if you are interested by pic
activity - "Max-Number" Busy%, etc.)
- Long lists of log-messages are presented now via scrollable list
- STAT-links now may be presented/hidden in the same/new window, option also saved via cookies
- Active database name is presented now inside of every window titles and also on the main page
- Log messages now may be added on-line via Multi-host view also
- Database querying is better optimized to get reports faster
- Collect goal is changed to "never die" - even you stop your database server, collects will continue after restarting, etc.
- SysINFO service added to STAT-service package (short machine description)
- Report Tool
- dim_STAT data may include several hosts now on the same time
- SysINFO datatype added
- Java2GIF package introduced - easy way to convert any HTML document containing dim_STAT applet(s) into HTML + GIF image(s) document(s)
- htmldoc tool included - convert your HTML files into well presented PDF document (http://www.easysw.com/htmldoc)
- etc.
- Server/Session cookies management via WebX
- Preferences setting
- Possibility to use more than one database according needs
- dim_STAT Bookmarks introduction
- Integration of Bookmark interface to the Report Tool (dim_STAT datatype)
- New datatype in the Report Tool - dim_STAT Snapshot (uses simple saved html page during STAT(s) analyze)
- Some improvements in STAT-service package
- psSTAT is much faster now especially during hard workloads, also has several new options like: nlwp, refname, -N names, etc.
- etc...
- Report Tool introduction (most important)
- Graph Applet improvements
- Resolution of strange problems with winter time
- WebX program new features, more speed
- Added netLOAD auto-configuring according kstat output
- More STAT links
- etc...
- psSTAT 64bit fixing (most urgent and important)
- installation flexibility improvements
- etc..
- Top-N selection option added for all multi-line STAT(s) (you may see service time of most busy disks, for example; etc.)
- improved performance of WebX program
- etc.
- harSTAT pre-integrated (har is Sol8 CPU counters viewer by Frederic PARIENTE http://techtonic.france/~fparient/har)
- netLOAD fixed for Gigabit Ethernet by Didier HERMANN (Sun France)
- improved interfacing with HTTP server (real-time flushing)
- some nice improvements in STAT-service
- anySTAT introducing as a set of examples to simplify new STAT creation
- STAT-service introducing
- "Active processes only" psSTAT option
- Add on-fly messages to LOG
- netLOAD pre-integrated
- memstat pre-integrated
- New version of WebDraw Applet (+date markers)!
- More friend user interface
- Administration functions included
- User dialog to Collector Script Generation
- Adding ioSTAT collection (hacked iostat), no more need disk conversion on Sol2.5, output is really regular
- Documentation :)
- New version of WebDraw Applet (alive graphs)!
- Multihost Parallel Analyze
- Direct collection executing
- Adding NEW STAT(s) support
- Installation procedure changed (you may install where you want, etc.)
- More compact presentation of options + LOG message intervals
- Direct-links to CPU usage, Paging, Debit I/O, Top 10 processes, etc.
- Solaris 7/8 compatibility fixed
- Suspend/ WakeUp/ Truncate STAT(s)
psSTAT analyze:
+ PID list option
+ Active Process Only option
+ Number of processes counter (very useful to analyze real parallelism of your Oracle instance :))
+ Grouping stats by several letters from process name
Collecting:
+ ignore first (summary) measurement from all statistics
+ automatic conversion sd??? to c?t?d? from iostat