Server monitoring is horrible. I wrote about it while I was just beginning to hack what I had hoped would be my solution.
I accomplished most of those original goals but the project didn't work as a whole.
To handle scaling and the large amount of configuration and state need for a monolithic monitor, I used Terracotta clustering. With Terracotta, I was able balance running tests over monitoring nodes to help handle the load. I could also write straight-forward code using normal Java locking semantics, without getting bogged down in database details.
My frontend development skills got a shot in the arm, too. Much of the interface was dynamically drawn using JQuery, and I learned a great deal more about CSS.
Many of the tests and plugins I wrote using Groovy, which I credit for warming me to dynamically typed languages.
But sadly Leemba never caught on. Despite the huge effort that I put into making it easier to use, the clustering was still difficult. Developing (and even sometimes using) Leemba required Terracotta knowledge. Without Terracotta, Leemba would run but all the configuration was just stored in memory, so that is really only for testing.
The Terracotta experiment bogged down after a while, too. To do the equivalent of schema changes, I was writing scripts to modify cluster memory. Everything had to be carefully written to take advantage of the cluster. Then, when I had just about perfected it, Terracotta deprecated the method I was using and tried to push developers to using ehcache or pre-bundled plugins. I'm not even sure if Leemba would still run on a current cluster.
Terracotta is still amazing, but trying to use it as a primary data store wasn't.
Still, I continue to use it for my monitoring. And I learned an incredible amount.