I’ve been getting frustrated trying to deal with a problem with one of our servers progressively getting more and more bogged down. Problem was that cpu utilisation, memory use, and network traffic have been pretty much constant over the last few months so it had to be something else.
Then I recalled that I’d set up mrtg to graph disk I/O and one look at the graph showed a significant symptom. Over the last six months the hard drive output had been progressively rising and was now 10 time more than it was in January. So something was increasing the load on the harddrive.
So I thought “what is on the harddrive that keeps growing?. Backup files? No they’re pretty much consistant in size. Log files? No, they get rotated every week. What else grows..? um er I know… mailboxes..!” And that was exactly what it was. Some POP3 users had turned off the delete function in their email clients and their emailboxes had blowout. Everytime they popped their mailbox the server had to crunch through several hundred meg of emails. Multiply this by a dozen users popping their mail every few minutes and you’ve got a massive data bottleneck.
A quick email to the main offenders was sufficient to get the server back on track again and half a day later everything is sweet.