Mailing list outage

From Bob's Basement

Start of outage: Afternoon of 2/6/07
End of outage: 6/6/07, 0:43
Systems affected: Mailman mailing list service

Issue

On 5th June it was noticed the mail was not being processed for the Bob's Basement mailing list. Further investigation also revealed that the mailing list web interface was reporting an error. The mailmanctl script and the qrunner scripts were not running. Attempts to restart the mailmanctl script was unsucessful. The script and qrunner processes were exiting without logging or printing errors.

Cause

This was caused by some changes to file permissions on the server, which were carried out as part of maintenance work at the weekend.

Resolution

The qrunner scripts were started manually to allow mail to be processed. Once the root cause was established, the erronous file permissions were corrected and the mailmanctl script restarted sucessfully.

Lessons learned

1) Don't play with file permissions
2) Don't use Mailman