Mozdev sysadmin meeting minutes for 2007-12-11

Present: davidwboswell (David Boswell), ericjung (Eric Jung) gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Mozdev downtime

- no obvious causes for downtime found yet - box was logging up until the server rebooted, so it was still active - suspected hardware problem since the box had tremendous uptime otherwise (418 days, IIRC) - we can possibly plug in the 2nd NIC in case there is a problem with the current one - contact information between parties was missing (except for the site), so contacting the right parties was difficult; David will collect contact info and distribute it via email so everyone has a copy

Disaster Recovery

- How can we get the news about a site outage out to people? - sysadmins are looking into restarting services automatically to try to recover from problems similar to what we experienced - sysadmins will put together recommendations on how to handle outages like this in the future - sysadmins will get Doug access to the KVM so he can debug (different timezone, so we cover more waking hours)

Discussed developer priorities

- mirror bugs are done and implemented (closest mirror redirection, downloads directly from D.MD.o, and download files rss feed) - server load from additional mysql queries seems to be minimal after freeing up memory on the server last week - lots of discussion on mozdev providing an update.rdf feed for projects - renamed RSS feeds to remove the .html extension - setup lots of projects with bugzilla permissions - Drupal 5.4 and 5.5 security upgrades - working on a session and authentication system for the new features - been doing design for project tagging - poll is ongoing to see whether to enable the download file rss feed by default

Discussed sysadmin priorities

- memory leak in Apache was tracked down to a small portion of Drupal code but not php-mysql related (specifically, the user registration pages but the problem wasn't tracked down more than that) - apache2/php5 upgrade on vebzom is progressing; this seems to fix the memory leak in Drupal as well - nightly processing of mailman held queue was crashing due to a bug in the french translations

Open sourcing more code?

- Pete says to use the hovercraft project, but otherwise didn't raise any objections - CVS patches are in hovercraft but will probably be difficult to update to current CVS - other mozdev code shouldn't be hard to get back into CVS; just needs to be planned well in order to minimize downtime - David will look into contacting people @ collabnet to see if anyone still maintains a mysql auth patch for CVS