Mozdev sysadmin meeting minutes for 2007-11-20

Present: David Boswell, (davidwboswell), Doug Warner (silfreed), Gerry Murphy (gjm), Chris Neale (cdn-work), Brian King (kinger), Eric Jung (ericjung)

Discussion was held publically in #mozdev

Discussed developer priorities

  • a little Drupal work (theming fixes, install profile fixes)
  • Thanksgiving theme
  • replaced mirror content
  • going to start putting news on front page from mozdev.org blog feed
  • would like to serve files directly from mozdev.org if the content on the mirrors is outdated
  • will be redirecting users to their closest mirror soon
  • gjm picked next Monday @ 10:00 UTC for the BZ 3.0 upgrade

Discussed sysadmin priorities

  • mailman is working without problems now
  • one of the list admins set the bounce address to the list admin address, causing a very large bounce queue to grow
  • mailman devs created a patch for this and it was applied to our servers
  • the bounced mail problem has been going on since May 5, 2007
  • outgoing queue in mailman was increased from 1 process to 4 processes; postfix outbound smtp clients has been increased 50%
  • held message nags are still causing most of the daily email lag to lists
  • still tweaking rewrite rules on vebzom.org

Mirror outage

  • content was accidentally deleted when stage was synced with production
  • Doug needed to fix up the mirror sync script because it wasn't maintained in CVS
  • the CVS tree still needs synced from production to staging; gjm will take care of that

Vebzom.org downtime

  • TWS rebooted the server without plugging a monitor in, so we don't know if anything was visible on the console
  • gjm was unable to find anything in logs
  • no ideas about what the problem was, unfortunately

MySQL Backups

  • setting up a replication server on vebzom.org is one option; would increase traffic to vebzom.org, but would totally elimninate the locking issue
  • splitting out bugzilla tables into its own database will speed up the dump of the tigris database and clean up the database at the same time
  • gjm an silfreed will coordinate on splitting the tigris db and updating code