Mozdev sysadmin meeting minutes

warning: Creating default object from empty value in /var/www/drupal6.pyxos.net/drupal-6.28/modules/taxonomy/taxonomy.pages.inc on line 33.

Mozdev sysadmin meeting minutes for 2008-07-01

Participants: davidwboswell, (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

  • working on WYSIWYG for Mozdev wiki
  • planning on using mediawiki syntax with FCKeditor (noticed DevMo is switching to DekiWiki)
  • at ccaygill's prompting (and patch) updated the local.conf documentation
  • performed some log analysis of Firefox 3 release traffic
  • discussed alternatives to awstats for performing custom stats analysis (no solution found)
  • hg was just setup in production; needs tested and then we can open up to projectowners for testing
  • once hg is in testing w/ POs I'll be back at working on the wiki and also with new file upload mechanisms

Discussed sysadmin priorities

  • some security updates
  • mailman upgraded to latest version
  • hg setup in production; still needs SSL cert

Web stats/Top 50 page

  • stats have caught up
  • hits for 2008-06-(17-20) are impressive (FF3 release)

Staging server migration

  • mozilla has recently been looking into racking a ton of Mac Minis and OSUOSL just received a shipment of hard drives
  • david will ping mozilla again and gjm will get back to OSUOSL

Next meeting July 8th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-06-24

Participants: gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill

Discussion was held publically in #mozdev

Discussed developer priorities

  • finished new bugzilla bug entry form
  • helped with Firefox 3 server load (adding additional caches, mostly)
  • fixed XPI sorting on project overview page
  • working on improving wiki formatting at Mozdev
  • investigating integrating Mailman and Drupal Forums
  • updated VCS log to include link to a file's history

Discussed sysadmin priorities

  • no updates, under the weather
  • tanker's out on leave right now; when he get's back they'll work on the OS update on vebzom and then rework the plan for production

Web stats/Top 50 page

  • ccaygill has been working on mycroft to reduce hits to old plugins that won't have updates
  • munin outages/gaps are related to stats maching blocking on something the stats machine despite it being idle; needs more investigations
  • gjm has begun looking at firewall connection table to catch bots and evil users rather than just relying on apache logs

Staging server migration

  • no updates

Next meeting July 1st, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-06-17

Participants: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill

Discussion was held publically in #mozdev

Discussed developer priorities

  • finished up changes to simpler bug entry form
  • increased timeout on new Mozdev pages that require login (file managment, project tag editing)
  • wrote Mercurial documentation for Mozdev
  • Firefox 3 release is today

Discussed sysadmin priorities

  • mailman upgrade is in the works; all our custom changes have been included in the new version so we won't be running a hacked version anymore
  • looking into using snort for detecting bot attacks on the server; not sure if itwill be feasible or not
  • ssl cert w/ multiple domains should be good to install

Web stats/Top 50 page

  • still somewhat behind; needs more attention

Staging server migration

  • no updates

Adding Drupal/Mailman sync to roadmap

  • PO found Drupal modules that should allow integration with Mailman; haven't doneany research yet
  • silfreed will look into the integration and we'll update the roadmap if it's more than 1 day's work

Next meeting June 24th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-06-10

Minutes from mozdev sysadmin meeting on Tuesday, June 10, 2008

Participants: davidwboswell (David Boswell), cdn-work (Chris Neale), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill, JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

  • added a new mirror for zurich/switzerland
  • documented some module settings that caused problems for a PO
  • setup an ad system so we can track stats for sponsor buttons
  • updated the /source.html vcs log page to support multiple VCSes
  • also updated the template /source.html to refer to more VCS types (commmented out)
  • updated hgweb to run from mod_python (3x faster, slightly more memory usage)
  • created patch for Apache to get hg setup in production
  • working on a new bugzilla entry format to help making requests to mozdev easier
  • try to finish up the bugzilla form today and that'll be it before I'm off for vacation this week

Discussed sysadmin priorities

  • log rotating is updated, but transfers to stats wasn't working properly so top 50 page is still a little behind
  • preparing for an OS update to FreeBSD 7; release on the servers is EOL
  • tested out eaccelerator again to prepare for ff3 release; still doesn't work with Mozdev code
  • mysql backup was tweaked to keep 3 generations instead of one

Web stats/Top 50 page

  • currently a little behind, should be catching up

Staging server migration

  • no updates

data retention policy

  • privacy policy is fairly clear that voluntarily submitted info (ie, to mailing list or newsgroup) is public
  • users don't understand/read that and just want their info removed
  • current situtation revolves around a person who wants their email to a listremoved w/ no personally identifiable information
  • previous instances where we told a user we would not remove an email resulted in angry responses
  • in the future we'll try to educate the user about publically posting to lists and move forward if the user is persistent (while trying to avoid censorship by removing the least amount of data possible)

mercurial rollout timeline

  • gjm will try to get things setup in production this week
  • silfreed will look over things and test out next week when he's back

ssl certs

  • the list of domains in bug#19110 should be good to start
  • GoDaddy requires that any changes to the cert be a new certificate
  • gjm and davidwboswell will work out details for getting a new cert

Next meeting June 17th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-06-03

Participants: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: JesperHansen, djc

Discussion was held publically in #mozdev

Discussed developer priorities

  • OSCon/Firefox Summit planning
  • Styling hgweb (still needs some work)
  • Sponsor work - Songbird page is now live
  • new Multi-VCS log RSS feed
  • lots of unit test improvements
  • updated feedback page to prefer filing bugs for change requests from project owners
  • finished setting up ad system for serving sponsor banners to track statistics
  • working on updating the /source.html page and testing mod_wsgi

Discussed sysadmin priorities

  • additional bot blocking
  • mail stopped over the weekend; problem is identified in amavis but still needs worked on
  • some bugs processing
  • gjm & silfreed are working on ideas for trapping bots

Web stats/Top 50 page

  • top 50 should be updating soon
  • old logs were inadvertently removed and will need reprocessed

Staging server migration

  • David thinks Mozilla might be busy until after firefox 3 is out
  • OSUOSL responded that they're still not ready

per-project web stats

  • interest from POs?
  • davidwboswell thinks the data would be very useful to POs
  • will require a lot of time to get through the backlog; not sure if we have enough CPU

Next meeting June 10th, 2008 @ 15:00 UTC in #mozdev

Mozdev status update W21-2008, sysadmin meeting minutes for 2008-05-27

Participating: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • improved bugzilla Mac OS X detection by adding more versions to OS list
  • worked with ccaygill of Mycroft to get HTTP header problem fixed that prevented downloads in IE7
  • started styling hg web interface; need some help from Mercurial devs to fix some problems w/ themeing
  • some more sponsor work
  • integrating cvs database logging and hg/svn database logging in various tools
  • updated multi-VCS-capable RSS feed should go live today (ex: http://www.vebzom.org/feed/vcslog/)
  • OSCon/Summit planning/prep
  • found some bugs in both downloads.MD.o and the update.rdf generation that the Brief project ran into; helped them work-around the problem for now and put them on the back-burner
  • agenda for this week includes styling the hg web interface and figuring out what can be done about the /source.html for projects that begin using hg
  • we should start figuring out how to coordinate the deploy of hg to production; there's going to be a number of steps so it would be good to go over them in advance and work out any problems before we try it
  • we should look more into mod_wsgi since the port should be available now; it'll save a lot of CPU compared to running as CGI

Discussed sysadmin priorities

  • some bots were active on the site last week; configuration was changed several times to try to prevent them from affecting the site again
  • some planning on a better shared-hosting setup
  • helping a project troubleshoot their mailing list config

Web stats/Top 50 page

  • no stats updating this week; other sysadmin projects are blocking getting it fixed

Staging server migration

  • no updates

handling low-level sysadmin tasks

  • things like stats processing, cvs cleanup, are time-critical but low-level tasks
  • in general, we should figure out how to handle these types of tasks so they don't eat up our sysadmin's time

open feedback requests - cvs management

  • gjm will get the server setup so we can have a "cvs admin" and setup silfreed w/ the privs

handling feedback mails

  • there's still an open bug for integrating feedback list and bugzilla
  • silfreed will join the feedback list to do manual bug filing for sysadmin tasks for now
  • update the feedback page to recommend filing bugs for common PO tasks (cvs cleanup, mailing list fixes, etc)

Next meeting June 3rd, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-05-20

Minutes from mozdev sysadmin meeting on Tuesday, May 20, 2008

Present: cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • sponsor work
  • fixed bugs in update.rdf generation
  • documenting move of hg to production
  • testing multiple-domain ssl certs - seem to work fine (multiple browsers, hg, svn)
  • updated test hgweb to require SSL pushes and added a robots.txt file
  • testing new cvs web viewer using viewvc which will also work w/ svn
  • plan to spend time this week working on mercurial to get it ready for production

Discussed sysadmin priorities

  • ssh updates on stats machine
  • used the ssh update opportunity to bring software on stats up to date

Web stats/Top 50 page

  • stats started processing this month's logs last night
  • top 50 page is back

Staging server migration

  • no updates

ccaygill: header Vary: Host issue

  • bug#19130
  • gjm is behind on bugs; we can implement this soon if necessary, though
  • Doug wants to check if this bug affects more than mycroft; will get back to this shortly

Next meeting May 27th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-05-13

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • setup mercurial commit logging last week
  • upgraded (staging) mercurial to 1.0
  • not much other mercurial work accomplished
  • updated external links in the mozdev.org header
  • released the secure update tool; fixed several minor bugs and updated documentation to make it a little easier to understand
  • updated the mozdev.org policy to allow us to fix web errors (and fixed some minor errors for projects I had notified several weeks ago)
  • working on updating application list so it can be more easily modified for sponsors
  • will be setting up an ad system to track displays, views, clicks, etc for sponsors
  • hope to finish up sponsor work by Wednesday, then work on testing SSL certs for Mercurial (and SVN)

Discussed sysadmin priorities

  • cleaned up disk space sunday night
  • log files have been moved to stats.mozdev.org for archival
  • nagios notifications were disabled for drive stats; gjm will check into that

Web stats/Top 50 page

  • still processing

Staging server migration

  • mozilla said things are still moving for getting a VM there

Next meeting May 20th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-05-06

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • hg testing released; two projects have asked for testing repos already
  • secure update.rdf generation released; there's been some additional interest on the list, but no direct questions
  • sent policy update to PO list; no objections so far
  • setup admin interface to select active VCS and initialize hg repository
  • left for hg: logging of commits to db, writing documentation, figuring out how to deal w/ source.html, figuring out what to do w/ /source/browse

Discussed sysadmin priorities

  • connection limitations are in place for CGIs (bugzilla, cvsweb, mailman)
  • bots/spiders have been hard on our server recently so this was very helpful
  • several software packages were updated on both servers

Web stats/Top 50 page

  • still ongoing

Staging server migration

  • no updates

Additional SSL Certs

  • hg.mozdev.org will be needed
  • look into bugzilla.mozdev.org?
  • in the future we'll need svn.mozdev.org
  • wildcard cert? We're not sure about the techincal details w/ that yet
  • probably cheaper to keep buying individual certs for the near future (4x $30/yr < $200/yr wildcard)

Next meeting May 13th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-04-29

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)
Community present: ccaygill, djc, JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

  • worked on the backend for choosing VCS and initializing mercurial
  • didn't get much time to work on the admin interface; working on that now
  • some more cleanup from the spam a week ago
  • research into what the proper mime type for .rdf files should be (application/rdf+xml)
  • fixed password reset/change forms (missed from php5 upgrade)
  • been short on time the past couple weeks due to being down a car and being driver for the family
  • Doug will look into any recent errors that should be fixed

Discussed sysadmin priorities

  • mostly trying to find spam sources and potential spam sources on our server
  • some minor non-user-visible config changes
  • doesn't look like there is any spam being generated from hosted php scripts

drupal admin messages going to webmaster@

  • spam notifications were fixed to go to the site email (which is set to the project owner) rather than user 1 (which is always webmaster@)

Web stats/Top 50 page

  • top 50 page is still blank; web stats are missing for april
  • processing choked on new log file format; restarted this morning

email delay

  • email was blocked from Saturday to this morning
  • virus scanner died

Staging server migration

  • no updates

Minor updates for unmaintained projects

  • still waiting for Doug to write a policy and post to the PO list; will try to address this week

pay drupal developer to write wiki page on configuring discussed defaults

  • the documentation would help with existing projects and new installs until we have the time to update the default install profile
  • more discussion needed

Next meeting May 6th, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-04-22

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner), djc (autocopy extension owner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • spam attack last week has been cleaned up; old notes system is currently disabled
  • mercurial is setup on staging at hg.vebzom.org; authentication is currently working against tigris database using existing CVS perms
  • currently working on getting admin tools setup to select VCS and create repositories
  • once admin tools are in place (hopefully this week) we can have devs test this on staging
  • no response from community for secure updates testing; just going to release it as "beta" and deal with bugs later

Discussed sysadmin priorities

  • spam attack ate up all the time last week
  • logging of outgoing messages was broken since PHP 5 upgrade; fixed and improved
  • gjm is monitoring abuse@mozdev now and replied to our upstreams about our recent spam attack
  • some security patches were applied to servers
  • copied production apache/nginx config to staging for Doug

Web stats/Top 50 page

  • log analysis has been restarted (about a month behind)

Staging server migration

  • no updates

Firefox updates server load handling

  • handled firefox 2.0.0.14 release well

Cooperation between AMO and Mozdev.org on abandoned project adoption policy

ericjung would like to bring passwordmaker.org's mediawiki install back to mozdev.org

  • should be possible; we'll need to work on the configuration a little bit

announce Mozdev.org sysadmin meetings?

  • Doug will post the next meeting time in his minutes
  • we should try to get the agenda for the upcoming week in a wiki for people to edit/comment

project status update form is being spammed

  • gjm will post to sysadmin list to ensure it's safe to be disabled
  • Doug will look into removing the form and links to it

Next meeting April 29th, 2008 @ 15:00 UTC

Mozdev sysadmin meeting minutes for 2008-04-15

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • got mercurial plan together; started getting mercurial setup on vebzom
  • deployed update.rdf generation, but need some testers
  • various bug triaging
  • we have some mod_rewrite bugs still lingering; trying to get my test suite fixed to track them down
  • otherwise working on mercurial and update.rdf testing

Discussed sysadmin priorities

  • still need to work on log rotation scripts
  • fixing mod_rewrite bugs as they come up
  • apache2 is serving http and https

Review roadmap changes

  • MXR tool (replaces LXR) support, CVS, SVN, and hg
  • updated roadmap will be announced to POs later today

Staging server migration

  • no updates

Firefox updates server load handling

  • no updates

Mozdev sysadmin meeting minutes for 2008-04-08

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • testing apache 2.2/php 5 changes
  • created project tagging policy/docs
  • started update.rdf generation for secure updates
  • started setting up test suite for mozdev.org code
  • this week is planned to be: finish up the hg setup plan (in progress), start working on hg, and try to get some sysadmin time to close some bugs (apache 2.2 rewrite bugs, web-visible cache directory setup)
  • asked about working on mercurial before svn; davidwboswell says an update to the roadmap is coming soon

Discussed sysadmin priorities

  • apache 2.2 and php 5 upgrade is complete; working on ironing out some bugs
  • https is being setup and tested

Firefox updates server load handling

  • no updates

Staging server migration

  • no news on VMs
  • server move should be highest priority now that apache/php setup is done

Other projects

  • if staging server setup drags on too long, sysadmin might move on to cvs perms with pam auth
  • project creation automation would be a good item to work on as well
  • openid would be a nice authentication mechanism for users; integrating this with other auth mechanisms isn't understood well right now

Code testing

  • mostly just unit tests now that PHP 5 is avaialble (PHP 4 couldn't do mock objects which was very limiting)
  • Doug will be writing some tests to verify certain web paths are working correctly
  • not really focused on full integration testing or continuous integration right now (the entire web stack isn't in version control, so we can't know when changes are made)

Mozdev sysadmin meeting minutes for 2008-04-01

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • project overview page/secure installs was generally well-received
  • several minor bugs in file management, presentation, and extension parsing were handled quickly
  • working on a tagging policy to help project owners know what to expect to be approved: http://www.mozdev.org/drupal/wiki/MozdevProjectTagging
  • improved download counters offered by mozdev
  • setup web-visible cache directory on the staging server w/ info for migration to production
  • work on update.rdf generation is going well; script to generate files is mostly complete; need to add 'update info' url to file management and test the resulting update.rdf files; this will require the web-visible cache directory setup to finish
  • should be starting on subversion setup plan this week and filing bugs using email I sent earlier to sysadmin@ as a base
  • visiting family this weekend; offline Friday

Discussed sysadmin priorities

  • nginx has been in production for about a week and has been handling load from updates fine
  • web stats are currently broken (since the 25th) due to nginx setup and apache2/php5 update
  • apache2/php5 is being setup in production today
  • backups have not been occuring (reason isn't known yet); gjm is working w/ osuosl to get them working again

Firefox updates server load handling

  • update was handled well; there were several spikes in bandwidth/requests but they were handled by nginx
  • silfreed/gjm will continue to work on improving nginx setup by serving more files, but the solution worked incredibly well
  • some projects get lots of 404 hits for their update.rdf requests; gjm will make a list of projects and silfreed will contact POs to make sure they're aware their users don't have an upgrade path

Staging server migration

  • mozilla has said "yes" to 2U rack space and probably a VM
  • osuosl is having disk capacity problems so a VM isn't immediately available
  • first plans for for VM (hopefully from OSUOSL) is moving download master
  • separating email and web traffic would probably be the next step
  • figuring out how to split projects up across servers would also be useful from a security viewpoint (split mozdev.org projects from hosted projects)
  • we'll still need to ask TWS to ship our server when we decide where its going (Mozilla or retirement)

Mozdev sysadmin meeting minutes for 2008-03-25

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner), ccaygill (MyCroft project)

Discussion was held publically in #mozdev

Discussed developer priorities

  • about ready to release the secure installation/project overview stuff; been working w/ a PO on testing things; trying to figure out a bug in the file release part, but otherwise things seem good
  • plan to get the secure installation announced and in production this week
  • did some testing of the lightweight web server last week and couldn't find any problems with the setup on vebzom
  • fixed a bug in Drupal authentication module when editing mozdev cvs users
  • changed the season starts for mozdev's themes to the equinoxes/solstices
  • secure updates shouldn't take too long - it's mostly backend stuff that won't really have a UI anyway; hopefully only a week left

Database policy (bug#15661)

  • we need to separate our policy decision from our technical limitations
  • Doug will file a separate bug about needing to be able to monitor what our project usage is

Discussed sysadmin priorities

  • preparations for apache2/php5 are underway
  • setup php eaccelerator at the end of last week
  • looking at spam filtering to reduce amount of cpu time used by increasing number of firewalled hosts

Firefox updates server load handling

  • 2.0.0.13 release is scheduled for today
  • still need to reduce amount of pages that get handled by apache/php

Staging server migration

  • moco might be able to provide a VM; hopefully hear back later this week
  • no news about VMs from OSUOSL yet