Reflecting on my first year at Mozdev

I've been at Mozdev for a full year now. This was a big step for me personally as well as for Mozdev; there was lots of trepidation from both parties, for a number of reasons:

  • I was Mozdev's first full-time employee (and without a formal office, all interviews were over the phone)
  • This was my first time being a full-time contractor instead of salaried
  • There were (and still are) lots of bugs; some old, some important, and they all needed reevaluated to better define their scope and meaning
  • There was a large, pre-existing open-source community that I had no experience with. I didn't know how (or if) they would accept me or my work
  • The features that were needed at Mozdev were rather large; mostly due to scaling them across many different projects

While the first few months were relatively scary for me while I learned the existing code base and environment (7 years of organic growth!), I was able to begin working on new features almost immediately. Most of what I worked on is on our roadmap, but I also spent a lot of time trying to setup new structures for maintaining code.

One of my biggest challenges this year wasn't developing features for Mozdev, though; it was integrating myself with the existing community. The community of developers and users at Mozdev is amazing and has an incredible breadth of knowledge and background, and I was an outsider coming in to change how they worked on their projects. I've tried hard to make sure changes and prototypes are announced ahead of time and get exposure as well as making myself available to Mozdev users who need support in order to help people get to know me a little better. Being on the front-line for support as well as new features is a new experience for me, and while it's had it's rough moments I think it helps people feel that we're listening to them and that we're here to help.

This past year I've mostly kept my head down trying to learn about the Mozdev community; this year I'd like to get my head up a little more and reach out to the larger Mozilla community so that we can all benefit. I'm sure the upcoming OSCON Conference and Firefox Summit will go a long way toward getting people together so we can brainstorm on how to improve Mozdev and make it a bigger participant in the Mozilla community.

It's been a great year, and I'm very glad that I took the chance on jumping into an environment I knew nothing about and that Mozdev chose to take a chance on me. I'm very thankful to be able to continue to help the Mozdev and Mozilla communities to grow and look forward to the next year and what it will bring.

Mozdev sysadmin meeting minutes for 2008-07-01

Participants: davidwboswell, (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

  • working on WYSIWYG for Mozdev wiki
  • planning on using mediawiki syntax with FCKeditor (noticed DevMo is switching to DekiWiki)
  • at ccaygill's prompting (and patch) updated the local.conf documentation
  • performed some log analysis of Firefox 3 release traffic
  • discussed alternatives to awstats for performing custom stats analysis (no solution found)
  • hg was just setup in production; needs tested and then we can open up to projectowners for testing
  • once hg is in testing w/ POs I'll be back at working on the wiki and also with new file upload mechanisms

Discussed sysadmin priorities

  • some security updates
  • mailman upgraded to latest version
  • hg setup in production; still needs SSL cert

Web stats/Top 50 page

  • stats have caught up
  • hits for 2008-06-(17-20) are impressive (FF3 release)

Staging server migration

  • mozilla has recently been looking into racking a ton of Mac Minis and OSUOSL just received a shipment of hard drives
  • david will ping mozilla again and gjm will get back to OSUOSL

Next meeting July 8th, 2008 @ 15:00 UTC in #mozdev

Mozdev status update for W26-2008

Semi-busy week with not much to show for it:

The big thing for this week will be trying to get FCKeditor working with the mediawiki syntax; Independence Day is this Friday for the USA so I'll be offline then.

Mozdev sysadmin meeting minutes for 2008-06-24

Participants: gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill

Discussion was held publically in #mozdev

Discussed developer priorities

  • finished new bugzilla bug entry form
  • helped with Firefox 3 server load (adding additional caches, mostly)
  • fixed XPI sorting on project overview page
  • working on improving wiki formatting at Mozdev
  • investigating integrating Mailman and Drupal Forums
  • updated VCS log to include link to a file's history

Discussed sysadmin priorities

  • no updates, under the weather
  • tanker's out on leave right now; when he get's back they'll work on the OS update on vebzom and then rework the plan for production

Web stats/Top 50 page

  • ccaygill has been working on mycroft to reduce hits to old plugins that won't have updates
  • munin outages/gaps are related to stats maching blocking on something the stats machine despite it being idle; needs more investigations
  • gjm has begun looking at firewall connection table to catch bots and evil users rather than just relying on apache logs

Staging server migration

  • no updates

Next meeting July 1st, 2008 @ 15:00 UTC in #mozdev

Mozdev status update for W25-2008

Getting back into the swing of things after vacation.

  • Updating Mozdev bugzilla bug entry form - this is ready to go into production to help making requests to Mozdev easier
  • Handling Firefox 3 release server load - the Firefox 3 release caused lots of additional web traffic to in a number of ways; people trying to get news about Firefox 3 (some projects host translations of Mozilla software) and additional update requests for extensions and search plugins caused by lots of people restarting their browsers in a short period of time. I added caching to a number of pages during this time to try to reduce the impact on the server.
  • Fixed XPI file sorting on Project Overview page
  • Trying to improve wiki formatting at Mozdev - I'm evaluating both using straight HTML for the wiki as well as FCKeditor w/ a mediawiki plugin. I'm leaning towards the latter right now, but the various plugins need some work.
  • Investigating linking Mailman and Forums - Currently the plugins require a separate email account to check for new messages and I'm not sure who should be responsible for setting up/maintaining this account (Mozdev or the project owner).
  • Updated VCS log to include link to a file's log history

This week I plan to continue working on improving Mozdev's wiki editing capability and look towards how pam_mysql can be configured to allow file uploads at Mozdev.

6 months of photos

Our recent trip to Lake Raystown:





Use GUI HTML editors for editing wiki content

We're looking to improve the experience of editing pages in our Wiki, and I'm wondering if we should just ditch "wiki" syntax and just support HTML and use a nice in-browser HTML editor.

It should be easy for us to continue to support CamelCase and square brackets to link to other wiki pages, so what else would be missing (other than an annoying wiki syntax)? Does anyone have any experience on going this route?

Mozdev sysadmin meeting minutes for 2008-06-17

Participants: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill

Discussion was held publically in #mozdev

Discussed developer priorities

  • finished up changes to simpler bug entry form
  • increased timeout on new Mozdev pages that require login (file managment, project tag editing)
  • wrote Mercurial documentation for Mozdev
  • Firefox 3 release is today

Discussed sysadmin priorities

  • mailman upgrade is in the works; all our custom changes have been included in the new version so we won't be running a hacked version anymore
  • looking into using snort for detecting bot attacks on the server; not sure if itwill be feasible or not
  • ssl cert w/ multiple domains should be good to install

Web stats/Top 50 page

  • still somewhat behind; needs more attention

Staging server migration

  • no updates

Adding Drupal/Mailman sync to roadmap

  • PO found Drupal modules that should allow integration with Mailman; haven't doneany research yet
  • silfreed will look into the integration and we'll update the roadmap if it's more than 1 day's work

Next meeting June 24th, 2008 @ 15:00 UTC in #mozdev

Mozdev status update for W24-2008

Short week last week due to vacation.

We're expecting Mercurial to be released soon at Mozdev; we'll be sure to post an update once it's released.

Mozdev sysadmin meeting minutes for 2008-06-10

Minutes from mozdev sysadmin meeting on Tuesday, June 10, 2008

Participants: davidwboswell (David Boswell), cdn-work (Chris Neale), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: ccaygill, JesperHansen

Discussion was held publically in #mozdev

Discussed developer priorities

  • added a new mirror for zurich/switzerland
  • documented some module settings that caused problems for a PO
  • setup an ad system so we can track stats for sponsor buttons
  • updated the /source.html vcs log page to support multiple VCSes
  • also updated the template /source.html to refer to more VCS types (commmented out)
  • updated hgweb to run from mod_python (3x faster, slightly more memory usage)
  • created patch for Apache to get hg setup in production
  • working on a new bugzilla entry format to help making requests to mozdev easier
  • try to finish up the bugzilla form today and that'll be it before I'm off for vacation this week

Discussed sysadmin priorities

  • log rotating is updated, but transfers to stats wasn't working properly so top 50 page is still a little behind
  • preparing for an OS update to FreeBSD 7; release on the servers is EOL
  • tested out eaccelerator again to prepare for ff3 release; still doesn't work with Mozdev code
  • mysql backup was tweaked to keep 3 generations instead of one

Web stats/Top 50 page

  • currently a little behind, should be catching up

Staging server migration

  • no updates

data retention policy

  • privacy policy is fairly clear that voluntarily submitted info (ie, to mailing list or newsgroup) is public
  • users don't understand/read that and just want their info removed
  • current situtation revolves around a person who wants their email to a listremoved w/ no personally identifiable information
  • previous instances where we told a user we would not remove an email resulted in angry responses
  • in the future we'll try to educate the user about publically posting to lists and move forward if the user is persistent (while trying to avoid censorship by removing the least amount of data possible)

mercurial rollout timeline

  • gjm will try to get things setup in production this week
  • silfreed will look over things and test out next week when he's back

ssl certs

  • the list of domains in bug#19110 should be good to start
  • GoDaddy requires that any changes to the cert be a new certificate
  • gjm and davidwboswell will work out details for getting a new cert

Next meeting June 17th, 2008 @ 15:00 UTC in #mozdev

Mozdev status update for W23-2008

I'm heading to Raystown Lake (PA) tomorrow, so I don't plan to get too much accomplished this week (some more work on the bugzilla guided entry page, most likely).

Mozdev sysadmin meeting minutes for 2008-06-03

Participants: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)
Community participants: JesperHansen, djc

Discussion was held publically in #mozdev

Discussed developer priorities

  • OSCon/Firefox Summit planning
  • Styling hgweb (still needs some work)
  • Sponsor work - Songbird page is now live
  • new Multi-VCS log RSS feed
  • lots of unit test improvements
  • updated feedback page to prefer filing bugs for change requests from project owners
  • finished setting up ad system for serving sponsor banners to track statistics
  • working on updating the /source.html page and testing mod_wsgi

Discussed sysadmin priorities

  • additional bot blocking
  • mail stopped over the weekend; problem is identified in amavis but still needs worked on
  • some bugs processing
  • gjm & silfreed are working on ideas for trapping bots

Web stats/Top 50 page

  • top 50 should be updating soon
  • old logs were inadvertently removed and will need reprocessed

Staging server migration

  • David thinks Mozilla might be busy until after firefox 3 is out
  • OSUOSL responded that they're still not ready

per-project web stats

  • interest from POs?
  • davidwboswell thinks the data would be very useful to POs
  • will require a lot of time to get through the backlog; not sure if we have enough CPU

Next meeting June 10th, 2008 @ 15:00 UTC in #mozdev

Mozdev status update for W22-2008

  • OSCon/Firefox Summit planning - got my flights booked and applied for my passport
  • Styling hgweb - I'm trying to get our hgweb instance styled better but am running into problems modifying Mercurial themes (maps)
  • Sponsor work
  • New VCS log RSS feed - This one now works with CVS, Mercurial, and eventually SVN
  • Lots of unit test improvements for various parts of the Mozdev templating system
  • Cleaned up feedback page to recommend project owners file bugs for change requests so we can track problems better

I didn't get nearly as much accomplished last week w.r.t. Mercurial, but we're still making good progress. I need to update the VCS logs that are generated on the /source.html pages and create a new source.html template that includes instructions for mercurial. I also need to write a bunch of docs, but have a fairly good template setup for what needs done.

Mozdev status update W21-2008, sysadmin meeting minutes for 2008-05-27

Participating: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • improved bugzilla Mac OS X detection by adding more versions to OS list
  • worked with ccaygill of Mycroft to get HTTP header problem fixed that prevented downloads in IE7
  • started styling hg web interface; need some help from Mercurial devs to fix some problems w/ themeing
  • some more sponsor work
  • integrating cvs database logging and hg/svn database logging in various tools
  • updated multi-VCS-capable RSS feed should go live today (ex:
  • OSCon/Summit planning/prep
  • found some bugs in both downloads.MD.o and the update.rdf generation that the Brief project ran into; helped them work-around the problem for now and put them on the back-burner
  • agenda for this week includes styling the hg web interface and figuring out what can be done about the /source.html for projects that begin using hg
  • we should start figuring out how to coordinate the deploy of hg to production; there's going to be a number of steps so it would be good to go over them in advance and work out any problems before we try it
  • we should look more into mod_wsgi since the port should be available now; it'll save a lot of CPU compared to running as CGI

Discussed sysadmin priorities

  • some bots were active on the site last week; configuration was changed several times to try to prevent them from affecting the site again
  • some planning on a better shared-hosting setup
  • helping a project troubleshoot their mailing list config

Web stats/Top 50 page

  • no stats updating this week; other sysadmin projects are blocking getting it fixed

Staging server migration

  • no updates

handling low-level sysadmin tasks

  • things like stats processing, cvs cleanup, are time-critical but low-level tasks
  • in general, we should figure out how to handle these types of tasks so they don't eat up our sysadmin's time

open feedback requests - cvs management

  • gjm will get the server setup so we can have a "cvs admin" and setup silfreed w/ the privs

handling feedback mails

  • there's still an open bug for integrating feedback list and bugzilla
  • silfreed will join the feedback list to do manual bug filing for sysadmin tasks for now
  • update the feedback page to recommend filing bugs for common PO tasks (cvs cleanup, mailing list fixes, etc)

Next meeting June 3rd, 2008 @ 15:00 UTC in #mozdev

Mozdev sysadmin meeting minutes for 2008-05-20

Minutes from mozdev sysadmin meeting on Tuesday, May 20, 2008

Present: cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • sponsor work
  • fixed bugs in update.rdf generation
  • documenting move of hg to production
  • testing multiple-domain ssl certs - seem to work fine (multiple browsers, hg, svn)
  • updated test hgweb to require SSL pushes and added a robots.txt file
  • testing new cvs web viewer using viewvc which will also work w/ svn
  • plan to spend time this week working on mercurial to get it ready for production

Discussed sysadmin priorities

  • ssh updates on stats machine
  • used the ssh update opportunity to bring software on stats up to date

Web stats/Top 50 page

  • stats started processing this month's logs last night
  • top 50 page is back

Staging server migration

  • no updates

ccaygill: header Vary: Host issue

  • bug#19130
  • gjm is behind on bugs; we can implement this soon if necessary, though
  • Doug wants to check if this bug affects more than mycroft; will get back to this shortly

Next meeting May 27th, 2008 @ 15:00 UTC in #mozdev