Doug Warner

• Fixed Drupal theme selection bug - Users were unable to change their theme due to me forcing the setting in settings.php globally.
• Planning fixes to mysql errors caused by backups - I've been working with Mozdev's sysadmins to try to figure out how best to deal with the two-minute outage daily due to our mysql backups. Currently we believe most of the outage is due to our large bugzilla database and are addressing that.
• Checking new apache rewrite rules - We have a new sysadmin that is making good strides at cleaning up some of our cruft that has accumulated over the years; I've been going over changes made and making suggestions.
• Caching download file information - In order to improve functionality on D.MD.o I've added caching of file size, last modified, and md5 and sha1 hashes.
• Worked on theme for Thanksgiving
• Updated RSS feeds to include atom links - http://validator.w3.org/feed/ now suggests that your RSS 2.0 feeds include an atom:link element, so that has been added to our project and CVS RSS feeds.
• Repaired our master rsync mirror content - Our master rsync mirror's content was accidentally deleted and therefore no new downloads had been pushed out to our mirrors over Thursday/Friday. Additionally, the most recent scripts that updated the content on our master server were also deleted and not checked into CVS, so those had to be fixed up a bit. Content was restored by Friday afternoon, in time for our master rsync server to go down over the weekend. All access appears to be restored as of Monday afternoon and mirrors appear to be updating.

Another big week of mirror work. I received responses from all the current mozdev.org mirrors as well as some that I wasn't aware was acting as a mirror. I have most of the mirrors assigned to regions so redirection to the closest mirror should be happening in the near future (once I work out a couple little quirks).

• Updated mirrors.mozdev.org content - Our mirrors.mozdev.org content was a little stale but is now being generated dynamically from our list of available mirrors. There's even a tool for you to enter your IP address and see what your preferred mirror set is.
• Download counters - I've setup tracking of downloads so that we could re-enable the downloads() function for our project sites so they can see how many downloads they're receiving for particular files. The Top 50 page has even been updated to show downloads for the last month.
• Drupal setup for several projects - Several projects have requested to have Drupal setup so that they can evaluate its features. The projects include stalemate, Mycroft, and Brief.

I would like to get the last pieces of the download redirection completed.

I will also be working on a secure installation method and would appreciate any comments people have about my proposal.

Happening simultaneously might be the Bugzilla 3.0 upgrade. We'll be announcing on the project-owners list when this is planned to occur and will be asking people to check out our test install to make sure we haven't overlooked anything.

• Finished up mirror management - I finished up the admin interface for managing mirrors. This included contacting all existing mirrors and verifying information (admin email, URLs, regions served). I'll still be spending some time adding additional information to the tool (such as availability reporting) and improving the regions served by each mirror, but the overall tool is working now.
• Redirection to closest mirror - At the end of the week I finally made some good headway on redirecting a user to their closest mirror. I'm not quite ready to release this into production yet as I don't have good coverage of all regions across the globe yet, but this should be coming soon.

This coming week should bring some much more user-visible features such as the download redirection mentioned about and an update to mirrors.mozdev.org content.

• Drupal rolled out to production - This resolves bug#7020, bug#10166, and bug#17020.
  There has been some community discussion about the /drupal (as in http://www.mozdev.org/drupal/wiki) stuck in the middle of the path for the blog, wiki, and forum. Right now that is an unavoidable side-effect of using one CMS for providing all the features. I offered that if a suitable alternative this could be easily renamed to something else, but nothing has been offered yet.
• Mirror management library and interface - I'm still working on a mirror management setup for Mozdev, but was mostly held up last week learning some of the new jQuery UI features. So far I'm very happy with my progress and believe the new interface will be a big help for Mozdev admins.
  Of course, the main reason we need a new management interface is to provide new features for downloads using our mirror system. We're hoping to add download counters, closest-mirror redirection for the user (with better support for recently-uploaded files), and possibly most importantly, secure installations of extensions.
• Brainstormed and researched ideas about our mail system delays - We're tracking the ideas in bug#17280 and bug#17667, but the summary is that right now we think that the 110k+ messages being held by Mailman are generating a lot of daily reminder messages. I haven't been able to confirm this, so if others with experience in mass Mailman setups can comment on whether that's possible, I'd love to hear it.

• RSS icon cleanup on front page
• Vebzom.org DNS was broken for awhile - This kept me from working on the new mirror management library
• Researched email delays - The problem might be related to the massive number of held messages that are sitting in Mailman; it might be the daily nag emails for admins to check the messages.
• More Bugzilla 3.0 theming (bug#15883)
• Updated downloads.mozdev.org so it won't list projects with no downloads or only a README.downloads
• Began using Drupal in production
• Community discussion over how Drupal should be linked
• Upgrade Drupal to 5.3
• Tweaked the mozdev drupal theme to make it less-cramped
• Added modules to improve Drupal's functionality - Pathauto, Token, Recent Changes, Spam
• Updated permissions so any logged-in user can edit Mozdev's wiki
• Updated the mozdev install profile to take into account new modules, permissions, and new install_wizard_api format

Next week I plan on finishing up the mirror management library and working on the admin frontend, working on the improved download redirection and download counter, continuing to utilize Drupal on mozdev.org and working with the community on the best application for their use.

The big events for the week are:

• New release of downloads.mozdev.org - The new version allows a browseable directory tree inside mozdev.org's site instead of redirecting to a mirror immediately. Our mirror network is still utilized for downloads, however.
• Drupal moved to production - Our Drupal installation is now in production. We'll need some time to get things setup there and add documentation, but this is one step closer to providing blogs, wikis, and forums for all our projects.

Other tasks for the week include:

• Updated bugzilla 3.0 templates for mozdev
• Fixed rewrite rules breaking webcvs occasionally
• Worked on various mirror bugs - Including bug#7137 (getCVSRev and Download counter) and bug#2807 (Use visitor's location to determine target download server)
• New mirror check script - I wrote a new mirror check script to verify our mirrors are available and in sync and serving the files correctly. It has found a problem already (which has been corrected).
• Ideas for improving downloads - I'd like to make the master version of our downloads available to our projects so that propagation delays in our mirror network don't slow the release of new software.

This week's tasks include writing documentation for Drupal, starting to get Drupal in use for www.mozdev.org, and continued work on improving the mirror system.

Most of this week was spent working on mirror improvements and preparing for the Drupal rollout.

• Determined Bouncer won't work for Mozdev - Bouncer's reliance on Product, Version, Language, and OS makes it not quite suitable for Mozdev since we don't make our project owners track language or OS for their downloads. We'd also prefer not to switch our download paths around at this point, so we can't rely on hacks to make it work.
• bug#8279 - Pull mirrors out of rotation automatically - re-worked the script that watches our mirrors to check whether the mirror is stale and if the content types are being served correctly. This is just waiting to go into production.
• bug#5723 - Mask mirrors with downloads.mozdev.org - I began work on hiding our mirrors behind an actual browseable downloads.mozdev.org. I have the script working and redirecting once a file is reached, but I need to do some work on sorting before it can be ready for release.
• bug#3000 - Disallow dot-files on mirror sites - I updated our master mirror to filter out some files.
• CVS commit log RSS feed changes - I enabled the RSS feed for all projects and added new options to just disable the feed or the link to the feed. I'll be looking into allowing filtering of the feed's contents in case a project owner wants to provide an RSS feed just for downloads and an RSS feed just for source changes, for example.

Next week I plan to continue on improvements to our mirror system.

Drupal is ready to be deployed, so that's in a holding pattern waiting for the sysadmins to get time to set things up.

Since Drupal's currently stalled, I've moved onto other projects. Lots of cleanup of various tools that have been implemented recently and started evaluating tools to improve our mirror setup.

• bug#17780 to create a CVS Log RSS feed was implemented - Each project's CVS activity can now be tracked through an RSS feed.
• bug#11393 - renable Top 50 page was released - The Top 50 projects page was re-released, complete with movement in the past day and an RSS feed so you can keep tabs on how your project stacks up to others on the site.
• bug#8279 - improve mirror detection - This actually involves a lot of mirror management problems we've been experiencing - we need to pull mirrors out of rotation when they go down, or when the files are stale, or when the files are being served with the incorrect mime type. We'd also like to be able to browse "downloads.mozdev.org" and only redirect to a mirror for the download (this makes it much easier to link to files). And when we have problems with a mirror, we need to know who to contact. I spent a lot of time today trying to setup Bouncer but didn't get anywhere with loading files into (although the web interface and Sentry appear to run correctly).
• Prepared instructions for Drupal rollout
• Research into how to integrate Drupal - Unfortunately it looks like the project owners will need to manually modify their project_nav.html to add the links and their head.txt to add the RSS feeds; we don't have a way to automatically do this right now. I did, however, modify the mozdevauth module to log whether Drupal was enabled for a site or not - this will give us the option to automate this at a future date.
• Helped implement upgraded to cvsweb 3.0.6

Plans for next week include:

• Evaluate Bouncer - I need to determine if Bouncer can even do what we need before I put much more time into it.
• Look into setting up Bonsai and LXR - these are tools that should be relatively easy to setup but are missing
• Research into how to automate project setup - There are a number of manual steps in setting up a project right now. Removing them would free up admin time for other tasks.

This past week's progress; somewhat less than I had hoped to finish due to Jocelyn being sick.

• Looking into using bouncer to manage our mirrors - I've only gone through and read all the documentation so far, but the tool seems like it would solve a number of mirror bugs for us (make "downloads.mozdev.org" easier to navigate and link to, remove stale mirrors, verify proper mime-types).
• Added links and info about McCoy - Extension owners will need to update their extensions if they want them to be able to be automatically updated so we wanted to provide our project owners with a tool that can help ease that burden.
• Inventorying old tools and making recommendations on removal - Mozdev has a number of old tools that can be removed (mostly for the sake of removing cruft and making things more transparent.
• Monitored/Moved along several mirror bugs - We've addded mozdev.bigsearcher.com, and chosen to remove ftp.cc.uoc.gr and mirrors.wikifusion.info because we can't contact their admins to fix a mime-type issue.
• Prepared upgrade cvsweb to 3.0.6 - I updated the config and fixed the CSS for the new version of cvsweb that will allow us to block directories (such as the protected directory).
• Fixed the admin publish tool
• Created a new top-50 projects page - Using the awstats extract our sysadmin has put together, I created a new top-50 page that shows our top-50 projects by pages, hits, or bandwidth.
• Created an RSS feed for CVS checkins per-project - An RSS feed that allows project developers (and other users) to track what's going on with a project.

This week's plans include working on the Drupal rollout plan (including how to integrate/link with current projects) and testing of Bouncer.

Drupal for Mozdev is getting close to being ready to implement in production, so I've been mostly working on other bugs this week.

• Drupal integration into Mozdev site - I started looking into how to link Drupal into Mozdev's site - specifically how to include it in the navigation. I haven't gotten too far with this yet.
• bug#17724 and bug#17733 to get some mirrors to fix their mime type problems
• Asked project owners to test out Drupal - Our drupal installation is pretty much ready to test, so I asked our project owners to check it out.
• Look into having bugzilla manage our public feedback - In order to keep track of issues reported by our users we're looking into having our feedback list forward into Bugzilla. We're looking to upgrade to Bugzilla 3 in the near future so we'll probably wait until after that. The biggest problem right now is that the email_in.pl script requires the user emailing to have an existing bz account, so Myk said I should look into adding that to the script when we're ready to work on this again and submit it as a patch.
• Disabling old tools - Mozdev has a number of old tools laying around that either aren't being used anymore or are being replaced by Drupal. I'll have a report for the board at our next meeting.
• Tool to disable guest checkouts - Some projects don't want to allow guest checkouts so we've developed a tool to help disable that if necessary (the reasons need to be good).
• Upgrade cvsweb - Upgrading cvsweb will allow us to keep the protected directory from even showing up in the list (since it's not possible to view it anyway). I have the groundwork done; so now we need to find time to implement it.
• Publish tool not working - Mozdev has a tool to publish/unpublish a project, but the publishing part wasn't working. I submitted a patch to get that going again.

Next weeks tasks will probably mostly depend on how testing goes with Drupal and our Tuesday board meeting. If things are still going slow with Drupal I might start research on setting up Subversion.

I took a slight diversion from working on Drupal this week to try to give some other tasks attention.

• Drupal Mozdev theme work - Just minor tweaks to make the theme more "mozdevy" and work better with Mozdev's rotating themes.
• Setup custom project - This is a project to hold custom code used for Drupal integration, or other projects involved with the running of Mozdev.org. Currently, things like Mozdev's authentication module, theme, and installation profile exist there.
• Drupal cron job integration with Multisite Manager - I improved my patch for Multisite Manager to make it more useful for a broader audience. You can the patches in Multisite Manager v0.9.5.
• bug#17520 - automate new project list on front page - In order to give us more time for other tasks, I automated the new project list on the front page by extending my previous work on bug#13381.
• Updated last updated block from bug#13381 - A straggling feature that I neglected to implement when I was working on the bug; the "last updated" block on Mozdev.org's front page is now built using the same list as the other project activity lists. A repercussion of work on both this bug and bug#17520 is that "new" projects and "unstarted" projects have a different definition. "New" projects are projects that have just recently become active for the first time; thus they can be both "new" and "active". "Unstarted" projects are projects that have recently been created and have no perceptible progress on them; these can be very old, in some cases.
• Started tracking Drupal rollout through bug#17703 - I let people who where tracking other separate bugs for blog, wiki, and forum features know that we're going to be rolling out Drupal soon and asked them to test things out. One user responded and was able to find some problems that were quickly and easily fixed.
• Add Drupal to new project form - I started looking into how this form was built in order to add Drupal as a feature that project owners can request when setting up a new project.

Upcoming tasks:

• Add Drupal to new project form - Of course this will have to wait until we're ready to release it to all project owners, but I'd like to understand how the form works and get it ready for release.
• Add Multisite Manager API - In order to automate setting up and removal of Drupal sites, I'd like to add an API to Multisite Manager. This will mostly be necessary for bug#11378, but it might be good to lay some groundwork.
• Additional testing and planning for rollout of Drupal

This is a short week for me, so I didn't get as much accomplished as I hoped. Still, there was some good progress made, and the Drupal install for Mozdev is progressing well. The test for project owners should be coming shortly.

• Multisite Manager/install profile fixes - Fixing Mozdev's install profile so it will choose the correct site defaults for the specified project.
• Multisite Manager cron integration - I started work on making Multisite Manager be able to run all its sub-sites cron jobs. Right now it runs the cron for all drupal sites that have been created through Multisite Manager, but I would like to ahve the option to enable or disable it globally as well as per-site.
• Registered #mozdev-admin on irc.mozdev.org - User reed stopped by #mozdev and helped me get #mozdev-admin setup correctly, so hopefully we should be good now for an admin-only chat room
• Drupal rollout and testing plans - Not much to report here other than I'm starting to make plans for how to rollout Drupal to production and get some test reports from project owners.
• Drupal performance testing - Drupal seems to be performing slowing at some points on the staging server despite me not having any problems with it on Argo. I'm starting to suspect other latency problems; perhaps with our colo host. I'm basing this solely on the fact that I have other semi-random SSH hiccups as well, so I'm trying to narrow it down.
• Looked into re-enabling the top-50 page - Now that we have stats processed again I'm looking into getting the info compiled into a more readable form (along with an RSS feed). Not much progress yet other than looking into the data a bit
• Mozdev/Drupal auth module work - I was finally able to figure out how to disable the ability to change selected roles when editing a user, so now admins or users won't be able to change whether a user is in the Project Owner or Contributor roles.
• Mozdev/Drupal theme work - I fixed the nodes and comments so the submission info was displayed properly (author and time)

So next week I'll be doing some more analysis of what the performance problems are on our staging server and getting our Drupal install to our project owners for testing. I'd also like to tackle bug#11393 and get that resolved, but we'll have to see how the week goes.

Mostly Drupal happenings this past week.

• Released staging copy to board members - The board members were able to get access to the staging copy I've been working on in order to test installing new Drupal instances, editing and creating content, and working with their user profiles synced to their Mozdev CVS accounts. Overall things went very well (from my perspective) - David ran into the most problems in his testing, but Brian and Eric also ran into some problems that I noted. Everyone ran into problems with the authentication module. David noted that the forums were intially hard to use because it wasn't clear that there wasn't a forum to post to. Several people ran into problems creating and editing wiki pages. There were also some strange permissions problems to deal with.
• Register #mozdev-admin on irc.mozilla.org - with the success of having #mozdev registered on irc.mozilla.org we wanted to make sure #mozdev-admin would be available when we needed it. Unfortunately it doesn't seem possible to make a hidden, invite-only room registered, since if all the admins leave there's no way to get back into it. So for now we're stuck creating the channel as-needed and setting mode +is and inviting the appropriate people.
• Fixed new projects showing up as active for bug#13381 - the way I determined whether a project was "new" or not changed around July 2007, so I had to make changes in the algorithm. I also found a bug regarding finding a project's recent bug activity that I fixed while I was in there. This resulted in the project totals being redistributed a bit more; mostly a few more active projects and fewer inactive projects - new projects stayed about the same.
• Drupal wiki fixes - I fixed the problems where we were having inconsistent problems editing and creating wiki pages by reordering how the freelinking module was used.
• Updated active content types and input filters for Drupal - David noted that having both the "story" and "page" content types was confusing, so I removed the "story" content type. I also disabled the PHP input filter to prevent abuse.
• Made project status RSS feeds update properly - user Mc on #mozdev reported that newsfox wasn't showing the projects being updated via the RSS feed, even though the page was changing frequently. I updated the feed so the guid and link fields are both changed based on the most recent activity so that new items show up correctly.
• Setup a default forum on new Drupal installations - in order to make it more usable and clear how to use the forums in Drupal I modified the installation profile to create a forum that can be used immediately along with the appropriate permissions for the various roles (forum moderator and project owner).
• Pinpointed errors shown during new site installation - David had a problem where there were a bunch of SQL errors during the site installation. It turns out this is due to visiting a drupal site before it's setup. I'm not sure if there's a solution for this or not.
• Improved mozdev auth module for Drupal - I fixed the problem where the user wouldn't be logged in if they were a mozdev user when the tried to login for the first time. This also fixed a bug where the user's roles wouldn't be synced from their project settings when they logged in the first time. I also made sure that if a user tries to register a username that already exists in mozdev's CVS that it will be blocked, giving them an error message to just login instead. This prevents an awkward situation where a user tries to create an account but can't login because their password will never work.
• Started researching how to run the cron jobs for all our drupal sites - I'm planning on using Multisite Manager to manage the various project drupal installations from one main site, so I'd also like to have that module run the cron jobs for the various hosts. I submitted a feature request to ask about running sub-sites' cron jobs from the master site, but I should probably be more specific that I can implement this if they give me an indication of how they would like the patch to be created.

So this week I'll be trying to get the cron jobs running properly for all sites and try to make sure the mozdev auth module is as usable as possible. I know for the auth module I would like to disable the ability to change the roles that are synced from the module (project contributor and project owner), but there may be other areas that should be enhanced as well. I also have some enhancements to make to the installation profile so that it works properly from Multisite Manager; there may need to be some patches to that module in order to work properly since I switch databases in the install profile to get information, we'll just have to see once I get into it.

A couple major activities this week:

• Completed bug#13381 - unstarted projects list.
  The new lists are active, and you can see the active, inactive, and new lists as well as the stats page. There's some minor bugs I need to look into (some new projects went straight to the active page instead of new), but things seem to be functioning pretty well.
• Drupal install profile - This will setup the default settings, roles, and permissions for a project that is using drupal. I was able to get most of the important parts working (setup default modules, roles, theme), but still have some minor things to get worked out (menu items, default site information).
• Drupal/mozdev authentication integration - this allows project owners and contributors to login to Mozdev's Drupal installation using their VCS username/password and also synchronizes their VCS status with drupal - ie, assigns the "Project Owner" or "Project Contributor" role for that project if it applies. So far the authentication and role sychronization is completed; I also disable the account forms that are pulled from Mozdev (username, email, password). There's some more minor work here to disable the ability to select certain roles (or possible ignore changes if that's not possible), but the module already seems to work and the rest seems like UI polish.

Made lots of good progress with Drupal modules and permissions setup this week.

I had some problems with the Liquid wiki module not supporting CamelCase correctly, so I had to use some other modules to implement that as well (pear wiki, freelinking, wikitools).

I've setup a default set of permissions to make it easier for project owners to give out edit privileges to various parts of the installation. In addition to project owners and contributors (matching our existing permissions for CVS access) I setup "Wiki Editor" and "Forum Moderator" to make these roles clearer.

I continued work on getting the Mozdev installation profile setup and ran into some road blocks with setting up the default "blocks" as well as the wiki configurations. I'll have to do some more work to get these installing properly.

I began looking into writing a custom authentication module for Mozdev which will sync the CVS users we already have with the users in Drupal and auto-assign them roles for that project (project owner or contributor). There are other projects that already do this, so I'll begin work there - this will greatly simplify the setup required for project owners and the Mozdev staff, so I think this is definitely worth the time.

Bug#13381 is still hanging out, but a lot of progress was made at the end of the week to solidify the lists and make sure they're as accurate as possible. There's some minor things to discuss, but it is pretty much ready for production.

I'll be on vacation from August 4th - 11th and I haven't figured out if Internet access is available or not yet; if I don't respond to emails promptly then I probably won't until I get back next weekend.