XUL extension parsing

XUL extensions are constructed very simply. Documentation on how they are constructed is excellent.

At Mozdev.org we have been focused recently on helping users find projects more easily as well as preparing for the upcoming Firefox 3 release by creating tools for our projects to perform secure installs and updates more easily. In order to do this we needed information about the extensions hosted at Mozdev.org.

Getting information about an extension is fairly straightforward. I created several classes for opening XPI files and parsing the install manifest. I also created classes to interface with extensions, extension types, applications, and application versions. These all get used when someone adds a new download file to their project (backend sql and data).

The process for adding a new file goes like this:

  1. Our CVS download file update script will call MD_ProjectDownloadFile::add()
  2. This function determines if the file is an extension. If it is, it:
    1. Parses the install manifest to get guid, name, description, and supported application information
    2. adds the extension if its new
    3. associates the extension with the project
    4. saves supported applications and versions for that specific file (supported applications can change over time)
  3. The project owner can then login and publically release the file (this allows the project owner to have test releases as well as give the mirrors time to propagate the file) as well as verify the hash of the file (for secure installs and updates)
  4. A project owner also has the option of using Mozdev's update.rdf file to provide secure updates; these are generated/updated when the extension is updated with a new release.

Since Mozdev.org doesn't want to modify our user's files we provide project owners a link to their update.rdf file for each extension so they can include it in their install.rdf when packaging their extension. For new extensions we provide a tool that allows them to upload their install manifest to get the path for the install manifest (or a sample install manifest that has it included)

So while there's a lot of little pieces to providing extension browsing by application or secure installs/updates, the code can be easily broken up in order to make the process easier.

Resources:

Mozdev sysadmin meeting minutes for 2008-04-08

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • testing apache 2.2/php 5 changes
  • created project tagging policy/docs
  • started update.rdf generation for secure updates
  • started setting up test suite for mozdev.org code
  • this week is planned to be: finish up the hg setup plan (in progress), start working on hg, and try to get some sysadmin time to close some bugs (apache 2.2 rewrite bugs, web-visible cache directory setup)
  • asked about working on mercurial before svn; davidwboswell says an update to the roadmap is coming soon

Discussed sysadmin priorities

  • apache 2.2 and php 5 upgrade is complete; working on ironing out some bugs
  • https is being setup and tested

Firefox updates server load handling

  • no updates

Staging server migration

  • no news on VMs
  • server move should be highest priority now that apache/php setup is done

Other projects

  • if staging server setup drags on too long, sysadmin might move on to cvs perms with pam auth
  • project creation automation would be a good item to work on as well
  • openid would be a nice authentication mechanism for users; integrating this with other auth mechanisms isn't understood well right now

Code testing

  • mostly just unit tests now that PHP 5 is avaialble (PHP 4 couldn't do mock objects which was very limiting)
  • Doug will be writing some tests to verify certain web paths are working correctly
  • not really focused on full integration testing or continuous integration right now (the entire web stack isn't in version control, so we can't know when changes are made)

Mozdev status update W14-2008

Last week was mostly monopolized by testing/debugging after our Apache 2.2 and PHP 5 upgrade. There's still some bugs to iron out, but it appears that the main site and most project sites are running well.

Other tasks include:

I didn't quite get to writing out the plan for adding additional VCSes to Mozdev.org due to the short week and testing/debugging of our new web stack. I plan on getting started on that this week but most of my attention is going to be focused on getting our new Apache/PHP setup stabilized.

Mozdev sysadmin meeting minutes for 2008-04-01

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • project overview page/secure installs was generally well-received
  • several minor bugs in file management, presentation, and extension parsing were handled quickly
  • working on a tagging policy to help project owners know what to expect to be approved: http://www.mozdev.org/drupal/wiki/MozdevProjectTagging
  • improved download counters offered by mozdev
  • setup web-visible cache directory on the staging server w/ info for migration to production
  • work on update.rdf generation is going well; script to generate files is mostly complete; need to add 'update info' url to file management and test the resulting update.rdf files; this will require the web-visible cache directory setup to finish
  • should be starting on subversion setup plan this week and filing bugs using email I sent earlier to sysadmin@ as a base
  • visiting family this weekend; offline Friday

Discussed sysadmin priorities

  • nginx has been in production for about a week and has been handling load from updates fine
  • web stats are currently broken (since the 25th) due to nginx setup and apache2/php5 update
  • apache2/php5 is being setup in production today
  • backups have not been occuring (reason isn't known yet); gjm is working w/ osuosl to get them working again

Firefox updates server load handling

  • update was handled well; there were several spikes in bandwidth/requests but they were handled by nginx
  • silfreed/gjm will continue to work on improving nginx setup by serving more files, but the solution worked incredibly well
  • some projects get lots of 404 hits for their update.rdf requests; gjm will make a list of projects and silfreed will contact POs to make sure they're aware their users don't have an upgrade path

Staging server migration

  • mozilla has said "yes" to 2U rack space and probably a VM
  • osuosl is having disk capacity problems so a VM isn't immediately available
  • first plans for for VM (hopefully from OSUOSL) is moving download master
  • separating email and web traffic would probably be the next step
  • figuring out how to split projects up across servers would also be useful from a security viewpoint (split mozdev.org projects from hosted projects)
  • we'll still need to ask TWS to ship our server when we decide where its going (Mozilla or retirement)

Jocelyn the Vi user

Jocelyn's definitely going to be a vi user; when she's done typing she immediately hits 'escape' and will pound 'escape' repeatedly when things aren't acting the way she thinks they should.

Mozdev status update W13-2008

Last week's big event was the launch of secure installs with our project overview page. In only 5 days we already have 27% (119/440) of our extensions registered with our system. Unfortunately we only have 9% of the registered files setup to enable secure downloads (ie, they have verified their file hash) - we'll have to figure out if this is a problem with our tools or process or if it will just improve as new files are registered.

Other tasks touched this week include:

This week I'd like to finish up the update.rdf generation and begin working on the plan for setting up subversion at Mozdev.org.

Mozdev project overview and secure installations

Mozdev's project overview pages are now live and feature secure installations using InstallTrigger.

{IMG:503x334:/sites/silfreed.net/files/project-overview-cdn.png}

Right now we don't have the download graph or the project stats in place, but the main functionality is there:

  • Project name and description
  • Project tags - projects are tagged with supported applications automatically
  • Links to project tools
  • Project activity
  • Project extensions and downloads with InstallTrigger links

We're really excited to have this type of page in place that gives a nice overview of a project and available tools as well as providing everyone a way to install extensions securely.

Mozdev sysadmin meeting minutes for 2008-03-25

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner), ccaygill (MyCroft project)

Discussion was held publically in #mozdev

Discussed developer priorities

  • about ready to release the secure installation/project overview stuff; been working w/ a PO on testing things; trying to figure out a bug in the file release part, but otherwise things seem good
  • plan to get the secure installation announced and in production this week
  • did some testing of the lightweight web server last week and couldn't find any problems with the setup on vebzom
  • fixed a bug in Drupal authentication module when editing mozdev cvs users
  • changed the season starts for mozdev's themes to the equinoxes/solstices
  • secure updates shouldn't take too long - it's mostly backend stuff that won't really have a UI anyway; hopefully only a week left

Database policy (bug#15661)

  • we need to separate our policy decision from our technical limitations
  • Doug will file a separate bug about needing to be able to monitor what our project usage is

Discussed sysadmin priorities

  • preparations for apache2/php5 are underway
  • setup php eaccelerator at the end of last week
  • looking at spam filtering to reduce amount of cpu time used by increasing number of firewalled hosts

Firefox updates server load handling

  • 2.0.0.13 release is scheduled for today
  • still need to reduce amount of pages that get handled by apache/php

Staging server migration

  • moco might be able to provide a VM; hopefully hear back later this week
  • no news about VMs from OSUOSL yet

Mozdev status update W12-2008

This week I hope to get the secure installation released to project owners and begin work on the secure update.rdf generation for mozdev.org projects. The high priority remains to try to help get mozdev.org server load handled during Firefox updates, but things seem to be progressing well on that front. Hopefully I'll be able to get a test subversion server setup soon and being work on integrating that into Mozdev.org's architecture.

Mozdev sysadmin meeting minutes for 2008-03-18

Present: davidwboswell (David Boswell), cdn (Chris Neale), ericjung (Eric Jung), gjm_home (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • spring theme is updated
  • the project overview page is mostly complete; just trying to get POs to help with testing
  • started looking into subversion and mercurial configuration
  • will be focusing on testing nginx to help get it deployed soon
  • will get web-visible save directory setup soon

Discussed sysadmin priorities

  • started filtering spam bots on production; blocking bots after 10 spam/week
  • project creation scripts were updated to force lowercase names
  • nginx is setup on the staging server; performs much better for serving static content than apache

Automated testing of Mozdev.org

  • could be doable w/ Selenium
  • would be neat to get the community to suggest/create tests for mozdev.org
  • Doug will file a bug

Firefox updates server load handling

  • covered in developer/sysadmin updates

Staging server migration

  • no updates

Mozdev status update W11-2008

This week I'll be focusing on testing the lightweight web server to reduce Mozdev's load, trying to setup a new web-writable/visible save directory, and trying to deploy the secure installations.

Mozdev project overview update

Reactions to our project overview mockup were very favorable, and I've made good progress this week getting the page setup and much more functional.

Here you can see our very own 'www' project's overview page:
{IMAGE:465x231:/sites/silfreed.net/files/mozdev-project-overview-www.png}

As you can easily see, we don't have the download graphs or project stats up there yet, but the other information is all available.

Here's a screen shot of the cdn project's overview page:
{IMAGE:465x231:/sites/silfreed.net/files/mozdev-project-overview-cdn.png}

In this one you can see that they have a number of extensions, and the Link Widget extension even has a release (version 1.6). This is a link directly to an InstallTrigger() call in order to install the extension securely (the links are all served from an SSL site).

Since the Link Widgets extension has a release, you can see it as the latest release on the Flock extension list:

{IMAGE:465x231:/sites/silfreed.net/files/mozdev-app-flock-link_widgets-cdn.png}

Mozdev sysadmin meeting minutes for 2008-03-11

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • working on getting a project overview page
  • this will have links for a project (bugs, source, etc) as well as list some basic information, stats, and downloads for a project
  • mostly needed for the downloads section to give people a place to do secure installations from
  • started working on the spring theme for Mozdev.org this week
  • fixed up a small bug with the new apache rewrite rules
  • fixed a bug on D.MD.o with recently uploaded files that prevented users from downloading them
  • trying to tie bugzilla products to mozdev projects more tightly so its easier to tell if a project has a bugzilla product and link between them
  • ericjung would like to see Doug's efforts be diverted into projects with a wider audience
  • other than the spring theme, work is done from the roadmap. making sure the Mozdev server can handle the firefox updates and releases has been a recent priority and I've been working towards secure installation and updates as per the roadmap

Discussed sysadmin priorities

  • focused on fixing small bugs that have been going on for awhile since the new server was setup (several years ago)
  • root email was going unnoticed; gjm started going through the emails and trying to coordinate with Doug to get problems fixed
  • bugzilla stats were broken; server has been working on this since yesterday
  • FreeBSD security update was applied to staging & production
  • web content was synced from production -> staging to help with Apache2/PHP5 testing
  • trying to secure the server by limiting where mysql passwords are stored
  • moved some more old things from /sandbox to cruft
  • log rotation is being improved
  • mysql tables are being backed up to individual files instead of large per-database files
  • newsgroup creation was tested and updated
  • minor reconfiguration of ntpd, dns
  • lowered max # of smtp servers to help reduce load on server today

Firefox updates server load handling

  • gjm and silfreed have a plan in place to start serving update.rdf requests from a light-weight web server to allow more requests to be handled
  • Mozdev will also work with Mycroft to help them serve their update files directly from disk instead of going through PHP
  • last will be making downloads.mozdev.org able to serve update.rdf files from the lightweight web server
  • progress should be made by next week

Staging server migration

  • OSUOSL might have VMs available; gjm is going to look into obtaining one

Subversion priority

  • Mozdev is missing some high-profile projects due to only supporting CVS
  • there is only a couple weeks worth of work to get secure installations and updates setup; server load handling is interspersed in that
  • "subversion support" is actually a rather large problem due to Mozdev's CVS integration up to this point; Doug will write an email with the problems and migration plans he has so far
  • current plan is to continue with the current roadmap plus fixing server load handling problems and get to subversion support ASAP

Mozdev status update W10-2008

The biggest topic of the past week has been mitigating load problems on Mozdev's server when Firefox releases an update. We now have a plan in place to reduce the load by using a lightweight web server to handle update requests. We're also going to help out the Mycroft project update their site to use a new cache directory that is web-visible.

Other tasks include:

  • Started on project overview page - This will be the page where secure installations are performed from.
  • Fixed bug with .php pages and $_SERVER['PATH_INFO'] in new Apache rewrite rules
  • Fixed some notices that were making testing difficult and cleaned up a lot of functions/files/definitions in hovercraft/sandbox/php that were outdated
  • Tying Mozdev projects to bugzilla products more tightly - Right now Mozdev projects are only tied to our Bugzilla products by the project and product name; I'm trying to improve this since there are some projects (only mozdev-related right now) where the two don't match and it's difficult to automatically generate the links, or even know if a bugzilla product is enabled for that matter.
  • Fixed D.MD.o bug for recently-uploaded files - Files that were recently-uploaded that hadn't made it to a mirror yet could not be downloaded to to a bug that only served the first 8192 bytes of the file.

We are trying to get Mozdev's load problem resolved or at least improved before the next Firefox update (2.0.0.13) on March 25th so my focus will be testing our new features as they become available and helping with implementation.

Otherwise I'll continue working on the frontend for performing secure installations of extensions.

Welcome Mya González

Scott and Kristin González received their new little girl, Mya Lynn González, yesterday.

She was 6 lb 11.6 oz and 18.5 inches long.

Congratulations Scott and Kristin!