Mozdev project overview page

Something Mozdev.org has lacked for awhile now has been a project overview page that shows you a nice overview of a project with links to tools for a project as well as some basic information and stats.

As part of working on secure installations for projects I needed a page that showed the available downloads for a project and I thought it would be a good time to put together a page that had a little more information on it as well.

There's a lot going on in this mockup; we have:

  • Project name and description
  • Project's tags
  • Links to available tools
  • Project activity
  • Project stats (downloads, page views, hits)
  • Extensions a project publishes
  • Files that have been released by a project
  • Secure installation of extensions

With all this information there (and potentially more in the future) we want to make sure it's not too crowded and that the information is relevant.

Let us know if you have any suggestions on how we can tweak things to make the information more usable for both people browsing for a project and for developers.

Update : added link to full-res mockup

Mozdev sysadmin meeting minutes for 2008-03-04

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Doug Warner)

Discussion was held publically in #mozdev

Discussed developer priorities

  • working on the UI to allow project owners to verify and release their downloads and extensions
  • working on mockups for a project overview page that would contain the publicly available downloads
  • trying to cleanup the queries used on D.MD.o
    • reduce CPU usage a bit by throwing users to a random mirror for update requests instead of their best mirror
    • able to speed up the query that gave users their best mirror
  • the extension list and tagged projects for an application are now merged into one list that shows both
  • notified POs that the Apache upgrade will be happening March 31st and helping POs prepare for that

Discussed sysadmin priorities

  • stats processing is up to date and the cause of the backlog is being addressed
  • more spam blocks are in place
  • there's a problem w/ spamassassin dying and letting spam through but the cause hasn't been found yet
  • thinking about discarding all spam to lists that has a high spam score

Firefox 3 release

  • mycroft is currently the biggest bottleneck when mass updates happen
  • we should focus on helping them get static updates provided
  • mycroft and downloads might have problems with updates not going through their scripts because stats will no longer be counted

Staging server migration

  • no updates

SSL Cert renewal

  • new cert is in place for 02/2008 - 03/2009

Protected wiki for sysadmin docs

  • server docs were setup on a protected wiki page that only "mozdev" people can view/edit
  • we can setup perms per-page on the www wiki or setup a protected wiki on admin

Per-project databases

  • we don't know what projects currently have access to mysql
  • projects currently can't configure db access w/o exposing the password
  • need to work to move projects w/ existing mysql access to their own dbs and then work on improving the policy to allow general access

Mozdev status update W9-2008

Jocelyn's first birthday and 2nd Warner Baby on the way

{URL=http://www.silfreed.net/gallery/v/life/Family-Photos/Jocelyn/Birthday+cake/}{IMAGE:http://www.silfreed.net/photo/gallery/d/29837-2/img_3347.jpg?g2_GALLERYSID=0d407f91ad9d5b3d00efd2f046e5a1e3}{/URL}
Jocelyn's first birthday has come and gone. It was a blast to have all her family around on her birthday while she opened presents and ate cake. It's definitely been a very busy year and has flown by. I'm looking forward to spending many more years with Jocelyn.

{URL=http://www.silfreed.net/gallery/v/life/Family-Photos/Baby2/7+week+ultrasound+-+2008_02_14/}{IMAGE:http://www.silfreed.net/photo/gallery/d/29718-2/peanut-20080214-153236.jpg}{/URL}
The second bit of news is the arrival of our new peanut! It looks like our new baby will be coming along somewhere between Sept 27th and Oct 1st, 2008. We're both very excited to have another little one in the future.

I don't like the new Runts

I don't like the new runts.

They replaced the green (lime) and red (cherry) ones that used to be there with a new yellow (pineapple) and red (mango). Here's a look at the perpetrators:

{IMAGE:/sites/silfreed.net/files/img_3360.jpg}

After you remove them, this is all you're left with:

{IMAGE:/sites/silfreed.net/files/img_3361.jpg}

Orange, pink (strawberry), and yellow banana (which nobody ever liked). At least they got rid of blue (blueberry) at the same time.

Mozdev sysadmin meeting minutes for 2008-02-26

Present: davidwboswell (David Boswell), ericjung (Eric Jung), gjm (Gerry Murphy), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • skipped; silfreed not present

Discussed sysadmin priorities

  • gjm has written some python classes to deal with mailman configs; this makes it easier to do validation & reconfiguration
  • defaults for new mailman lists have been setup
  • gjm has been reviewing site content and submitting bugs

Firefox 3 release

  • no updates

Staging server migration

  • migration proposals sent to sysadmin list; people will review and comment there

Dealing with newsgroups

  • project creation scripts are now fixed to setup newsgroup/mailman sync

Incremental compressed backups with perms and no root keys

I've looked for incremental backups that don't require root before, but I finally found it. Similar to rdiff-backup is a program called duplicity. It uses tar, gpg, and the rsync algorithm to store encrypted differential backups.

duplicity needs to be scripted a little bit to automate the backups. I came up with a script that will backup certain directories fully once a week and incrementally the rest of the week. I also disabled the encryption since the transfer method is secure (ssh in my case) and the end server is trusted. If you were using Amazon's S3 service or a public FTP share you might want to tweak this a bit.

/etc/cron.d/backup

32 3 * * 1-6 root /usr/local/sbin/backup.sh<br />32 3 * * 0 root /usr/local/sbin/backup.sh --full

/usr/local/sbin/backup.sh

#!/bin/bash<br /><br />v=&quot;-v0 --no-print-statistics&quot;<br />cmd=&quot;&quot;<br />duplicity=&quot;duplicity&quot;<br />opt=&quot;--no-encryption --volsize 100&quot;<br />dest=&quot;scp://taonas@thor.home.silfreed.net&quot;<br />maxage=&quot;1M&quot;<br />PASSPHRASE=<br />export PASSPHRASE<br /><br />for arg in $@; do<br />    [[ $arg == &quot;-v&quot; ]] &amp;&amp; v=&quot;-v4&quot;<br /> [[ $arg == &quot;--full&quot; ]] &amp;&amp; cmd=&quot;full&quot;<br />done<br /><br />dcmd_backup=&quot;$duplicity $cmd $opt $v&quot;<br />dcmd_age=&quot;$duplicity remove-older-than $maxage $opt $v --force&quot;<br /><br /># backups<br />$dcmd_backup /etc $dest/etc<br />$dcmd_backup /var/log $dest/var-log<br />$dcmd_backup /home \<br />     --exclude /home/silfreed/tmp \<br />    --exclude /home/silfreed/src/silfreednet/tmp \<br />    --exclude /home/silfreed/src/workspace \<br />  --exclude /home/silfreed/src/mozdev/workspace \<br />   --exclude /home/silfreed/.thunderbird/7g3nnt02.default/ImapMail \<br /> $dest/home<br /><br />mysqldump_dir=/tmp/mysqldump<br />mkdir $mysqldump_dir &amp;&amp; \<br /> chmod 700 $mysqldump_dir &amp;&amp; \<br />     mysqldump -u root  -A &gt; $mysqldump_dir/mysqldump.sql &amp;&amp; \<br />      $dcmd_backup $mysqldump_dir $dest/mysql<br />rm -rf $mysqldump_dir<br /><br /># age out paths<br />for path in /etc /var-log /home /mysql; do<br />     $dcmd_age $dest$path<br />done

Mozdev status update W8-2008

  • Download file registration/release backend (bug#12486, bug#17302) - the UI to allow projects to register and release their download files is almost complete. This will allow project owners to allow secure installations of their extensions and generation of update.rdf files once those features are complete.
    There were a lot of small backend fixes such as adding support for 'deleted' files in our cache, fixing some detection of when a file is replaced, and fixing a bug introduced over the previous weekend that broke our caching of new files for several projects.
  • Reduce D.MD.o load - I was able to reduce the load on our servers by redirecting update requests to a random mirror instead of the user's best mirror.
    I found that 62% of users hitting our server for an update only requested the update.rdf file (or equivalent). Out of that 62%, only 1.5% actually retrieved an update after that. So we were spending a lot of time trying to figure out where to send the user for a very small file.
  • Misc page cleanups (bug#18671, bug#18669, bug#18677, bug#18667)

This week I'm taking off tomorrow for my daughter's 1st birthday, but I hope to get a good start on the project download file interface that will allow secure installations of extensions. I'll also be done with handling a project's applications and will have that released and the documentation to go along with it.

Mozdev sysadmin meeting minutes for 2008-02-19

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm_home (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • sick last week; didn't get a lot of stuff done
  • mostly spent time fixing minor bugs with the new application/extension parsing
  • started working on interface for managing download files (allowing public releases, verifying hashes) to get ready for secure installations
  • for site reorganization I'm going to be working on getting a list of content put together in the wiki to see how it could be organized
  • trying to figure out how to relieve some load from D.MD.o; best I can think of doing is disabling the new download tracking
  • possibly disable the download tracking when server is over some specified threshold? possibly; will need to investigate

Discussed sysadmin priorities

  • gjm was ill last week
  • tanker helped fix permissions on project

Firefox 3 release

  • looking for ideas to get our server through the update cycle that goes along with a release
  • disable services dynamically under load
  • offload some services to vebzom.org
  • migrate postfix/mailman/bugzilla to another server
  • looking up an IP's mirror is a heavy operation; silfreed might look into caching these results
  • we need a better understanding of how much a user hits our site when performing updates; silfreed will analyze log files

Mailman defaults

  • we'll change the defaults for new lists and point people to the wiki page if they need to reduce their list management time

Staging server migration

  • proposal still needs typed
  • OSUOSL admins say it would be possible to have a second [physical] machine hosted there

Dealing with newsgroups

  • gjm worked with Pete to get newsgroup creation scripted, tested, and ready to integrate with project creation scripts. still needs implemented

Mozdev status update for W7-2008

This week I will spending some time ensuring our download system can handle FireFox updates (we had some problems handling the last FireFox 2.0.0.12 update), working on ideas for improving site navigation, and continuing on a download file registration system to be used by Project Owners for approving files and verifying uploads.

Mozdev sysadmin meeting minutes for 2008-02-12

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • project searching now searches words 3 characters and more (so searches on things like 'rss' or 'moz' work now)
  • we're automatically parsing extensions and pulling extension information and supported application information
  • the application list with descriptions, icons, and links to home pages is live
  • the list of extensions per application is live; still needs some work to enable features such as the "latest version"
  • project searching now includes applications and extensions
  • will be spending some time soon thinking about the redesign and how to make finding things on our site easier

Discussed sysadmin priorities

  • more research into the CVS mysql auth patch - it's going to be incredibly difficult to port this to a more recent CVS so that we could add additional features to it
  • we will be investigating using pam_mysql/nss_mysql to manage CVS permissions on the filesystem instead
  • pam_mysql will make implementing other services easier as well (ftp, ssh, VCSes)
  • 15m downtime on Friday due to a spike in the server load caused by firefox release
  • spam scanning has been improved to use in-memory partition for parsing messages (increases throughput)
  • spam scanning has been stricter and is now blocking ~800k dailing (up from ~140k)
  • web log analysis is moving faster; a caching DNS server was installed to reduce DNS latency
  • firewall config was updated to allow more connections
  • apache2/php5 update has been on hold; Doug says PHP code should be ready and will need more testers soon

Mailman defaults

  • less spam should help a bit, but no new updates
  • we should document the changes we are looking to change in the wiki so that we can point people there when they have questions

Staging server migration

  • plan is still in the works; needs typed up an emailed out

Dealing with newsgroups

  • no updates

Mozdev Status Update for W6-2008

  • Finding supported applications automatically - I have the portion that parses .xpi files' install manifests (install.rdf files) working - it currently tracks what extension an .xpi file is as well as what applications and version ranges it supports. The frontend for displaying the applications Mozdev.org projects support and the extensions for each application is almost complete; only some minor styling work left. An admin interface is also complete for managing information about the applications. Left to do is to link this data into the project search interface (bug#18525) and create a download file manager for project owners to be able to "release" a download file for public consumption.
  • Updated the project search interface to support more search words - previously we only supported searching projects and descriptions with words over 4 characters or longer; we updated our database to support searching words 3 characters or longer - this should make the tool much more useful for finding projects.

This week I'll continue to work on the frontends for displaying supported applications and searching applications and extensions as well as the download file management interface (which will be crucial for secure installations as well).

Welcome Noe Paxton!

Congratulations to Ben and Jen! Reproducing the details from their email:
Noelani Elizabeth Paxton
Born: 12:05AM Feb 11 2008
7lbs 12.2oz 18.25 inches

Dear Lazyweb: Garmin Forerunner 50 linux support

I'm thinking about getting a Garmin Forerunner 50 but am concerned that the ANT USB thing won't work with Linux. I've seen a thread about ANT development drivers going into the kernel, but I haven't found any firsthand experiences. Anyone with any information about it? If it helps, the Garmin Forerunner 405 supposedly uses the same ANT technology.

Mozdev sysadmin meeting minutes for 2008-02-05

Present: davidwboswell (David Boswell), cdn-work (Chris Neale), ericjung (Eric Jung), gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • project search interface is live; needs some tweaks and am coordinating with sysadmins
  • parsing extensions for supported applications is underway; I'll be starting testing soon, then need to create a UI for searching and displaying the extensions per application

Roadmap updates

  • What's left before subversion? What apps are supported by an extension (bug12486), secure installs (bug#17302)
  • ericjung would like to see an online version of mccoy
  • Doug will send a note to the PO list to see if anyone is interested in helping code the backend; if not, we'll try to fit it into the roadmap

Discussed sysadmin priorities

  • stats processing is behind; logs are being generated quicker than being parsed
  • checking on mailman default settings bug#17667

Staging server migration

  • gjm and tanker have some proposals; discussion will continue in the sysadmin list to include all parties
  • being able to support SSL for some subdomains would be helpful - for example, bugzilla - more discussion will occur on-list

Dealing with newsgroups

  • gjm has created a "sanitycheck" python class to verify list/news setup and has lots of "issues found"