The home movie bug

I recently purchased a Canon HG-10 camcorder. The primary reason was to start capturing Jocelyn's growth on video; she's almost walking now, and we wanted to get some video of our families through the Christmas holiday.

The camcorder is extremely light. It records directly to hard disk instead of MiniDV using the AVCHD codec. The 40GB hard drive at 15Mbps (highest quality) will store about 5.5 hrs of video - while the standard battery only lasts about an hour (the extended batter lasts a little more than two).

Unfortunately this codec is so new (only out this year, AFAICT) that very little video editing software or even player software exists. Even on Fedora 8 I can't play the .mts files (VLC comes close, but no cigar). Hopefully the underlying libraries that Cinelerra, Xine, mplayer, VLC, and others use will be updated soon to support the codec. In the mean time I guess I'll just hook the camcorder up to the TV.

Mozdev Status Update for W51-2007

  • bug#7021 - Project Tagging - Getting close to completion for this one - I have the project owner interface for tagging their project completed as well as the profile login/logout area. All that's left is the tag cloud and list of projects per tag for the user-facing pages, and an admin interface for making changes to tags.
  • bug#15327 - Document PHP usage - We needed to document that PHP was supported for project owners to use on their websites, so I added some information on what pages are available for editing in PHP.
  • bug#15312 - Upgrade to PHP 5 - PHP 4 is EOL at the end of the year, so we're trying to get PHP 5 setup. I've been going through our code trying to get it setup in a more secure fashion - mostly by starting to fix undefined variables and try to make our site work without register_globals.
  • Reported Comcast blocking Mozdev emails - Comcast was temporarily blocking emails from our server; we reported this and it appears to be resolved. The problem seems like it lasted from ~2007.12.15 - 2007.12.18.
  • Protected Wiki Pages - I added the Content Access drupal module so that we could make certain wiki pages "protected" - meaning either only editable by a certain group or even only viewable by a certain group. This module is available for all project owners to use in their Drupal module; feel free to contact me on the list if you have questions about its use.
  • bug#18161 - Upgrade bugzilla db to support UTF-8 - Our sysadmin has been testing out the UTF-8 upgrade and is almost ready to deploy this. Sometime near the beginning of the new year our bugzilla installation should properly support UTF-8 characters.

I plan on being available sporadically over the next several week or so until after the new year. I plan on finishing up the project tagging first, then probably getting our site ready for PHP 5. After that I plan on working on pulling application information out of install.js files in preparation of auto-labeling projects with the applications they support as well as setting up an automated update.rdf feed and secure installations.

Mozdev Sysadmin Meeting Minutes for 2007-12-18

Present: davidwboswell (David Boswell), gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • comcast's email block is removed
  • trying to plan how to move more code into CVS
  • writing auth, error, database, session, and tagging libraries
  • the backend for project tagging is done; going to start on frontend
  • hope to have project tagging done this week
  • PHP 5 will be very helpful with better support for objects

Discussed sysadmin priorities

  • gjm has been busy on other projects recently
  • apache/php update is continuing on vebzom; should be finished soon
  • BSD's ports system really doesn't want PHP4/5 and Apache 1.3/2.2 installed at the same time

Open sourcing more Mozdev.org code?

  • getting our PHP code into CVS shouldn't be hard; will just take some coordination to minimize downtime while the code is moved around
  • up-porting the CVS mysql auth patches to a current version of CVS might be pretty difficult
  • pam-mysql/nss-mysql might provide us the means to use a regular version of CVS and auth against or custom DB
  • we'll look into pam/nss integration after apache/php upgrade is complete
  • should we have a sysadmin-focused roadmap? Perhaps a protected wiki page would be better - Doug will research

Dealing with newsgroups

  • sometime in the past the automatic mail list/news gateway was broken
  • periodically a new project asks about the newsgroup integration and it has to be manually setup
  • this involves both the newsgroup being created as well as mailman being setup properly
  • smssidebar has requested newsgroup be setup; gjm will look into it

Documenting PHP usage

  • PHP is available, but some POs don't know that
  • concerns over how open our PHP installation was and the privileges that are granted was discussed; discussion was moved to the sysadmin list
  • Doug can document that PHP is allowed for now; it can be modified later if needed

Next meeting 1/8?

  • next two Tuesdays are holidays
  • email will work if something else comes up
  • meeting is rescheduled for 2008.01.08

Mozdev status update for W50-2007

This week I spent a lot of time trying to get a better code library setup for Mozdev code that would be easier to maintain and test (mostly meaning, starting to write stuff in OOP style despite PHP 4's lackings). Towards that end, I've ended up getting session managent, authentication, database, and error classes setup. I'm trying to make use of PEAR libraries as much as possible; so far I've used HTTP_Session, Log, and MDB2 to name a few. Using these have really helped me move things along faster, even if it's very difficult doing OOP in PHP 4 (hopefully PHP 5 comes soon). Most of this is being done because we need a more centralized way of "logging in" to the website for the upcoming features (project tagging, secure installations, and eventually better account management).

In a nicer, easier to follow list:

  • Trying to open source more of Mozdev's code - This mostly means getting more code in CVS and making sure out CVS auth patches work. The former shouldn't be too difficult, but the latter is probably going to be a problem.
  • Tagging projects - I finally got a lot of the prerequisite code done (mentioned above), so I started working on writing the libraries for managing tags per project.
  • Did NOT enable download file RSS feed by default - The results say not to, so we'll let it off by default. Project owners can still enable it if they desire.
  • Posted new project adoption policy (bug#18259)
  • Mozdev downtime - We had two bouts of downtime this week for the server; we still haven't discovered a reason for the first, longer one. The second one only lasted about a half an hour and was the result of a DOS against our server. We're going to be looking into measures to handle that in the future; more news on that after it's addressed.

Next week I'll continue working on the project tagging. Once that's finished up the next project is work towards secure installations, tagging projects by applications it supports, and possibly automatic generation of update.rdf files if the community desires.

Mozdev sysadmin meeting minutes for 2007-12-11

Present: davidwboswell (David Boswell), ericjung (Eric Jung) gjm (Gerry Murphy), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Mozdev downtime

  • no obvious causes for downtime found yet
  • box was logging up until the server rebooted, so it was still active
  • suspected hardware problem since the box had tremendous uptime otherwise (418 days, IIRC)
  • we can possibly plug in the 2nd NIC in case there is a problem with the current one
  • contact information between parties was missing (except for the site), so contacting the right parties was difficult; David will collect contact info and distribute it via email so everyone has a copy

Disaster Recovery

  • How can we get the news about a site outage out to people?
  • sysadmins are looking into restarting services automatically to try to recover from problems similar to what we experienced
  • sysadmins will put together recommendations on how to handle outages like this in the future
  • sysadmins will get Doug access to the KVM so he can debug (different timezone, so we cover more waking hours)

Discussed developer priorities

  • mirror bugs are done and implemented (closest mirror redirection, downloads directly from D.MD.o, and download files rss feed)
  • server load from additional mysql queries seems to be minimal after freeing up memory on the server last week
  • lots of discussion on mozdev providing an update.rdf feed for projects
  • renamed RSS feeds to remove the .html extension
  • setup lots of projects with bugzilla permissions
  • Drupal 5.4 and 5.5 security upgrades
  • working on a session and authentication system for the new features
  • been doing design for project tagging
  • poll is ongoing to see whether to enable the download file rss feed by default

Discussed sysadmin priorities

  • memory leak in Apache was tracked down to a small portion of Drupal code but not php-mysql related (specifically, the user registration pages but the problem wasn't tracked down more than that)
  • apache2/php5 upgrade on vebzom is progressing; this seems to fix the memory leak in Drupal as well
  • nightly processing of mailman held queue was crashing due to a bug in the french translations

Open sourcing more Mozdev.org code?

  • Pete says to use the hovercraft project, but otherwise didn't raise any objections
  • CVS patches are in hovercraft but will probably be difficult to update to current CVS
  • other mozdev code shouldn't be hard to get back into CVS; just needs to be planned well in order to minimize downtime
  • David will look into contacting people @ collabnet to see if anyone still maintains a mysql auth patch for CVS

Mozdev unexpected downtime

Our main mozdev.org server is currently unavailable making all services unavailable (web, CVS, downloads, mailing lists). The problem seems to have started at approx. 22:45 UTC (17:45 EST). We have our colocation provider looking into the problem and will update once more information is known.

Update 21:20 EST/02:20 UTC
Mozdev.org is back up as of 21:15 EST/02:15 UTC. Services will most likely be slower for normal for a short period while the server processes a backlog of email. No word yet as to what might have happened; we'll post again once more information is known.

Mercurial repo for RPM package development

I've setup a Mercurial/Hg repo for my packages.

I was going to try out git, but decided Mercurial would be my first venture into DVCSes.

The wiki at Mercurial's website was pretty helpful in getting a central web repository up and running; create a single script and point it at the repository (which is just a directory that you've run 'hg init' on), some light virtualhost config and hgrc settings, and you're pretty much done.

The bulk of the work for me has been on the backend; getting VMs setup in vmware server on my file server; moving my packages to my file server instead of my laptop; migrating the packages, specs, and sources into a packages/ directory (instead of being along side RPMS, SRPMS, and tmp/).

I still need to make sure mock works well in all my VMs and get InstantMirror on my home firewall, but things are starting to look like they're setup pretty well for my little home packaging setup.

Hopefully getting the development process easier for myself will make me more willing to do more packaging and make me realize how many packages I have to merge into Fedora.

Mozdev status update for W49-2007

Lots of progress on squashing mirror bugs!

Other tasks:

This week I hope to begin working on the tagging of projects; to get to that I first need to get a login system for Mozdev.org so this and the upcoming tools have a central "account" to log into rather than each page requiring credentials.

New downloads.mozdev.org features

New features have arrived at downloads.mozdev.org (D.MD.o). I'll just enumerate them to save space:

  1. Redirect users to their closest mirror (bug#2807)
    You can check your closest mirror (and list of other available mirrors) by visiting the following link: http://mirrors.mozdev.org/ip-check.html. You will be redirected to your closest mirror automatically; no need to select it from the list when you download a file.
  2. Providing downloads for recently uploaded files directly from D.MD.o (bug#17900)
    This makes it much easier for project owners to distribute test packages and get critical updates out to their users.
    For regular file releses we ask that you post your download file several hours (at least 4 hours) before you announce your new file or post an updated updates.rdf file. This will ensure that your new file has a chance to be distributed to as many mirrors as possible.
  3. Additional download statistics - mirrors.mozdev.org has been updated with new stats that show the number of downloads from our mirrors in the past 30 days by mirror and by region.

We hope that everyone can make good use of these new features. As always, feel free to report bugs in our bugzilla instance if you encounter any problems.

Mozdev sysadmin meeting minutes for 2007-12-04

Present: cdn-work (Chris Neale), davidwboswell (David Boswell), gjm (Gerry Murphy), kinger (Brian King), silfreed (Douglas Warner), tanker (Michael Dosser)

Discussion was held publically in #mozdev

Discussed developer priorities

  • theme cleanups for mozdev.org w.r.t. http/https
  • downloads.mozdev.org updates - redirection to closest mirror and serving files directly from D.MD.o - work is staged, waiting deployment later today
  • mirror work is almost done; the "secure installs" feature is kind-of mirror work, but not lumped in there right now. we'll discuss this during the roadmap updates
  • bugzilla permissions have been requested by about 10 projects so far
  • upcoming this week is to deploy the download server changes and start working on categorization of projects

Discussed sysadmin priorities

  • cleaned up the bugzilla db; errors reported by the sanity check are fixed
  • mailman messages are still stuck in the held queue
  • web stats are being reprocessed to correctly handle our "unique" log format
  • apache config was modified to reduce the number of requests per child in order to free up memory
  • current theory is that there is a memory leak in php-mysql because the problems started around when new features were added to downloads.mozdev.org and drupal was released - both of which make much more use of mysql than before

Open sourcing more Mozdev.org code?

  • there have been project owners in the past and recently who have been interested in contributing to the site but are unable to
  • the goal is mostly just to make the code available; not so much to get mozdev.org to a "packageable" state as in the hovercraft project
  • the code under /sandbox/php is available in the hovercraft project but is very out of date
  • getting this code into CVS would ease the barrier-of-entry to new developers - they most likely wouldn't need server access as early
  • Pete and Chris probably have more ideas on why this isn't in CVS yet - Doug will write them to check

Revising roadmap priorities (reversing order of 2nd stage?)

  • adding secure installs to the first priority block
  • flipping order of second block so that subversion is first
  • there are still a lot of questions about how we will support SVN, CVS, and potentially other VCSes and general web space
  • apache 2 + PHP 5 upgrade will need done soon, but won't be added to the roadmap

CVS mysql auth patches

  • we don't know where the MySQL patches to CVS are currently
  • our CVS server is fairly old (v1.11.11) and it would be good to update it
  • Doug will contact Pete to find out where the most recent patches are
  • it would probably be best to put this under custom.md.o and advertise the patches

Bugzilla UTF-8 upgrade

  • will require some downtime after testing on vebzom.org

Amazon VM discussion - VM vs Physical Machine

  • VMs are fast to deploy, but slow at I/O
  • push spam or other email processing to the VM? possible, but Amazon charges for bandwidth as well as CPU time and disk space, so might not be cost effective.
  • OSUOSL is offering VM services - this might be the best solution for us for adding quick horsepower

Mozdev status update for W48-2007

This past weeks activities included:

This week we're going to re-evaluate some of the items on our roadmap that are in the second group. I'll be trying to implement some of the download redirection features I've been working on (if they don't overload the server again) and writing up the documentation on how to setup/request per-product permissions in Bugzilla. I'll most likely start the research on cleaning up our product listing and categorization in order to get that ball rolling along.

Reply to Bill Ruud of Shippensburg University for sending a .docx newsletter

Note: I can't open .docx files so I don't even know what his "Fall Letter" was about. I don't think that sending out office documents is the best idea for a newsletter, either.

Thanks for sending out what I assume is a status update and report on what has and will be going on at Shippensburg University. Unfortunately, I, and I assume a large number of other recipients, cannot open your attachment.

For newsletters and the sort, the most preferred method of distribution would be a text-only email - this ensures the widest ability to receive and open the content.
HTML-encoded email with a text-only option included would be the second best since you could add formatting to your content while ensuring that the content is readable by a wide audience - this frequently requires twice the work as it's probably best to format both the HTML and the text-only documents separately so-as to ensure the best readability.
I think the last option would be to send out a link to a web page so that everyone could be viewing the document through a web browser that would ensure support of HTML - this would unfortunately require much more work than the previous two options as any styling would have to be tested in multiple browsers.

Your choice to send the newsletter as an attachment of a Microsoft Office 2007 document drastically limits the audience that can receive the newsletter. Many (myself included) prefer to run open and free software on their computers and cannot access these documents. There are even more that do run Microsoft products but can't afford to keep up with constant software upgrades or other alternative OSes such as Mac OS-X; for those, it's very common to use the same free software to do their office work (OpenOffice.org, usually). It is much more desireable to use an ISO standard for office documents, such as the OpenDocument format (ODF; http://en.wikipedia.org/wiki/OpenDocument), so that everyone has the same access to software that can open the documents. Still, this isn't the preferred format for newsletter distribution.

I hope my ideas can be useful for the making of your next newsletter. I don't expect that it's your job to keep up with the best distribution methods for newsletter, so perhaps another option is to run your drafts through your marketing or IT departments for ideas on how best to get your newsletter to your intended audience.

Thanks for your time.

-Douglas E. Warner
Shippensburg University Alumni 2002

Update 2007-11-29: Bill Ruud sent out another email that was text-only; thanks! Hopefully others will benefit from my lengthy rant as well.

New Drupal/Gallery2.2 plugin configuration challenges

I like to my rewrite rules performed in the central Apache config (well, individual vhost files) so it can be revision controlled and centrally managed. The recent upgrade to the Gallery 5.x-2.0 module broke the rewriting rules I was using. Below are my new rewrite rules for a site-wide Gallery/Drupal install, but first the main snipped and differences of what was changed. For reference, my gallery install is at /photo.

The recommended URL rewrite rules would look like this:
RewriteCond %{THE_REQUEST} /gallery/([0-9]+)-([0-9]+)/([^/?]+)(\?.|\ .)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /photo/main.php?g2_view=core.DownloadItem&g2_itemId=%1&g2_serialNumber=%2&g2_fileName=%3 [QSA,L]
RewriteCond %{THE_REQUEST} /gallery/([^?]+)(\?.|\ .)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_path=%1 [QSA,L]

I had to modify/remove the emphasized parts; I needed to change %{THE_REQUEST} to %{REQUEST_URI} because the path /gallery is also in drupal's admin paths. I wanted to anchor the /gallery gallery path, but can't do that with %{REQUEST_URI}. Also, because I removed %{REQUEST_URI}, I needed to remove the (\?.|\ .) match from the end since there would no longer be those characters after the URL.

My modified version would like like this (emphasis added to show changes):
RewriteCond %{REQUEST_URI} ^/gallery/([0-9]+)-([0-9]+)/([^/?]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /photo/main.php?g2_view=core.DownloadItem&g2_itemId=%1&g2_serialNumber=%2&g2_fileName=%3 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/([^?]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_path=%1 [QSA,L]

This wasn't good enough for me, though, as I wanted the clean URLs I had for most gallery paths in gallery 2.1. So my final Gallery2.2/Drupal config looks like the following:

# drupal/gallery integration
RewriteCond %{REQUEST_URI} ^/gallery/popular
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=dynamicalbum.PopularAlbum [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/updates
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=dynamicalbum.UpdatesAlbum [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/random
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=dynamicalbum.RandomAlbum [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/d/([0-9]+)-([0-9]+)/([^/?]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /photo/main.php?g2_view=core.DownloadItem&g2_itemId=%1&g2_serialNumber=%2&g2_fileName=%3 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/v/([^?]+)/slideshow\.html
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=slideshow.Slideshow&g2_path=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/c/view/([0-9]+)\.html
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=comment.ShowAllComments&g2_itemId=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/c/add/([0-9]+)\.html
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=comment.AddComment&g2_itemId=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/v/([^?]+)\.davmount
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=webdav.DownloadDavMount&g2_path=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/rating/([^?/]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=rating.RatingAlbum&g2_limit=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/rss/([^\/\?]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=rss.Render&g2_name=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/srss/([0-9]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=rss.SimpleRender&g2_itemId=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/key/([^?/]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_view=keyalbum.KeywordAlbum&g2_keyword=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/w(/[^?]*)?
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_controller=webdav.WebDav&g2_path=%1 [QSA,L]
RewriteCond %{REQUEST_URI} ^/gallery/v/([^?]+)
RewriteCond %{REQUEST_URI} !/index\.php$
RewriteRule . /index.php?q=gallery&g2_path=%1 [QSA,L]

# drupal configuration
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^/(photo|album|download)(/.*)?
RewriteRule ^(.*)$ /index.php?q=$1 [L,QSA]

Mozdev sysadmin meeting minutes for 2007-11-27

Present: Chris Neale (cdn-work), David Boswell (davidwboswell), Eric Jung (ericjung), Gerry Murphy (gjm), Doug Warner (silfreed)

Discussion was held publically in #mozdev

Discussed developer priorities

  • drupal tweaks to install profile
  • helped move bugzilla bugs along for bz 3 upgrade
  • updated front page to pull news from our blogs
  • cleaned up mime type detection to fix some bugs I encountered
  • worked on per-product perms for bugzilla
  • we can start offering per-product perms to users and I can handle setting them up
  • I can talk to bz devs to see what problems we might encounter in the future w/ lots of groups

Server load discussion

  • David mentioned he's noticed the server getting slower recently
  • server started swapping in October and hasn't stopped
  • gjm/tanker will start looking into ways this can be improved (additional server, apache cleanup work, etc)

Discussed sysadmin priorities

  • mailing list delays have been eliminated
  • spam/viris filtering was scanning mailing list data twice
  • held message time was changed to 30 days; dropped queue from 120+k messages to 66k messages
  • bz upgrade to 3.0 went smoothly
  • bugzilla database is split from the tigris db now
  • apache rewriting improvements continue
  • will be looking into PHP5 upgrade on vebzom soon
  • discovered a problem with the web stats - countries is showing the virtual domain's contries, not the visitor's country; work is pending
  • fixed bugzilla cookies working outside /bugs/ - potential security leak

Open sourcing more Mozdev.org code?

  • Skipped this week

Blog updates on the front page

  • have a manually viewable "news" section on www and a blog w/ 2-3 items

Add Drupal and Mailman to project request form?

  • we should clean up and streamline the project request form
  • remove bugzilla components and add a tools section
  • tools and contributors should be hidden by default

VCS survey poll current results

  • 48 responses; 77% SVN, 17% CVS, 14% Hg, 15% git