Looks like there's a nifty undocumented feature for gpg-agent that limits your max passphrase cache time to 1 hour, despite what you set the default-cache-ttl to. Luckily, it's just undocumented, not non-existant.

There is also a 'max-cache-ttl' parameter that you can set which defaults to 1 hour. The default-cache-ttl defaults to 10 minutes. This is a bit short for my needs; at work I am typically sitting in front of my computer for > 4hrs, and don't want to have to type my key in every hour (btw, you can also flush the cache by sending gpg-agent a HUP). So, I just increased max-cache-ttl to something a bit more useful. I still haven't figured out what the difference is yet, though.