A handy dandy regular expression to parse out fields from Apache’s combined log format:
/^(\S+)\s # requestor
(\S+)\s # ?
(\S+)\s # ?
\[([^\]]*)\]\s # time
"([^"]*)"\s # URL
(\d*)\s # result
(\d*)\s # bytes
"([^"]*)"\s # referrer
"([^"]*)"$/x # user agent
I’ve been spending a lot of time porting over some code from ASP.NET to mod_perl2. Along with that rewrite, I’ve been migrating a side business away from managed hosting running Windows 2003 to Linux. Between the porting and migrating, managing the clients and working to grow the business, I’ve had time for little else.
I’ve discovered some pretty cool things along the way, though.
Apache2’s prefork model seems to work much better than its worker model, specifically when interacting with mod_perl2. Gathering specific statistics is difficult so this is strictly empirical data. The worker model worked just fine during testing and initial testing. Once I started pushing some real traffic through the system (on the order of 1M requests/day) I began noticing some odd behavior. Internal to my application, I track how much time it took to complete each request, from start to finish. Those times remained consistent throughout the process, but when I tried going through Apache it took a minimum of 20 seconds.
Something was obviously wrong so I started eliminating possible bottlenecks, like network, processor, and memory. All of those checked out fine. I ran ngrep (one damn fine tool, btw) and watched the request. It was hitting Apache, hanging for a while, and then spitting back a response. So I tested against a static file — it was fast. I wrote a small mod_perl2 handler that did nothing but return OK. It hung. So, I figure that something with the request being handed off from Apache2 to mod_perl2 wasn’t working right.
I ran Apache2 through strace but I didn’t see anything enlightening. I googled, read, and googled some more. I tweaked all of the worker mpm settings with no success. There were plenty of processes waiting for connection but no apparent reason for the requests to be delayed. Then I remembered someone on IRC asked me about prefork vs. worker. I had thought, because it was the default when I apt-get installed mod_perl & Apache2, that the worker model was the one mod_perl2 preferred.
So on a hunch I removed the worker mpm in favor of the prefork one. I had to tune the prefork settings a bit from the default. Suddenly everything became responsive. No processes sitting in “sending reply” and lightning fast response times from Apache. Memory usage seems to be lower and almost nil processor usage. It’s working so well, in fact, that I’m afraid to go to bed for fear it’s just my imagination.
Just a little bit of code I’ve been working on. Call this a proof-of-concept to test the geo-targetting library that I’m using. Taking that (a commercial product) and combining it with the Google Maps API, I’ve come up with something kind of fun:
It’s not 100% accurate and it only goes to the city level but it’s close enough for most purposes I need it for. It’s some pretty interesting technology, too. My next step is extending it to not only target the city you (or your ISP) is in, but other cities around you in a certain mile radius.
I’m writing a new web application using mod_perl 2.0. It’s heavy on network I/O so I’m doing some benchmarking and testing with simulated I/O to see just how many requests/second I can expect a single server to handle. While reading through Practical mod_perl I discovered one of the greatest tools ever: ab.
stone@moradin:~ $ ab -n 5000 -c 10 http://localhost/echo
This is ApacheBench, Version 2.0.41-dev < $Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/Benchmarking localhost (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Finished 5000 requestsServer Software: Apache/2.0.54
Server Hostname: localhost
Server Port: 80Document Path: /echo
Document Length: 33 bytesConcurrency Level: 10
Time taken for tests: 2.326156 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Total transferred: 1145916 bytes
HTML transferred: 165132 bytes
Requests per second: 2149.47 [#/sec] (mean)
Time per request: 4.652 [ms] (mean)
Time per request: 0.465 [ms] (mean, across all concurrent requests)
Transfer rate: 481.05 [Kbytes/sec] receivedConnection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.7 1 8
Processing: 2 2 1.2 3 10
Waiting: 0 1 0.9 2 8
Total: 3 3 1.2 4 11
WARNING: The median and mean for the waiting time are not within a normal deviation
These results are probably not that reliable.Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 4
95% 5
98% 7
99% 7
100% 11 (longest request)
This is awesome. Not only does it rock, but it’s included by default with Apache.
I’ve been doing some bug fixes and performance tuning on some perl modules of mine, mostly related to database access. I stumbled across this presentation from July 2004. If you’re doing any kind of DBI work, check out this presentation. You won’t be sorry.
I’m making good progress on podlet. I started working on the syncronization logic. There are a few important things to take into consideration before copying files to the iPod:
- What files need to be copied?
- Does the target have enough free space?
- Do we need to prepare the music before copying it?
ped recursively through my music folder, I found a few non-music items that I wouldn’t want to sync. I need to figure out a way to limit what files get syncronized. I can check file extention or mime type (assuming that checking the mime type does more than look at the extension). I also want to only syncronize music formats that the iPod support.
Checking that there is enough is critical, and there should be some kind of resolution progress if I’m trying to sync more data than the device can hold. I’m considering some kind of ‘conflict resolution’ dialog, so that if I can’t copy all of the music, the user can opt to not transfer files to the iPod, or let them open the iPod browser to remove some files.
Lastly, is there any processing we need to do on the file before we transfer it. This is more of a future enhancement (once the gstreamer-perl bindings are ready). Say you have an iPod mini that you use while jogging. For this, you’re more concerned with the quantity of music, rather than the quality. This option would let you to downgrade your music to a different format/bitrate so that you can fit more on a device.
I’ve found the Gnome2::VFS call to get the free space on the device. Once I decide on how check the file type (and I’m leaning towards mime type), I can calculate the needed space and then handle resolving any conflicts.
I’m working towards a 0.10 release sometime in the next week or so. I want to get the basics of syncronization working, so that I can get a few people to test that out while I start to add the bells and whistles like the iPod browser.
I finished integrating the code to auto-detect the iPod via HAL/Gnome2::VFS. I still need to test the VFS cleanup – making sure there are no leaks and whatnot. I also need to tackle automake/autoconf, or whatever it is that supplies ./configure and such. I’m not sure how I’m going to handle dependancies yet. libgnome2-vfs-perl in sid isn’t new enough. The latest version, along with Gnome 2.8, still sits in experimental.
I think for now I’ll worry about the ./configure stuff and make a note in the install file. It’s fairly easy to install the latest libgnome2-vfs-perl and it shouldn’t impact anything else on the system.
I finished writing the iPod auto-detection code tonight, which means i’m almost ready to move on to syncronization.
This was a fun bit of code to work on. It was my first time working with Gnome-VFS. It took a little while to get the perl binding to work. That ended up being an old version of the library in Debian. Once I installed the latest version, it worked like a charm.
stone@moradin:~/src/internal/podlet/tests$ ./ipod_detect.pl
iPod detected at /dev/sda2
Essentially, it gets a list of all mounted volumes and iterates through them, checking for the existance of the iTunesDB on the volume. When it finds the file, we’ve found the iPod. I have to add a couple signals to detect insertion and removable of volumes and I can call this piece done.
I spent my lunch figuring out how to use Glade2 and Perl. Gtk2::GladeXML makes it painfully easy to integrate Glade components.
I have a big chunk of the framework in place and I’ve started attaching widgets. First on the plate is the preferences dialog. The first tab sets the stage about how we’ll recognize and communicate with the iPod. Automatic mounting, or discovery, will use DBUS and HAL to know when the iPod is attached and mounted. Manual will simply probe the mount point and wait for the iPod to be mounted. There are two supported formats – iTunes and GNUPod. GNUPod is great if you only use your iPod with Linux, but if you ever want to sync between Windows, Linux, and a Mac, stick with the iTunes format.
Gtk2 pretty. Glade2 rocks.
I’ve talked about working on iPod support for Rhythmbox, and iPod support in general under Linux. I did do some work with Rhythmbox, but in the end I faltered and gave up on it. The future of Rhythmbox isn’t clear to me, and from the start I had qualms about writing application-centric support for something like the iPod. After much internal debate, I’ve decided to take a fork in the road.
I’ve been using Perl quite a bit lately. Then I discovered GNUpod. The wheels began to turn. I started to play with gtk2-perl. More cogs began to spin.
Thus was born podlet, a perl application that uses Gtk2::TrayIcon (which follows the freedesktop.org System Tray specification). In essence, it will be a complete syncronization tool, not just for music but also contacts, calendar, and tasks (not limited to, but including this bounty).
Proposed look of podlet, sitting in the Notification Area. The icon changes based on the current status – idle, syncronizing, or disconnected.
One- and two-way syncronization of music. I use my iPod with my laptop, but I don’t want to keep a full backup of it on there. One-way syncronization would let me syncronize music to the iPod without filling my hard drive. Two-way syncronization, as you would expect, would make sure that your library on your iPod is a mirror of that on your computer.
Calendar, contact, and task syncronization should work with any application you choose to use. There will be a plugin system to support various sources for this PIM data, so you can sync your iPod with Evolution, vcards, Thunderbird, Sunbird, etc.
In the end, I think podlet will make a pretty decent little syncronization tool. I’ve thrown my work into CVS and started to hack out the core of what needs to be done. I’ve also started to play with Glade. There will be a preferences dialog to manage podlet’s settings, and possibly a druid to be run on first run to walk the user through setting up syncronization.