Broken dependencies

Ah the joys of being a Fedora contributor!

The first hint of trouble are the failed to build from source error messages. Apparently, the development versions just stopped building. Upon investigation, it appeared I had an incorrect dependency. Since I was using uuid libraries, I had a dependency on uuid-devel. Well, it turns out that that was for a different set of uuid functions. The ones I wanted were in e2fsprogs-devel… the package for file system maintenance ?!? Well, apparently I’m not the only one who found that odd, and they split that package into ones that made more sense. Hence the breakage.

Net result: Existing packages work, although I’ll update their dependencies to the correct ones. New packages (For Fedora 12, so you don’t have to worry about it yet) will be updated.

Systemtap

So I promised a while ago to describe my systemtap experience.

My initial intent was to use dtrace, and as it isn’t available on Linux, I spent some time playing with OpenSolaris. In the course of my web searches I stumbled across Systemtap that is supposed to provide at least equivalent or sometimes better functionality. I won’t comment on that, because as soon as I found it, I abandoned dtrace. This is nothing against dtrace, but my primary development platform is Linux.

The task I was tackling was investigating performance for my ARM4 agent. In particular, I wanted to improve performance when run on machines with multiple cores to improve scalability. Conventional debuggers don’t really do this, and profilers only give you part of the picture… especially when multiple processes are involved. One question I had in particular was how much time I was spending waiting on mutual exclusions.

So I start playing with the example scripts. Overall it’s pretty easy. In fact, I probably spent as much time setting up a good test scenario as I did developing my systemtap scripts. Their examples for monitoring IPCs are pretty good. Not stellar, but good.

I got a few surprises. Primarily, I was astonished to see how much time I was spending copying memory using message queues. This is by far the bulk of the time expended on a 2 core Opteron system. Secondly, I was astonished at just how difficult it is to monitor mutual exclusions!

Linux uses an exclusion mechanism called futexes. User space memory is used for the exclusion and the operations are fast. Unfortunately, at least with the version I was running on RHEL 5.2, you get no visibility into the user symbol tables. So your futex is displayed as a memory location. That’s pretty useless. I had to go back and modify my code to print all the relevant memory addresses just so I could see which one is which! This was a lot of work that definitely flies in the face of the non-intrusive measurement philosophy.

Overall, I’d rate the experience satisfactory, but systemtap still needs improvement.

I found a few areas where I could tweak my code and get significant performance improvements. I saw some areas where the architecture could be improved, but that’s another problem for another day.

I see rumours on the web of user space taps. It may exist already, but if it’s in the version I used, it’s unclear how to use it. A far more important task for systemtap developers in my mind would be reading program symbol tables. Knowing a futex is generating a hot spot is useless to me if I can’t find where the futex is!

Throughout this process, I’ve been amazed at how “almost” the development tools on Linux are. CodeAnalyst (oprofile) and Systemtap are both tools I need, and both come up short. As good as Linux is for writing code, I can’t fully do the work that needs to be done using the tools available. I still wind up testing on other machines.

I’d give systemtap a B, and Linux in general a C.

New version of ARM4 agent released

Well, there were a few other things I wanted to do before a new release, including finishing the SELinux release, but the current fixes were significant enough that I didn’t want to delay. Performance is improved. Stability is improved. Portability is enhanced. Bugs are fixed. Time to set it free.

It’s available now on Sourceforge or through the project web site (which links to Sourceforge). Fedora 10 will have it available shortly. The Fedora 11 build system is locked pending its imminent release, but I’ll get it done as soon as the lock is released.

I still have to get the Apache module packaged for Fedora, but that doesn’t make much sense until the SELinux stuff is taken care of.

More CMake oddities…

The story continues…

I’ve got the thing working, and am now working on packaging. This is where things get far messier than they should.

First off, the default source generators are for tar.gz, tar.bz2, and tar.Z. More than I need, but OK. Messing with it means I need to mess with it for ALL target platforms, and that seems to defeat the purpose. Where things go seriously awry is when the second generator runs, in that it also hauls in the first archive, and the third hauls in the first two. Even more seriously, they haul in their working directory with FULL copies and all temporary files. Run this more than once, and you quickly run out of disk space. And of course, CMake provides no functionality for removing these old temp files.

Yes, you can work around these. But why? This is some pretty basic stuff! My ~11000 lines of code quickly turned into an archive over 800 MB! That’s too big for a CD!

The second issue is when I try to build a RPM from a source file. This is a process that needs to be non-interactive, but that’s not how CMake is designed to run. The Fedora guidelines for create a .spec file using CMake don’t work. The primary problem is CMake just wont work if your current directory is different from the one used to make the archive, which it is pretty much guaranteed to be. Defining the appropriate variables on the command line doesn’t seem to override the cache conflicts. The workaround seems to be to create a script file to do this, but I really should be able to all that from the command line.

The more I play with CMake the more I’m convinced that it could be a good system some day, but it’s not there yet. I’ll continue to use it for small projects that might require a windows build, but there’s no way I’m ready to convert my larger packages. CMake is far from ready for that. They seem to have tackled some of the larger autoconf/automake issues, but have ignored so much basic stuff along the way.

Python language bindings for ARM now part of Fedora

They should be available within a couple of days after they work their way through the Fedora release system. The package name is python-arm4.

Now those of you familiar with the Application Response Measurement (ARM) standards will know that there are no official Python language bindings. That will probably remain true. I developed these myself and they’re not endorsed, but they have been used by some members of the CMG’s ARM Customer Group, of which I am a member. They’re true to the C language bindings while maintaining a Python feel.

You can read more about it here.

A lesson in optimization

I’m continuing to work on my workload characterization project, and today I got some results worth mentioning. By caching 5 QPP instances (one per data sample) and one string object, I was able to shave 46% off the run times. That’s both wall clock time and CPU time!

Sometimes it’s worth taking a couple of minutes to profile.

On that note, I must say I’m quite impressed with Apple’s Shark utility. It’s quite simple and reliable. Oprofile on the other hand has always disappointed. It’s always been iffy as to whether it will run at all, let alone give me results that are useful. AMD’s CodeAnalyst interface improves the look, but I can’t get it to run on RHEL 5, despite having had it run before. A source build also refuses to build, saying that libraries that do exist don’t. I’m sure I could get it figured out, but I’d only have to do it again the next time I needed it, and Shark just works. This has been a pretty consistent problem with oprofile and CodeAnalyst which is why I don’t profile more.

UPDATE:
A little more work allowed me to get a reduction of 55% on the original run time. The fully optimized time is 72% less than the debug version, even after these optimizations! Less than 2 minutes is so much better that the day it took me to run my original experimental Java version that never did work…

ARM4 daemon is now part of Fedora

Well, it’s been a long road. I’ve been working on the agent for about 5 years now, and working through the Red Hat approval process for about a year. The code’s in and committed for Fedora 10 and later, so it should start showing up as soon as it gets through the updates process.

More packages should come soon, including Python language bindings and an Apache module.

See http://www.arm4.org for more details.

Now an approved Red Hat contributor!

As of this afternoon, I’m an approved Red Hat contributor. That means I can commit new packages to the Fedora distributions, and potentially to the Enterprise extras. Once things trickle through and I get some other pending work done, I’ll be committing a package with the ARM4 daemon, hopefully followed soon after with Python languages bindings and an Apache module.

The process has been slow, but I do like their methodology. Ubuntu has a mentor based acceptance policy, which I’m OK with in principle, but I’d like to know more of what’s involved. That, and other issues with how Debian has packaged things has kept me from persuing that more, but as my work clears up a little I’ll take a stroll down that road as well.

My Linux History

I’ve used a large number of operating systems over the years. In fact I’d be hard pressed to name them all. Most no longer exist. The one I use most consistently is Linux.

My first exposure to Linux was the Slackware distribution on a 386 (no math chip!). It was a pre-1.0 kernel, but I can’t recall if it was 0.99… or 0.98… I do remember downloading it over a 300 baud packet switched ham radio link that my brother had set up. It took weeks. A new kernel took days. It all fit on my 20 MB hard drive. I can’t remember the amount of memory installed, but it was probably less than the 640K max that haunted us in those days. Ah, memories!

The control freak in me stuck with Slackware in those early days. Red Hat had all this newfangled RPM technology, but at the time integrating custom package builds was quite simply a pain. I didn’t switch to Red Hat until many years later.

So fast forward a few years and I have some machines to play with. This is when I started playing with Red Hat again. It was around the time of RH 5. At this point, Red Hat became my distribution of choice, and it remained so for many years.

Fast forward a few more years, and I got my first Opteron. The processors were out before I could get a system for them, but I bought a Tyan S2880 as soon as it was available. It still runs today even as other machines have come and failed. I’ve upgraded it a few times with memory and processors, and the addition of RAID storage, and it’s still a top of the line system.

When I first created the machine, giving it life was a challenge. 32 bit Windows ran on it, but Windows Update wouldn’t because it claimed not to recognize the processor.  This led to the machine being infected by a worm. It was a couple of weeks before Windows updated their update. Their 64 bit OS was coming “any day now” for more than a year. It didn’t actually arrive until Intel produced their 64 bit processors, and I’m one of the conspiracy theorists on this point.

So I went with Linux. At the time, SuSE was the first to market with a 64 bit OS, so I gave them a try. I had issues with drivers from the start. I’d bought this beautifully fast SATA drive, but the SATA drivers wouldn’t work at all. They then fixed it so they’d work, but I couldn’t boot from them. To be fair to the Linux community, a lot of the problem was Tyan’s poor driver support. I added an IDE drive for my boot disk and all was well. As for the OS, I was hooked. Their update mechanism was head and shoulders over Red Hat at the time. I was with them a long time.

And then came Novell. They took a wonderful update mechanism in the form of YUM, and in the process of integrating it with their own system completely and utterly mucked it up. It’s usable now, but it wasn’t for a long time. Of course, now they’re trying to integrate the worst features of the Windows interface, so I’m not likely to go back to them in the forseeable future.

In the meantime, my work environment was switching from Solaris development to Linux development, and as part of that I switched to RHEL. I still run RHEL 5 on the machine even though I’ve moved on. I did what little Windows development I needed to do in a virtual machine, and I still don’t have a native Windows machine. I don’t miss it.

So the two main issues I have with RHEL have been update and expense. I run multiple machines for testing and development, some of them very old. I can’t justify the expense of update contracts for all of them. RH definitely improved their updates with the introducton of yum, so I’m liking that. By necessity though, the packages aren’t cutting edge. Sometimes though, you just need that. So I develop in Fedora running in a virtual machine. Life is good.

In the meantime, I’d been running a hosted server for a number of smaller projects, including hosting some charity sites. Volume is low, expense is low, but supporting multiple domains made my configuration complex. The service provider gave me limited options for OS. Specifically, I could get a Fedora or SuSE distribution one release behind. This meant I was upgrading once a year. This meant, because of my complex configuration, a week of downtime and intense work to get everything up and running again! When they offered Ubuntu as an option, I had to check it out.

I installed it in a VM first to play with it. The first thing I liked was their minimal server install. More importantly, I liked their upgrade path for the LTS (Long Term Support) versions. Ubuntu was into 8.0.4, but my service provider was stuck on version 6. Ubuntu allowed me to install version 6, and immediately upgrade to version 8! Ubuntu it is! I now use Ubuntu for all my servers. I haven’t played with the desktop yet.

I’m not a total convert to Ubuntu, but that’s Debian’s fault. Specifically in how they’ve built some of their packages.  One of my projects used the Berkeley DB system, but Debian modified it so that all the utilities were renamed to include version numbers. This meant a substantial overhaul to my build scripts, and automatically obsoleted my package builds. That and their build system is poorly documented. RPMs are a whole lot easier.

So here I am. Some Red Hat, still some SuSE but that’s not likely to continue, and some Ubuntu. I’ve loved and hated them all!