Monday, April 23, 2012

Creating A Source Tarball From A Github Commit

I won't claim this: I got the initial script from Eric Smith in the Fedora development mailing list. I then enhanced it a bit to make it more useful for other Github projects on which I work. But I definitely wanted to share this with everybody since it's pretty sweet and simple. To get a source tarball based on a specific git checkin with Github, you can use the following script:


usage() {
  printf "Usage: ${ME} COMMITHASH [username] [project]\n"
  printf "\n"
  printf "\t[username] -- Overrides the default in ~/.tarballrc\n"
  printf "\t[project]  -- Override the default in ~/.tarballrc\n"
  printf "\n"
  printf "EXAMPLE ~/.tarballrc file:\n\n"
  printf "TBUSERNAME=[your username]\n"
  printf "TBPROJECT=[my project's name]\n"
  printf "\n"

ME=$(basename ${0})
die() { printf "$@\n"; exit 1; }

if [[ "${1}" == "-h" ]]; then
  exit 0

[ -s ~/.tarballrc ] && source ~/.tarballrc


if [[ -z "${username}" ]]; then die "You must provide a username."; fi
if [[ -z "${project}" ]];  then die "You must name a project."; fi
if [[ -z "${commit}" ]];   then die "You must specify a commit."; fi


git clone ${REPO}

( cd ${project} && \
  git archive --format=tar --prefix=${project}-${commit}/ ${commit} \
) | xz - >${project}-${commit}.tar.xz

This script will take as input 1) the commit hash for the checkout, and optionally 2) the username for the Github repo and 3) the Github repo name itself. So, for example, to clone my Newt Syrup repo, you could use:

tarball 6f0056 mcpierce newt-syrup

and this would give you the code for all commits up to 30 July 2011.

The Valve Employee Handbook

I heard about a leaked employee handbook today and was intrigued, to say the least. As a professional open source developer who used to write games, I was interested to say the least. And what I have to say is:

Wow, that sounds very cool.

No management structure. Self-directed projects. A true meritocracy with anonymous peer reviews. All sorts of freedoms that sound very good on the surface.

And one thing that really rings true is the part about how working long hours is a bad thing. it's been an issue for me for years now, how some people treat working a 60+ hour week as the norm. I won't go into it here and will instead post on that subject later. Just suffice it to say that I love to hear a company say that, when long hours happen, it's a sign of a problem in the company and should not be viewed as how things ought to be.

Go give it a read. It's a nice glimpse (assuming it's for real) into the world of the gaming company that has brought us such things as the Portal and Left4Dead franchises, Half-Life and Team Fortress 2. And if you're interested in working for them, then check out their jobs page on their website.

Wednesday, April 18, 2012

Telling Ruby Where To Find C++ Headers In Native Extensions

On my current project I'm responsible for maintaining the Ruby language bindings for some C++ code. And part of what I've worked on was to write some Ruby extensions in C to overcome some blocking I/O issues. So initially I wrote a simple extconf.rb file that looked like this:

require 'mkmf'

and that was it.

Now, a few months later, we need to have some enhancements to this. And we also need to support two versions of Ruby (1.8 and 1.9). So I decided to enhance this simple extconf.rb file to ensure that certain libraries and headers were present depending on the version of Ruby being used.

The problem is that some of the header files for our project themselves include headers from the C++ standard library. And, by default, Ruby will use gcc on Linux when testing for headers. So when the following snippet executed in extconf.rb:

fail("Missing header file: exceptions.h") unless have_header("qpid/messaging/exceptions.h")

it would fail in exceptions. h includes <string> and gcc doesn't know how to load a C++ header file.

Telling Ruby To Use C++

To overcome this problem, you need to tell Ruby to use a C++ compiler rather than a C compiler when necessary. To accomplish this, I added the following to extconf.rb:

Config::CONFIG['CPP'] = "g++ -E"

Now when Ruby creates the Makefile and first checks for the headers it uses g++ instead of gcc. Also, you MUST include the "-E" commandline argument: this argument tells the compiler to stop after the preprocessor stage. That way it ONLY checks that the included header file and its header dependencies are all present.

Tuesday, April 10, 2012

Fedora Packages, Ruby Gems And Patching Sources

I'm the Fedora package maintainer for several Ruby language gems. One new package I'm preparing to release is the Ruby language bindings for the Qpid messaging framework.

One of the challenges I had to overcome was to apply some patches to the release that overcome some blocking I/O issues in the underlying codebase. We decided that the first release, for Fedora 16, would be based on our 0.16 release of Qpid, which doesn't contain the fixes I've written to provide non-blocking I/O functionality. (long story short, Fedora 16 provides Ruby 1.8, Fedora 17 introduces Ruby 1.9, and Ruby 1.9 has a better threading model). I've proposed an RPM based on our 0.16 code and needed to apply a set of patches on top of that for code that's not going to be a part of the upstream codebase for 0.16.

So the challenge was, how do I apply these patches on top of a gem when creating a the RPM? I'll need to unpack the gem, apply the patches and then rebuild the gem before continuing. Not an easy task, to say the least. But one that, with a little ingenuity, was overcome with ease. And I owe thanks to my buddy Ashcrow for help with a few issues.

Part 1: Unpacking The Gem

We have a set of seven patches that need to be applied to the base gem to provide our non-blocking I/O functionality. They are defined higher up in the spec:

Patch1: 0001-Ruby-extensions-to-use-the-non-blocking-I-O-commands.patch
Patch2: 0002-Updated-the-Rakefile-to-build-the-nonblockio-code.patch
Patch3: 0003-Modified-the-Qpid-Ruby-code-to-load-the-non-blocking.patch
Patch4: 0004-Modified-the-testing-environment-to-accomodate-non-b.patch
Patch5: 0005-Updated-the-spout-and-drain-ruby-examples.patch
Patch6: 0006-Cleaned-up-the-Ruby-bindings-documentation.patch
Patch7: 0007-More-cleanups-on-the-Ruby-documentation.patch

The first thing you need to do is open up the gem in order to apply the patches to the source code. In the spec file in the %setup section we have:

%setup -q -c -T

pushd ..
gem unpack %{SOURCE0}

pushd %{gemname}-%{version}
gem spec %{SOURCE0} -l --ruby > %{gemname}.gemspec

The spec exits the buildroot (holding an anchor so it can return) and unpacks the gem, specified by %{SOURCE0}. It then enters the subdirectory that gets created (again, holding an anchor) and generates a gemspec file based on the contents of the directory. This is necessary in order to repackage the gem once we've done applying the patches.

Part 2: Applying The Patches

This is the easiest part of the whole process, but it was also the spot where Ashcrow needed to set my head straight.

In our source tree for Qpid, the Ruby language code extists at qpid/cpp/bindings/qpid/ruby. But when you're inside of the unpacked gem, you're already at the bottom level of that tree. All of the patches that get applied, though, assume you're starting above the qpid directory.

To apply the patches, you have to tell the patch macro to ignore the first n levels of nesting in the patch files:

%patch1  -p6
%patch2  -p6
%patch3  -p6
%patch4  -p6
%patch5  -p6
%patch6  -p6
%patch7  -p6

Here we it to ignore the first 6 levels of directories when applying the patches, since they're generated with the prefix of "{a,b}/"; i.e., a/qpid/cpp/bindings/qpid/ruby/lib/qpid/encoding.rb is one example of a referenced file.

Part 3: Repackaging The Gem

One of the changes in this set of patches included deleting two source files, the aforementioned lib/qpid/encoding.rb and spec/qpid/encoding_spec.rb. When the repacking step occurs the gem command choked on the missing files. So we had to fix this by removing those entries from the list of files to be packed into the gem.

# eliminate the encoding-related entries in the gemspec
sed 's/\,\ \"spec\/qpid\/encoding_spec.rb\"//;s/\,\ \"lib\/qpid\/encoding.rb\"//' %{gemname}.gemspec > %{gemname}.gemspec-1
cp -f %{gemname}.gemspec-1 %{gemname}.gemspec

gem build %{gemname}.gemspec
cp -f %{gemname}-%{version}.gem %{SOURCE0}

The first step uses sed in order to remove references to the two files that were removed while copying the gemspec to a backup file, and then overwrite the original gemspec with the newly updated one.

The next step rebuilds the gem and then copies it back over the original sources.


If you're building the RPM locally this will REPLACE your source gem with the modified version. So you'll want to replace the source gem after each local rpmbuild session or else you'll hit complaints from patch about trying to apply changes that apparently have already been applied.

To avoid overwriting the original SOURCE0, use the following line to install the gem:

gem install --local --install-dir .%{gemdir} \
            -V \
            --force ../%{gemname}-%{version}/%{gemname}-%{version}.gem

At this point the gem installation process proceeds as normal. The patched sources are now ready to be installed when the RPM is installed.