Peace, Love and C++

my sbuilder setup (sb)

Wed, 30 May 2012 20:00:27 -0400

I’ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly.

It’s gone from one small shell script to a set of 8, and growing. Quickly.

Until I spin these off into their own repo, they’ll be in my dot-bin

Before getting too far, please relize this is only how I’m using sbuild. There are many other (more) sane configurations that you can (and should) use.

This was not made to be useful for anyone other then me :)

I’ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot.

Firstly, forgive me for my sins. This is all a huge hack, but it’s use isn’t the worst thing in the world.

As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable.

Now — the big question — how do you use this. Rather, this is more of a how do I use sbuilder :)

Firstly, I’ve created a file for each schroot set that I’m using. Here’s my /etc/schroot/chroot.d/sid.

[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config

It’s very similar to the one that sbuilder-createchroot creates.

If you’re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf.

Next, create a dist.d directory in /var/sbuild — such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists.

The arguments for all the sb* commands is always in the order of variant, dist.

variant defaults to “generic”, which I’ve set as an alias to the generic chroot, and the dist defaults to “sid”. At some point, I’ll change the default to be the last UNRELEASED entry in the changelog.

So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable.

After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay.

The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it’s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory — this is to prevent mistakes, not malicious scripts).

By the way, for those interested, there’s a small issue with what I’m describing it’s very subtle. Basically, there’s a few assumptions made by how schroot operates, and handles “sessions”. Since it’ll keep stuff mounted if the session is still active, it makes purging all this a real … pain.

The magic behind the scripts is in the sb-endsessions routine — which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn’t have a situation where this kills off another build.

The code (which might have issues) looks like:

SESSIONS=`schroot --all-sessions -l | grep "^session:$DIST-"`
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done

By backing of schroot’s mounts, we can do our mounting / removing without worrying about klobbering schroot.

Now, there’s also the case of wanting to interactively log into a chroot. For that, there’s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user.

Eventually, I’ll be adding some scripts to upgrade all the build chroots for a dist at once.

Goofing off with sbuild "variants"

Tue, 29 May 2012 22:41:00 -0400

I’ve recently found myself in need of more then one sbuild chroot, each one set up slightly differently then the next. For a while, providing different behavior with unstable and sid was all well and good, but I figure, like any good computer scientist, I should solve the general problem.

As a result, I’ve written an sbuild wrapper to help maintain sanity. It uses a series of symbolic links to hide everything out of the way, and some bash hackery.

Basically, I’ve set up a /var/sbuild directory, full of a few different files. For each distribution, there is one folder, one symbolic link and up to two other files. There’s the dist.d file (such as /var/sbuild/sid.d) that contains all the “variants”, dist symbolic link (such as /var/sbuild/sid) which points to the currently active chroot, and up to two lock files.

The script will default to sid-default (that is to say, exactly, /var/sbuild/sid.d/sid-default - which is a symbolc link to the “default” chroot), but may take two arguments, in the order of variant, then dist.

Currently, I have two chroots, one for gcc, and one for clang (set up like Sylvestre used for cland.d.n) — but I may add some more with different setups.

Currently the behavior is to have the second wait kindly if you kick off two sbuilds at once, and fail out a third.

There are a bunch of bugs, and I’m not sure that this will work very well, but it’s working.

The next step is to combine this with unionfs overlays to help protect accidentally trashing the chroots.

You can find the script on my GitHub, pasted below.

#!/bin/bash

DIST="sid"
VARIANT="default"
SBUILD=/var/sbuild

if [ "x$1" != "x" ]; then
    VARIANT=$1
fi
if [ "x$2" != "x" ]; then
    DIST=$2
fi

if [ -e $SBUILD/$DIST.lock ]; then
    PID=`cat $SBUILD/$DIST.lock`
    echo "W: We *must* wait for the finish of PID $PID"
    echo "W: Blocking execution until this file is removed."
    if [ -e $SBUILD/$DIST.wait ]; then
        echo "E: Someone else is waiting. Bombing out."
        exit 2
    fi
    echo $$ > $SBUILD/$DIST.wait
    while [ -e $SBUILD/$DIST.lock ]; do
        sleep 1
    done
    echo "I: Dropping out of lock. Purging wait."
    rm $SBUILD/$DIST.wait
fi

echo "I: Aquiring lock"
echo $$ > $SBUILD/$DIST.lock

CURRENT=$(basename `ls -xl $SBUILD/$DIST | awk '{print $11}'`)

# We set up the name like:
#   dist-variant
# such as "sid-clang" is cool.
#echo "I: We're currently $CURRENT."

if [ "x$CURRENT" == "x$DIST-$VARIANT" ]; then
    echo "I: This matches the desired setup. No action."
else
    if [ -e $SBUILD/$DIST.d/$DIST-$VARIANT ]; then
        echo "I: Re-setting $DIST symlink."
        rm $SBUILD/$DIST
        ln -s $SBUILD/$DIST.d/$DIST-$VARIANT $SBUILD/$DIST
    else
        echo "E: No such dist! Looking in $SBUILD/$DIST.d/$DIST-$VARIANT."
        exit 1
    fi
fi

echo "I: "
echo "I: Dist:    $DIST"
echo "I: Variant: $VARIANT"
echo "I: "
# echo "I: Before we go farther, we're updating."
# sbuild-update -d $DIST --upgrade

sbuild -d $DIST

echo "I: Releasing lock"
rm $SBUILD/$DIST.lock

My first foray into non-compliant Debian packages

Tue, 29 May 2012 20:40:00 -0400

I’ve decided to start putting together packages to rebuild the bare bones install I use for most of my systems, since rebuilding everything by hand is starting to get annoying. I call it hairy candy.

Because my setup relies on some chem interesting chem hacks, none of this is fit — in any way — for the archive.

If the vim team was to catch wind of what I was doing, they’d be none too happy :)

While these packages work great for me, I can’t stress enough how badly they can break something.

Using Zack’s guide, I was able to get an archive up and running, no problem. Taking his advice, I even got signing working. Simple.

mini-dinstall rocks — a nice alternative to a PPA, for sure. At least, for one arch :)

Now, the one spin I took on Zack’s work was to add a “landing” page — a listing of what’s in the archive.

I did this using a bit of Python & Jinja2. The script follows.

#!/usr/bin/env python
# Copyright (c) Paul Tagliamonte, under the terms of the Expat license.

from jinja2 import Template
import sys

def parse_packages(fd):
    ret = []
    cur = {}
    key = None
    for line in fd.readlines():
    if line.strip() == "":
        ret.append(cur)
        cur = {}
        continue

    if line[0] == " ":
        cur[key] += "\n" + line.strip()
    else:
        key, val = line.split(":", 1)
        cur[key] = val.strip()
    return ret

pkgs = parse_packages(open(sys.argv[1], 'r'))
context = {
    "pkgs": pkgs
}

t = Template(open(sys.argv[2], 'r').read())
print t.render(**context)

Invocation looks something like:

render.py $ARCHIVE/debian/wicked/Packages $DATA/index.html

I kick this script off in the singing script, since mini-dinstall doesn’t have a exit hook.

Works great, thanks everyone!

playing with non-lintian lintian checks

Thu, 24 May 2012 20:25:03 -0400

Crazy title, I know.

Over the past $WHILE, I’ve found myself checking for the same things. Some of these are fit for inclusion into lintian it’s self — others, not so much.

Here’s a rundown on what my lintian wrapper (tragically also called “lintian” and just higher up on my $PATH. I know it’s a hack.)

Since I only use lintian on .changes files - it pre-processes for some information that is stuffed in the changes.

The first step is to get all the bugs that the package closes. It then takes these bugs, and queries the BTS to see what package they’re filed against. If the bug is filed against a different bug then the package’s source, it’ll emit a warning. It has a greylist for stuff like “WNPP”. This will fail if something’s filed against it’s binary package rather then it’s source :)

Next, the wrapper will get “other” info from the changes. It checks if the suite is in the blacklist (“UNRELEASED”, and “testing”, (stable for me as well, since I don’t usually deal with SRUs often)).

It’ll also check for the changed-by and maintainer, ensuring they’re the same. This will fail for co-maint’d packages.

If you’ve not guessed by now, this system relies on false-positives :)

Nextly, if there are .debs in the Files, it’ll go through and check the control of each. It’ll print out a stanza with the short-description inside a template, to see if the description “works”.

After that, it’ll attempt to make HTTP requests to the Homepage (and eventually URLs in the description and VCS-Browse)

If the package has “python” in the source name anywhere, or in any of it’s binary packages, it’ll trigger jwilk’s lintian4python over the source as well.

After this, it passes through to regular lintian.

Oh, and it colorizes all the output :)

Here is a (mostly lint-free) package:

As you can see, it’s very (very) verbose, but i’d rather cross stuff off a checklist then worry about something being off.

Anyway, it’s a work in progress. Not even a serious one, at that. It’s a bunch of hacks put together, so I can’t really endorse anyone else using this stuff. If you feel crazy enough to do so, all the tools are found on my dot-bin repo on GitHub

introducing... poŝto

Fri, 18 May 2012 22:50:00 -0400

Firstly, please excuse the ugly formatting of this ultra-wide image on my blog, and I’m sure the respective planets that I’m on.

poŝto is my first “real” node project, and I’m planning to get it to a point where it makes email less painful then it otherwise is. The code’s on my github, and I’d like to thank Rick Waldron for his work in helping me, as well as Scott Wells for the name.

For those wondering, poŝto is an Esperanto word meaning (according to wiktionary):

a public institution (usually government-run) to deliver mail, post
mail distribution in general
one’s mail, collectively

The front-end’s name is “poŝtilo” — the suffix denotes “tool” such that poŝtilo means something like “mail tool” — fitting!

More to come…

gcc 4.7 patchathon part V

Thu, 10 May 2012 20:16:00 -0400

Another day, s’more patches.

667343 (qterm)
672007 (pokerth)
672010 (cryptkeeper)
672052 (qastools)
672071 (pxe-kexec)
672005 (l2tp-ipsec-vpn)
672003 (dibbler)

Consider helping out!

gcc 4.7 patchathon part IV

Fri, 04 May 2012 13:41:30 -0400

S’more patches today to the grooven’ vibes of Home Movies.

667221 (kdesvn)
667200 (hexxagon)
667190 (gpsim) - not really a patch, updated existing patch
667230 (kredentials)
667233 (kvpnc)

Consider helping out :)

Transparency Camp (Day 1)

Sat, 28 Apr 2012 16:54:04 -0400

I’m at Transparency Camp, so if you’re around, feel free to find me and say hi!

I’ve been pretty much off the radar completely, so this is just to say I’m alive and doing well :)

For those who don’t know, Transparency Camp is an unconference on open government, government transparency and all that good stuff.

gcc 4.7 patchathon part III

Fri, 20 Apr 2012 22:53:25 -0400

S’more days, s’more patches:

Hacked out a few over the last few days, this time, to some good eats. I’m making a habit of this tv-hack-time thing.

patch for 667194 (gsmartcontrol)
patch for 667177 (gdisk)
patch for 667178 (gkrellkam)
patch for 667180 (gkrellmitime)
patch for 667216 (jigzo)
patch for 667217 (jstest-gtk)

Consider helping out with the gcc-4.7 ftbfss, there’s a bunch!

gcc 4.7 patchathon part II

Tue, 17 Apr 2012 23:31:12 -0400

Another day, more patches. This time, it was to the rockn’ vibes of Chopped.

Patch for:

667365 (s51dude)
667331 (prerex)
667328 (posterazor)
667313 (openmsx-debugger)
667308 (oggvideotools)

In the mean-time, I put together a script for GCC 4.7 transition bugs that are owned by the QA team with patches in the BTS.

It’s easy as cake to throw together a QA upload, so I encourage anyone out there to do some easy hacking :)

pyutf (or, as it's known on my machine - ☃�.py)

Sat, 14 Apr 2012 15:45:00 -0400

code can be found on my github

Forewarning, I can’t actually tell if that second UTF (PILE-OF-POO) actually renders, since my typeface is lame.

Anyway, introducing an idea that fermented a few days ago with some of the bopeeps ( iros, jugglinmike, tbranyen, greg) - known initially as ☃�.py (pronounced snowman-pile-of-poo-dot-py)

Here’s a code sample:

#!/usr/bin/env pyutf

lis = [ 1, 2, 3, 4, 5, 4, 3, 2, 1 ]
g = λ x: x * π
∀ a ∈ lis:
    print g(a)

Which outputs:

3.14159265359
6.28318530718
9.42477796077
12.5663706144
15.7079632679
12.5663706144
9.42477796077
6.28318530718
3.14159265359

Patches welcome, and bonus points for a py2snowmanpoo / snowmanpoo2py pull request. We’ll call it a code compressor.

Also, if someone could fix the loading the whole script at startup thing too.

Yes, this is a joke. But also, yes, it works.

gcc 4.7 patchathon

Sat, 14 Apr 2012 10:44:01 -0400

For no reason other then I had some time, I went on a patchathon against gcc-4.7 ftbfss while watching frasier

This log is really for my own use :)

Blocked 667409, 667394 and 667393 with newly filed bug 668700 in libcrypto++ (regarding providing gcc-4.7 invalid headers)
Patch sent in for 667418 (wvstreams)
Patch sent in for 667415 (wmhdplop)
Patch sent in for 667334 (pwlib)
Patch sent in for 667333 (ptlib-wolf)
Patch sent in for 667327 (populations)
Filed serious bug 668698 against cxxtools for providing a broken header
Filed serious bug 668700 against libcrypto++ for providing a broken header

GeoParse update

Wed, 11 Apr 2012 20:14:03 -0400

I’ve thrown a bit of work at the wall, and some of it stuck. I’ve been working (slowly) on GeoParse, which is a (hopefully pythonic) library to clean, pamper, separate and really dig into fairly low-quality GPS data (like, say, from Google Latitude)

Just the basics are in place now, but work’s slowly progressing. Hopefully, I should have something workable in the short-term future.

Processing Geo-location data

Wed, 04 Apr 2012 21:02:23 -0400

I’m in the middle of plotting and designing a library to handle loading, processing and interpreting Geo-location data.

I have a fairly substantial corpus of data (latitude data since 2010-ish, on and off), and a bit of motivation to hack something up.

So far, I would like to see a lib to do most (or all) of the following:

Discard static-ey reports (e.g., within a second, I’m four miles apart and back)
Translate from two lat-lon pairs to feet
“Time-Space points” & PointDeltas
Point “math”
Process “stop” data out of the corpus
Process “trip” data out of the corpus
Search against a list of “trips” to determine which “trip” it most likely matches, given a new set against the corpus

Not everything is settled, but this is on my list for 1.0, I think.

Django + Git = ?

Sat, 31 Mar 2012 10:07:59 -0400

I’d be interested in getting an instance set up of a Django powered Git project viewer (or rather, something between GitHub and viewgit)

The ideal implementation would use something like dulwich, and be super snappy.

If anyone feels like writing this, let me know. If something like this exists, please let me know even faster.

Software that doesn't suck volume 1

Sat, 24 Mar 2012 11:21:32 -0400

As many of you know, I’m highly critical of most software. Anyone that’s met me in person has most likely heard my all-too-common tirades on why software makes my life worse.

I figured, since I’m so hard, and I’ve recently found some software I feel good about using, I’d start another one of my ongoing series on software I don’t hate, or rather, software I might actually say I like.

Out of nothing but pure luck (the author followed me on twitter, so I ended up checking out his profile, finding some stuff I liked, and checked out his github), I came across Github Notifier, which works on Chrome, Firefox and Opera.

This plugin shows the octocat on the toolbar, with a count of the unread GitHub notifications.

The GitHub API (both V2 and V3) don’t have support for getting notification counts, so this plugin / extension works by scraping the notification page. I feel OK about this because the code is pretty minimal and clean.

Here’s it in use (on my machine)

Thanks, @sindresorhus!

Setting up uWSGI + nginx

Wed, 21 Mar 2012 21:35:00 -0400

I’ve spent some time recently converting from using PHP to using Django for my web applications. As a result, I’m migrating from apache2 (to be totally fair, it does have wsgi support, but I’d rather re-design using the latest fad) to nginx and uWSGI.

So, here’s some notes to myself on setting it up again.

uwsgi has an issue with —plugins python, use “python27” instead. It’s my hunch it’s because of a symlink.
Set up a static alias to link django static resources into
Invocation of uwsgi is below
Run uwsgi as “www” user, who’s home is in /srv/www
The tools I used in place of daemontools is on my github. Revisit this.

Given

SRV="/srv/www/uwsgi"
DATA="$SRV/app/$APP"
SOCK="$SRV/sock/$APP.sock"

Invocation

exec uwsgi \
    --plugins python27 \
    --chdir   $DATA/.. \
    --socket  $SOCK \
    --chmod-socket \
    -wsgi ./$APP/django_wsgi.py

I wrote some tools to daemonize this, but I have a hunch daemontools might be better.

OpenStates Hackathon

Tue, 13 Mar 2012 17:02:09 -0400

The OpenStates / GASP hackathon here at PyCon is going killer. We’ve got a bunch of brandy-new contributors hacking away, and sending in pull requests. We’re squashing bugs like crazy, and it’s going super well.

Shout-out to all the new contributors!

I'm going to PyCon!

Wed, 07 Mar 2012 16:39:33 -0500

I’ll be at PyCon with some of the Sunlight crew, and we’ll be hosting a Hackathon after the talks. I hope anyone in the area will consider hacking with us this year.

If you’re going to PyCon, send me an email, tweet or other digital message - the talks this year look awesome!

Syn rewrite

Thu, 01 Mar 2012 20:00:41 -0500

I’m thinking of rewriting Syn, but this time, with the lessons learned from Syn1, 2, and 3. While I don’t want to rewrite it (again), it’ll be a nice time to simplify the project, and give it a chance to actually function (well) as a LFS-esque package manager.

Hopefully, this can be the final rewrite.