After the bleak, joyless work of releasing software, plugging holes of preventing users from clicking buttons they should not have clicked in the first place, writing unctuous documentation and release notes and wrapping code into neat little installers with tiny bows and bells that gently tingle when touched, there are few things as profoundly satisfying as taking code that looked like line noise in the first place, applying some non-trivial transformation to it and still have it look like line noise, only now it’s twice as fast, or half as long or of some other inaccessible quality that can, on some technical level, be referred to as “cool”.

Quite some time ago, I decided that this time, “cool” will mean “runs on graphics hardware, possibly faster”. For this end, I took my trusty T61 to work, downloaded the latest CUDA toolkit, found out that nearly all examples crashed when compiled, installed the latest beta drivers from NVIDIA, noticed that everything worked now and started playing around with the examples.

After some time marveling at ball pit simulations and similarly telling examples, I printed out the CUDA programming guide and spent a considerable amount of time marveling at how well the NVidia green looked when printed with our institute’s color laser printer, before reading its more important parts.

Technical Details

Roughly said, modern graphics hardware from NVIDIA consists of an array of multiprocessors. Each multiprocessor has eight cores, some shared memory, a controller unit and units for transcendental functions. Kernels (methods that run on the device, as opposed to host code) are automatically distributed to the available multiprocessors (their number depending on the hardware, with as many as 120 on the Tesla cards).

Kernel invocations are organized in 3-dimensional thread blocks and 2-dimensional grids. A single thread block can have at most 512 threads (i.e. x × y × z ⩽ 512) and is always executed by a single multiprocessor. Thread blocks are organized in a grid (with at most 65,536 elements on each dimension), which are distributed to the available multiprocessors, and thus inherently scalable.

A multiprocessor divides the threads of a block into warps and all threads of a single warp are executed in parallel. Optimal performance is reached when the control flow in threads of the same warp does not diverge based on the input data, the cores are optimized to execute the same code in several threads on different data (therefore the name SIMT—Single instruction, multiple thread—for this architecture).

There is also a memory hierarchy, from registers to on-die shared memory to global device memory. Data has to be explicitly copied from the system RAM to device memory, which is costly and should be minimized.

In other words, if you have a computation that has to be executed many, many times on different input data, move it the GPU instead and use the freed CPU cycles for shuffling around the data. In other words, ZOOOOOOM.

Some Experiments

Back in the day, we had to write code for training Gaussian Mixture Models. At the core of the algorithm, an auxiliary vector containing each point in the training set has to be created for each Gaussian. The creation of the auxiliar vector includes exponentiation, which proved to be quite costly, since it accounted for 50% of the running time of the program¹.

Instead of doing comparedly boring optimization work for running that algorithm on a single processor, and even more boring work to have it run on two processors, I partially ported it to have the auxiliary vectors be computed on the GPU, with the result that by moving only this one part of the computation leads to the three-fold speed increase for the whole program.

On my machine (2 GiB RAM, 2.2 GHz Core 2 Duo, NVS 140M with 2 multiprocessors), running the C++-only version² takes 33s for 90,000 points and three Gaussians. Running the computation with the same data set on the GPU takes 11s.

How is it done

In order to invoke a kernel running on the GPU, the code has to be compiled with nvcc, NVIDIA’s compiler for CUDA. It separates device, which is compiled to some binary code, from host code which is transformed and handed over to the platform compiler. Therefore, it is needed to write some intermediate methods which can be called from “normal” C++ and that take care of shuffling data to the device and invoking the kernel methods³:

typedef float PROB;
 
#define BLOCK_SIZE 16
 
Matrix d_in;
Matrix d_out;
 
__device__ __constant__ KernelGaussian d_params[20];
 
extern "C"
void LoadPoints(Matrix in, size_t params) 
{
    d_in.height = in.height;
    d_in.width = in.width;    
    size_t size_in = in.width * in.height * sizeof(PROB);
 
    cutilSafeCall(cudaMalloc((void**) &d_in.elements, size_in));
    cutilSafeCall(cudaMemcpy(d_in.elements, in.elements, size_in, cudaMemcpyHostToDevice));
 
    d_out.width = in.width;
    d_out.height = params;
 
    size_t size_out = d_out.width * d_out.height * sizeof(PROB);
    cudaMalloc((void**) &d_out.elements, size_out);
}
 
__global__ void computeAuxVectors(Matrix in, Matrix out) 
{
    int p = blockIdx.y * BLOCK_SIZE + threadIdx.y;
    KernelGaussian g = d_params[blockIdx.x];
    PROB d = (in.elements[p] - g.mean) / g.stddev;
    out.elements[blockIdx.x * out.width + p] = g.factor * expf(-0.5*d*d);
}
extern "C"
void RunComputeAuxVectors(
        KernelGaussian* g, 
        std::vector<std::vector<PROB> >& out, 
        size_t points, size_t params)
{
    cutilSafeCall(
        cudaMemcpyToSymbol(
          d_params, g, 
          params*sizeof(KernelGaussian), 
          0, cudaMemcpyHostToDevice));
 
    dim3 dimGrid(params, points / BLOCK_SIZE);
    dim3 dimBlock(1, BLOCK_SIZE);
    computeAuxVectors<<<dimGrid, dimBlock>>>(d_in, d_out);
    cutilCheckMsg("Kernel execution failed");
 
    for(size_t i = 0; i < params; ++i) {
        cutilSafeCall(
            cudaMemcpy(
              &out[i][0], &d_out.elements[i * points], 
              sizeof(PROB) * points, 
              cudaMemcpyDeviceToHost));
    }
}

The matrix d_in holds the points, which are constant over the whole computation and therefore copied over to the device exactly once. d_out is the matrix that holds the auxiliary values computed for each point and each Gaussians. For faster retrieval, the Gaussians are stored in constant device memory (d_params), which is set to contain 20 elements at most, i.e. the algorithm is limited to learn mixture models with at most 20 Gaussians.

In the method LoadPoints, device memory is allocated for the points and the output vectors and the points are copied from host to device memory using cudaMemcpy.

The actual invocation of the kernel is done in the host method RunComputeAuxVectors, which first copies the parameters into the constant storage and invokes the kernel computeAuxVectors using the new <<>> syntax. grid holds the dimensions of the grid, which is number of gaussians× points / BLOCK_SIZE. The block size is the width of a thread block and somewhat arbitrarily chosen to be 16⁴, currently this also limits the number of points to be a multiple of 16⁵. After the blocking kernel invocation, the results are copied back into host memory.

Important:

• My card only supports single-precision floats, newer ones also have double-precision
• blockIdx and threadIdx are global variables provided by CUDA compiler
• __global__ methods are run on the device and can be called from both host and device code
• Currently, the device memory for d_in and d_out is never freed, the program simply exits

Results

Working with CUDA is definitely fun (if rewarding), the system itself is quite unforgiving though. Wrapping each invocation of a CUDA method into cutilSafeCall therefore is strongly recommended, in case of an error it will print a (not always helpful) error message and exit. Though in this case the code running on the GPU itself is quite trivial, it can be hard to figure out what is going wrong, because the GPU itself is a black box, and it’s not possible to simply print out a message⁶. It’s possible to emulate the code on the CPU, and there are other tools available which I haven’t tried out.

I’ve also started toying around with my original idea, having the constraint checks for the TIGER query evaluation run on the graphics hardware and initial results are mixed. Still, if nothing at all, this gives me justification to spent a lot of money on expensive graphics hardware for my next computer—it’s all for science, this harsh and demanding mistress.

1. There might be a smart way of getting rid of the exp() call, though I haven’t found it yet
2. compiled using g++ 4.4.0 -O3
3. looking at the SDK examples was quite helpful here
4. I didn’t try out other block sizes
5. this is a restriction of the implementation, one could simply pad the points with however many zeros as needed and compute a little more
6. or maybe it is, and I don’t know how to do it…

Apart from having mastered a small coding challenge, as it turned out to be, this also gives you greater control over the scrolling itself, like making sure that certain elements are visible, viewport panning etc.

Anyway, especially when using PyGTK, it’s a bit unclear on how to proceed. From the documentation, it somehow gets clear that it has to do with the signal set_scroll_adjustment_signal:

This signal is emitted when a widget of this class is added to a scrolling aware parent, gtk_widget_set_scroll_adjustments() handles the emission. Implementation of this signal is optional.

This is not a signal name, but a signal ID¹ that has to be set in GtkWidgetClass².

Some more documentation reading reveals that you can set this signal by using the set_set_scroll_adjustments_signal method on a widget:

class ScrollableWidget(gtk.DrawingArea):
    __gsignals__ = {
        "set-scroll-adjustments": {
            gobject.SIGNAL_RUN_LAST,
            gobject.TYPE_NONE, (gtk.Adjustment, gtk.Adjustment)),
    }
 
    def __init__(self):
        gtk.DrawingArea.__init__(self)
        self.set_set_scroll_adjustments_signal("set-scroll-adjustments")

It doesn’t really matter how you call the signal as long as it takes two arguments (the horizontal and vertical adjustment). This will make the method set_scroll_adjustments (which you can’t override from within Python) return True when it is called and signal that the widget supports scrolling.

This, however, is only half the way, because the scrollable widget still needs the adjustments handed in via said methods. It’s of course possible to connect to the signal explicitly, but there’s an even more direct way by using action signals.

Action signals are the C programmer’s idea of “generic methods”. In order to create such a signal, it has to have the flag gobject.SIGNAL_ACTION and they are directly connected to a function which is then called on each signal emission. While in C, you have to provide a function pointer, in Python you can just implement functions with a compounded magic name³ and have it called automatically. I haven’t found any documentation on that in the PyGObject or PyGTK docs, only some examples in the web:

class ScrollableWidget(gtk.DrawingArea):
    __gsignals__ = {
        "set-scroll-adjustments": {
            gobject.SIGNAL_RUN_LAST | gobject.SIGNAL_ACTION, 
            gobject.TYPE_NONE, (gtk.Adjustment, gtk.Adjustment)),
    }
 
    def __init__(self):
        gtk.DrawingArea.__init__(self)
        self.set_set_scroll_adjustments_signal("set-scroll-adjustments")
 
    def do_set_scroll_adjustments(self, h_adjustment, v_adjustment):
         # do some useful stuff here, like saving them
         ...

The method being called on emission has to start with do_, following by the signal names with hyphens replaced by underscores.

The adjustment objects can then be configured to one’s own liking to have scroll bars show up or not. However, to know when the user did some scrolling, it’s necessary to listen on some signals:

    def do_set_scroll_adjustments(self, h_adjustment, v_adjustment):
        h_adjustment.connect("value-changed", self._scroll_value_changed)
        self._hadj = h_adjustment
        v_adjustment.connect("value-changed", self._scroll_value_changed)
        self._vadj = v_adjustment

To make the scroll bar show up, modify upper, lower and page_size on the adjustments.

self._hadj.lower = 0
self._hadj.upper = 50
self._hadj.page_size = 10

This tells the scrollbar that the size of the underlying picture (upper - lower) is 50, while the visible size (page_size) is 10.

The page size obviously depends on the current size of the widget, which can be retrieved from the underlying gtk.gdk.Window:

width, height = self.window.get_size()

The current position of the scroll bar is controlled by the property value of the adjustment object and should be in the range [lower .. upper - page_size]. Whenever the property is changed, the value-changed signal is emitted, which we’ve connected to previously, and the widget can be repainted.

If you’re curious, you can also see the whole gloriousness in actual working code.

1. which you usually don’t seen when coding with PyGTK
2. ditto
3. a technique which I thoroughly dislike and should be converted to be used with decorators

The picture contains a 24h clock and shows the number of commits which were done in this hour of the day, scaled by the highest commit count (that being 2 in the afternoon).

What does all this tell us, apart from the fact that I had a free hour today? Well, I never code at 5 or 6 in the morning. For the rest of that, it’s more worthwhile to split the statistics in two parts:

08/2007 – 10/2008, 234 commits

10/2008 – now, 279 commits

Wow, I used to be cool. Hacking late in the evening. Now, it’s just a day job.

The query language in TIGERSearch is limited due to its lack of universal quantification. This restriction makes it impossible to ask simple queries like „Find sentences that do not include a certain word”. We propose an easy way to formulate such queries. We have implemented this extension to the query language in a tool that allows querying parallel treebanks including their alignment constraints. Our implementation of universal quantification relies on the view of node sets rather than single node unification. Our query tool is freely available.

This means that I’ll likely be in Berlin for the conference early October.

I haven’t really managed to do a lot of interesting things for the TreeAligner lately. The query evaluation for complex queries should be a little faster due to a smarter algorithm/data structure, we now have Undo/Redo support (shamelessly copying over all the frameworks from the Eclipse people, only with less code) and the tree renderer is a little faster.

The next step now should be the web-based query server so that interested researchers can try out the query evaluator without having to download the TreeAligner package and going through the hassle of installing GTK+ on Windows.

Surprisingly, all those toolkits were written in Python and made our C++-affine lecturer wonder if he probably should look into this language some more, apart from suggesting that we „maybe […] should get a life.”

So what does my toolkit do?

• Determinization of NFSAs to DFSAs
• Creation of DFAs from simple regular expressions
• Application of FSTs
• dot graph output for *FSAs

All in all, not very impressive, and not very hard. Still, if you like PyParsing and generator-expression-prone Python code, you might want to have look at the TinyFST code.

1
2
3
4
5
6
7

def uncurry(f):
    return lambda t: f(*t)
 
def longest_common_prefix(Sa, Sb):
    return len(list(
        takewhile(uncurry(eq),
                  izip(Sa, Sb))))

1. I bet the tenses are all wrong again.

The C++ version isn’t that much faster than the Scala version if I remember my experiments correctly (about 4x). Judging from the call graph of the C++ version, most of the time is spent in the exp function anyway, which is as fast as it gets.

The input file simply lists one float value per line, and the initial parameters for the Gaussians can be specified in the source files. Be advised that the number of Gaussians used to approximate the data needs to be known before the algorithm is run.

Topics include:

• Evolution of Java
• New abstractions in programming languages
• The functional turn
• Scala: „The next programming language™”

Generics in Java

When I started to program Java 5 professionally after some years of blissful absence from the Java world, I thought myself to be well-prepared for generics. After all, I had done metaprogramming in both C++ and Python for several years.

Of course, experience never saves you from the perils of learning. It took some time until I finally got generics, including the common misunderstandings about covariance and the like. Fortunately, in the project I was working on at the time, we were allowed to go wild and try out all new features in Java 5 at length. We were the first ones to work with the new version and also carrying out the internal training, so we really had to understand what generics were about, and why all tutorials usually contain more don’ts then dos.

When I finally understood them, I was really disappointed. The type system wasn’t generic at all! Type annotations are just some sugary coating stripped out by the compiler after the program passes the type checks. Still, generics proved to be useful from time to time. Some problems just kept coming back, and I will briefly outline them there.

The ugly cast

There is going to be an ugly cast at some point, where you (the programmer) know more about the static or runtime type of an object than the compiler. Our strategy was to isolate ugly casts in minimally small methods with @SuppressWarnings("unchecked") annotations. One prominent example is serialization:

@SuppressWarnings("unchecked")
public List<String> foo(ObjectInputStream i) throws IOException {
  return (List<String>) i.readObject();
}

Generic types + class objects

A lot of generics pain is remedied when you always hand around Class<V> objects whenever you create instances of classes with generic type arguments. This is often cumbersome, as it tends to make your APIs larger, but at least provides some kind of runtime type safety.

We used this often enough to call it a pattern, though I think we never gave it a proper name.

Sun’s Java compiler

We were in for some hard lessons when we found out that Eclipse’s Java compiler was much better at type inference than Sun’s javac. These kinds of errors were especially hard to track down, and some of them were unfixed at least up to 1.5.0 Update 12.¹

The backlash

Now, after Java generics have been in the wild for a little more than three years and presumably have seen a wider adoption, a backlash is forming. While early coverage was mostly apologetic of all the oddities that had to be introduced to keep bytecode compatibility², a lot of complaints about the (perceived) complexity of generics is heard.

Before I’m going to dive into an example, let me state the following:

• Yes, Java code does get uglier and less readable with generics,
  …but a lot of that could be addressed with typedef’s.
• Yes, generics have a lot of gotchas,
  …but most of them are due to backwards compatibility. I would have liked to hear the millions of IDE monkeys cry in horror if BC had been broken.
• Yes, generics are difficult to grasp.
  Get over it. Seriously.

Killing Wildcards

In his recent article „Simplifying Java Generics by Eliminating Wildcards”, Robert Lovatt argues that Java generics could be simplified by removing wildcards altogether and make covariant generic types the default behavior, similar to arrays.

Please note that the following code examples assume that you have read the article.

In arrays, we observe the following behavior:

List[] listArray = new List[10];
Collection[] collArray = listArray;
collArray[0] = new HashSet(); // will result in an ArrayStoreException

His argument is that this behavior could simply be adopted for generics, making this code compile:

List<List> listList = new ArrayList<List>();
List<Collection> collList = listList;
collList.add(new HashSet());

Now, suppose the compiler would accept this piece of code (which it doesn’t), how should an exception similar to ArrayStoreException be thrown? The generic types are not known at runtime, in contrast to arrays³ , since they couldn’t be added without braking backwards compatibility. The only way to ensure the type safety is to have the class object inside List and check if newly added objects have the correct type, laying the burden of type checking on the class designer. While this may be acceptable for the standard library, it is certainly not acceptable for general usage.

An example taken from Lovatt’s article to display the lacking power of generics in Java (and Scala) is:

 ListScalaStyle<Integer> iList = new ListScalaStyle<Integer>();
 ListScalaStyle<Number> nList = iList.prepend( (Number)2.0 ); // OK
 ListScalaStyle<Integer> iList2 = nList.tail(); // Error, still a Number list

This is exactly the pathological case of the ugly cast. You, the programmer, know the static type of something and expect the compiler to be able to infer it as well.

To show why this cannot work in general, I’ll use a trick I’ve found to be very helpful: adding a little bit of randomization.

public ListScalaStyle<Number> getListWithTail() {
  if(Math.random() > .5) {
    ListScalaStyle<Integer> iList = new ListScalaStyle<Integer>();
    return iList.prepend((Number)2.0);
  } else {
    ListScalaStyle<Double> iList = new ListScalaStyle<Double>();
    return iList.prepend((Number)2.0);
  }
}
// ...
// can never work
ListScalaStyle<Integer> iList2 = nList.tail(); // Error, still a Number list

This little example is of course not a total refutation – having the compiler being able to infer more type information statically might always be useful. However, it will always be limited to small pieces of code. It also forces the compiler to actually examine the bytecode of functions in order to see the flow of objects, because prepend might do something wildly different. This removes many advantages of polymorphism, a technique at the very heart of Java.

Conclusion and Outlook

With generics, Java gets more complicated. It allows programmers to make interesting abstractions, but also freely hands out all kinds of guns for shooting yourself in the foot. This is definitely a deviation from Java’s original design principles and, ironically, makes it a bit more like C++ – something which the designers tried to avoid as hard as possible.

In the upcoming articles, we will examine the question why having generics this way still might be a good idea (though for totally different reasons), why the Java designers did it in the first place, and what the generics disaster (Tim Bray) teaches us about the design and evolution of programming languages.

1. most of them are fixed in Java 6
2. see Generics gotchas for a good example
3. see the documentation of Class.getComponentType()

For those who are wondering what kind of code name Gamla Stan is: STA releases are named after Stockholms subway stations.

Align your trees while the release is still hot!