Over the Bridge

In general, you'd think gauging the performance of a piece of code is easy, right?

Well, I was under that same impression. Even at work, whenever I had to figure out whether something was running faster, my frist instinct was to test it a couple times on a few input sets, and if it sort of felt right, then, well, that was good enough.

I saw fascinating talk over the weekend, "Performance Matters" by Emery Berger. (Linked Below)

Basically the gist of it is that there are a number of things which can randomize performance of native applications (to say nothing about interpreted applications), and that it's relatively hard to determine whether or not any particular change you make (beyond ones that are extremely obvious that make things an order of magnitude faster or slower) is actually affecting the software in a positive or negative way. Small gains in performance, can actually be due to things like memory layout, rather than any particular change you've made.

Essentially, Berger and his team have developed a tool that randomizes the memory layout of the compiled application (amongst other things), that allow you to perform a bunch of runs and get a reasonably accurate distribution of performance measures that will help you figure out whether or not you've actually improved your software.

One of the most imporatnt points he drives home is that, even though you now do have data that accounts for memory layout changes, you can't simply look at it, and do a simple linear comparaison between previous results, and say it's faster. He refers to this as "eyeball statistics". Basically, he makes the point that you want to a take statistical approach to this situation, and make statements like "within a 95% confidence interval, what is the probability that the speedup we're seeing here is by chance?". He does assert that the runs are normally distributed (he does say there are reasons for this, but to my memory doesn't go into them), and so thus, we can treat any run data we have as a normal distribution and perform some easy stats on it to get our answer to that question.

Anyway, by far and away the most interesting talk I've seen in the past couple years. Give it a listen, if you're interested in optimization with any sort of native application.