ellipsix informatics

Switching servers

The (virtual) computer this site runs on is showing its age, so I'm switching over to a newer one within the next day or so. Just so you know in case you have any trouble accessing the site.

While I'm on the subject, kudos to Linode for having a very solid migration plan and for making continual upgrades to their hardware while lowering prices. The new server costs about a third as much as the one I'm using now.


Introducing pwait

Today I'm announcing pwait, a little program I wrote to wait for another program to finish and return its exit code. Download it from Github and read on...

Why pwait?

Because I was procrastinating one day and felt like doing some systems programming.

Seriously though. Sometimes you need to run one program after another one finishes. For example, you might be running a calculation and then you need to run the data analysis script afterwards, but only if the calculation finished successfully. The easy way to do this is

run_calculation && analyze_data

(sorry to readers who don't know UNIX shell syntax, but then again the program I'm introducing is useless to you anyway).

Which is fine if you plan for this before you run the calculation in the first place, but sometimes you already started the calculation, and it's been running for 3 hours and you don't want to stop it and lose all that progress. The easy way to do this is to hit Ctrl+Z (or some equivalent; it depends on your terminal) to suspend the calculation, and then run

fg && analyze_data

which will resume it and run the analysis script afterwards.

Which is fine if the program is actually running in a terminal where you can get to it to suspend it, and doesn't already have something else set to run after it. But what if it's not?

Or what if doing this doesn't give you enough hacker street cred?

This is where pwait comes in. You run it as

pwait <pid>

and it will wait for the process with ID <pid> to finish, intercept the exit code, and return it as pwait's own exit code. You can use this to passively observe whether a program finishes successfully or not. Or, at least, semi-passively.

Blurry animation of pwait at work

How it works

pwait uses the ptrace system call to attach itself as a tracer to the process you want to wait for. A tracer process can do all sorts of things to its tracee, including stopping and starting it, examining its memory, changing values in its memory, filtering signals that are going to be sent to it, and so on. ptrace is mainly used by debuggers. But pwait ignores most of its tracer superpowers, only watching out for one thing: the signal the process receives when it is about to exit. The exit code is contained in that signal. So pwait copies that status code and exits itself.

Using ptrace has some drawbacks. For instance, you can't have multiple tracers tracing the same process. This means you can't wait for a program you're debugging. (I can't imagine this ever really being a problem, but you never know.) You also can't wait for a single program with multiple instances of pwait. (I could imagine this being an inconvenience.)

To get around some of these issues, I added a netlink mode. netlink is a way for the Linux kernel to pass messages to and from normal (userspace) programs. Of course, there are many ways messages get passed back and forth between the kernel and userspace, but netlink is rather generic and you can get a broad spectrum of information out of it. Of particular interest to pwait is that netlink can be configured to emit a message every time a process exits. pwait can then register itself to wait for that signal. It gets notified about every process that exits on the whole system, but it'll just discard all those notifications until it finds one that matches the process ID it's looking for.

netlink is definitely on the "new and fancy" end of Linux kernel tools; in fact, I could only find one website that demonstrates the functionality I needed for pwait.

How to get it

I'm not particularly confident in this program yet, so there's no formal release. Just head to Github and click on "Download ZIP" in the lower right, or just clone the repository if you prefer. Bug reports and feedback are very welcome!


First steps toward new scicomm conferences

Join the Google group mailing list to stay informed or to help with planning!

My post last week considering options for a new science communication conference series got a pretty strong response, at least relative to most things on this blog. As it turns out, there already are some people in various stages of planning new (un)conferences in the style of Science Online, much like what I was thinking about. I won't say anything about them here because those people haven't revealed their plans yet, but I hope they will go public soon!

I also completely forgot that Science Online was not monolithic; it had regional branches around the US and around the world, which were largely separate from the main organization. At least two of them are still holding events: Science Online Leiden and Science Online DC. (There are also branches in Boston, Denver, and Vancouver, maybe others that I don't know about, but they seem to be inactive.) These smaller groups could play a big role in the future of the science communication community, since as several people have pointed out, it's a lot easier to organize events that involve fewer people. Perhaps a big international conference is too much to plan for in the next couple of years, but bringing together communicators from a couple of neighboring states? Not so hard. If there's no science communication group in your area, why not start one? If you do, it'd be an excellent thing to announce on the mailing list!

In fact, the same goes for topical conferences, like the massively successful Science Online Oceans. Again, a smaller conference is easier to organize, and it could build up over time to become something for the whole community.

Of course, there's no reason for me to only be writing about Science Online affiliates. I just do it because those are the events and groups I know, or can easily find out about. Actually, a lot of people I've heard from think that we should see any new conference, not as replacing Science Online, but as an opportunity to construct an event that the science communication community wants, from the ground up. I agree. After all, Science Online had its share of problems; the brand is somewhat tarnished, and any new events would probably do well to set themselves apart from that history.

Toward a new conference

While other people pursue their plans for new conferences, I've been musing on the seven-step "plan" (if you can call it that) I laid out in my earlier blog post. Here are some thoughts on the early steps, in light of what people have told me in the past week:

  1. Putting together a group with organizational experience: the Science Online "regulars" were no strangers to organizing events. After all, if you want to communicate with people, bringing the people to you is step 1. So the talent and the experience are out there. I've actually been in touch with several people who would be very capable of planning a new conference, once they decide it's time to go ahead and do it.
  2. Figure out what went wrong with Science Online: a lot of things. Here's a (partial) list, in fact. Here's another one. But this step is ongoing.
  3. Gauge interest: Yes, people are interested. Maybe not all the same people who used to regularly attend Science Online events, but a lot of them are interested enough that — as I mentioned above — they were talking about plans for some kind of new event even before my first blog post on the matter. The trick seems to be putting the interested attendees in touch with the interested organizers, which is what I'm trying to do right now.

The rest of the details — time, location, content, name, sponsors — is stuff for the future. For now, I think it's all about communication. So, whether you're interested in planning a conference or just want to be kept up to date on what everyone else is doing, please, join the mailing list!


Adventures in China: the toys of the trade

My boss got me a new toy today.

it's a Mac!

This is one of the perks of working for a well-funded research group, I guess. And a new research group. It's not often that you get your foot in the door right when they're buying equipment.

It's also a perk of being a phenomenologist (which is like being a theorist but sometimes we measure things we can't calculate). Unlike experimental physicists, who have to spend their budgets on all sorts of exotic lab equipment (which I'm given to understand means obscene amounts of duct tape and aluminum foil), all you need for phenomenology is a computer, pencil and paper, and a place to sit. So there's really no reason not to blow as much money as possible on nice equipment. And this is nice equipment. It's literally the best Apple computer you can buy over here, featuring a 27 inch display (oooooh) and OS X Yosemite, the newest update to the operating system.

Not that I don't have reason to complain. The system stalled twice before I even managed to finish the setup procedure.

Eternal Flame

I guess I have to start making offerings to the Apple Gods now? Or the spirit of Steve Jobs?


On compiler warnings (and off them, too)

Quick, what's wrong with this C++ program?

#include <iostream>

int test(int arg) {
    cout << arg << endl;

int main(int argc, char** argv) {

Did you guess nothing at all? Because that's what GCC says:

$ g++ -o funnyprogram funnyprogram.cpp


Pretty much every other programming language that makes you explicitly identify a function's return type will also make you actually return something from that function. C and C++ don't, and furthermore GCC doesn't even warn you that anything is wrong. This can occasionally lead to serious bugs, as I discovered today in this real-world example. I had a function that checks the name of an object and returns an enum value based on that name.

virtual const HardFactorOrder get_order() const {
    // relies on a particular convention for get_name()
    // but can be overridden for hard factors where that convention doesn't apply
    std::string name = get_name();
    if (name.compare(0, 3, "H01") == 0) {
        return MIXED;
    else if (name.compare(0, 2, "H0") == 0) {
        return LO;
    else if (name.compare(0, 2, "H1") == 0) {
        return NLO;

That was all good when all the objects involved had names conforming to the convention, but my latest batch of updates to the code involve objects with totally different names, and I forgot to override get_order(). So the default implementation above was getting used. Instead of failing with an error when none of the patterns matched, it was just not returning anything, and the variable that I set the return value to was getting assigned some random binary nonsense. Something like 2692389, where the legal values were 0, 1, and 2.

Needless to say, if GCC had complained about this from the start, I wouldn't have spent at least an hour staring at tiny text in a debugger.

tiny text in a debugger

There are over a hundred warnings that GCC can be configured to emit. Some of them are relatively useless, but most probably should be enabled if you want to save yourself a lot of debugging time. I pored through the manual and came up with the following set of warnings for myself:

-Wall -Wextra -Wformat-security -Wmissing-include-dirs -Wuninitialized
 -Wtrampolines -Wconversion -Wuseless-cast -Wjump-misses-init -Wlogical-op
 -Wstrict-prototypes -Wctor-dtor-privacy -Wold-style-cast -Wno-reorder
 -Wno-unused-parameter -Werror=delete-non-virtual-dtor -Werror=return-type

That's quite a mouthful, though a build system like Cmake can handle it easily; just copy and paste into the appropriate spot in the configuration file. For times when you invoke GCC manually, you can put those options in a file, perhaps ~/Wreally-all, and then run

g++ @/home/user/Wreally-all -o program program.cpp ...

which will include the contents of the file as if you had specified it on the command line.

Feel free to use this as a starting point for figuring out what set of warnings is most useful for your own environment.



I'm having a little too much fun with my newly-discovered ability to embed Twitter widgets. Enjoy these scientific twists on popular movie titles.


Another Mathematica bug

Math is hard.

Not for Barbie, but for Mathematica.

I ran into a weird Mathematica bug while trying to evaluate the sum

\sum_{k=1}^{\infty} \biggl[-\frac{\pi^2}{6} + \psi'(k + 1) + H_k^2\biggr]\frac{z^k}{k!}

Split this into three parts. The first one is the well-known expansion of the exponential function

-\frac{\pi^2}{6}\sum_{k=1}^{\infty} \frac{z^k}{k!} = -\frac{\pi^2}{6}(e^z - 1)

The second is not the well-known expansion of the exponential function.

\sum_{k=1}^{\infty} \psi'(k + 1)\frac{z^k}{k!} \neq \frac{\pi^2}{6}(e^z - 1)

Obviously not, in fact, since if two power series are equal, \sum_i a_n z^n = \sum_i b_n z^n, for an infinite number of points, each of their coefficients have to be equal: \forall n,\ a_n = b_n. (You can show this by taking the difference of the two sides and plugging in a bunch of different values of z.)

I guess Mathematica doesn't know that.

In[1] = Sum[PolyGamma[1, k + 1] z^k/k!, {k, 1, Infinity}]
Out[1] = 1/6(-1 + E^z)Pi^2

I had my hopes up for about two days that these two terms would cancel out, but I should have gone with my instinct that something was fishy about that result.

Apparently we have to go all the way back to version 7 to get a correct answer.

I'm still working on the rest of it.

Can you trust your calculator?

This is actually emblematic of a big problem that is on many people's minds right now because of a recent article in the Notices of the American Mathematical Society, (PDF) "The Misfortunes of a Trio of Mathematicians Using Computer Algebra Systems. Can We Trust in Them?" In this paper, the authors construct some fairly simple mathematical expressions for which Mathematica gives egregiously wrong results. (The expressions may not look simple to us, but they're polynomials with integer coefficients, pretty much the easiest mathematical objects for computers to handle.)

Forget math, it's programming that's hard!

What all this shows is that relying on the results of a computer program is dangerous without some kind of independent verification that it does what you think it does. Anyone who's ever written a program should understand that. But it's all too easy to forget; we get lulled into a false sense of security by the 90% of the time that programs do work, plus the 9% of the time that they seem to work because whatever errors they produce are buried in a giant pile of output.

I think sometimes we could all do with a reminder like this not to get too trusting.


The future of science online without Science Online

Join the Google group mailing list to stay informed or help with planning!

It's been a little more than a month since Science Online, the organization famous for its annual series of conferences (and infamous for the sexual harassment promulgated by one of its founders) announced that it was disappearing for good.

As a result of our state of insolvency, the ScienceOnline board of directors voted on Oct. 6 to proceed with a plan for dissolution, which we will implement over the coming weeks.

One unfortunate but necessary consequence of this decision is that we have to cancel the ScienceOnline Together 2015 conference scheduled for Atlanta in February. We have notified those who have already registered for ScienceOnline Together 2015 and will be fully refunding registration fees.

The decision took a lot of people by surprise, since the organization had been in the middle of planning their 2015 conference in Atlanta. But all of a sudden, with no warning, it was all over.

When the news broke, a lot of former attendees took to Twitter to share their favorite memories of past conferences.

My time at last year's conference, as a first-time participant, was pretty close to being the best three days of my life.

I'm not ready to give up on that.

I'm not the only one, either.

Science Online had some great aspects that were worth preserving. The "unconference" format of moderated discussions, in particular, did an amazing job of drawing new people into the community. And the half-hour coffee breaks between sessions, organized lunches, evening activities, really emphasize the idea that networking is the most important part of an event like this. I want future generations of science communicators to be able to experience that inclusiveness too.

A new conference?

At the moment, I don't see anyone else making serious plans for a successor to Science Online Together, so I'm going to push for this myself.

There are a few things that need to be done before a new conference series can take Science Online's place.

  1. Put together a steering committee with people who have expertise in organizing, and who are also legitimately committed to seeing the legacy of Science Online continue. Goodness knows I don't have any idea how to organize a conference, especially not from halfway around the world, but I do know it takes a lot of work from many, many people.

  2. Figure out why Science Online failed in the first place, and how to avoid making the same mistakes. Everyone seems to have their own ideas, ranging from the cost of the conference and travel, to the change in location, to the fact that many "regulars" were taking a break this year. Most people probably suspect that Bora's sexual harassment scandal and the organization's inadequate handling of it at Science Online 2014 played a role.

  3. Decide whether there is enough interest to hold a conference at all, and if so, how large it should be. Right now the dominant attitude in the community seems to be that Science Online was nice while it lasted, but now it's dead and gone and that's all there is to it. If that's really the case, then it's pointless trying to start a new conference series. But I don't think that's the case. You don't sell out registration spots in an hour year after year unless you have a really committed audience.

    • This probably involves getting a message out to all those who would have attended Science Online Together 2015, and others in the science communication community. I intend to ask the board whether they would be willing to pass along a message, or share the contact information that people made public when they registered.
  4. Decide on a time. Moving the conference from the traditional February-March time slot to another time (summer?) might improve people's ability to attend. Besides that, though, does an annual conference make sense? If the cost is prohibitive, moving to every 18 months or every two years might help keep the series strong.

    • This would also be something to ask potential Science Online attendees about.
  5. Decide on a location. The Science Online people already had Atlanta selected, but I heard some complaints about it, so is it really the best choice? Would it be better to rotate between different locations, as a lot of other conference series do?

  6. Check for interested sponsors. Some of the existing Science Online sponsors might be willing to continue funding a new conference series, but I bet that gets less likely the longer we wait to ask them. This is also a prime opportunity to bring new sponsors into the fold. I'm sure there are companies that are involved in online science communication that would love a chance to advertise themselves to an interested and influential audience.

  7. Come up with an awesome name. The obvious choice is to try to get the rights to use the Science Online trademark (it is a trademark, right?). But I think, given the organization's history, it may make more sense to come up with something new to distance the new conference series from the shortcomings of the old one.

    Actually, I've got this one covered. The Science Communication Initiative Online eXchange. It's descriptive and sounds modern and dynamic, and was definitely not just the first thing I could think of that would let us keep the #sciox hashtag.

Originally I had considered trying to take over the conference center booking, the existing event planning, and other preparation that had already been done by Science Online for the 2015 Atlanta conference. But that would have required acting very quickly, and also knowing that the full complement of 400 people would show up to the new conference. That's probably not realistic.

At this point, I'm thinking of planning a small, informal meeting in 2015, perhaps still in March, as a planning session for a full conference in 2016. It could be held on the same days as the original Science Online conference was scheduled, or it could be done in conjunction with the Atlanta Science Festival March 21-28, as discussed on the planning Facebook page.

What do you think?

I've created a Google group (just an email list) for planning the future of this conference series. If you want to help or just want to see how it's going, head to Google and sign up! You do not need a Google account, any email address will do. (Any problems, email me or Tweet at me.)


How one line in one file made me reinstall Gentoo

Hey, internet. Long time no see.

(It's often claimed "long time no see" is a literal translation from Chinese "好久不见", "hǎo jiǔ bú jiàn", but actually nobody knows for sure.)

On Thursday, my computer crashed. Not just that it crashed, but it somehow corrupted itself so that I couldn't even boot it. It survived for two seconds after being turned on, before bailing out with this error:

Linux crashes after two seconds

init[1]: segfault at 0 ip 00007ff10ea3fe05 sp 0007ffff7cb49148 error 4 in libc-2.19.so[7ff10e919000+19e000]
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

This is so early in the startup process that nothing is running. It's pretty much just the kernel and init (which, if you don't know, is the very first program to run when a Linux system starts, the one that runs all other programs).

So of course I recompiled (it's Gentoo) and reinstalled both the kernel and init, as well as libc, several times, as well as several other programs that I'm pretty sure didn't even get a chance to start, so how could they have anything to do with this crash? Still, better safe than sorry I guess.

None of it made any difference. After replacing every program that could possibly be running at the time of the error, init still crashed. This is probably the most frustrating error I've seen in 15 years as a Linux user.

So I spent my weekend reinstalling everything on my computer.

Verifying downloads before GPG

One of the easy ways a nefarious group could hack into my computer is by intercepting the source code I download when I'm installing a program and modifying it to do, well, whatever they want. China's government tries to maintain pretty strict control over the internet in their country, so I wouldn't put it past them to do this. (To be fair, I wouldn't put it past the US government either.) One way I could get around that is to download everything from a trusted server in the US or some other country, but the internet access here is kind of slow, especially going in or out of the country. It'd be a lot quicker to download all my source code within China, and I need all the time I can get.

The alternative is to use cryptography. Gentoo already does this, to some extent: whenever I install a program, the installer first checks the SHA256 hash of the source code against an internal database to make sure the source code hasn't been tampered with. But that internal database is also something I need to download, and how do I know that hasn't been tampered with?

Of course, the Gentoo Release Engineering Team (the people who maintain this internal database) have thought about this, and they cryptographically sign the database files. I'll skip the details, but through the "magic" of public-key cryptography, I can check that the SHA256 hashes I download are the same ones they upload by checking one single 40-letter key fingerprint, just once.

Here's the catch: to verify the hashes, I need to have GPG installed, and to install GPG, I need to verify the hashes of its source code and all the other code it depends on. My bright idea to get around this was to define this little function, a fake version of GPG:

echo "gpg invoked: gpg $*"
echo "please do this manually and press ok if successful"
read -n 2
if [[ "$REPLY"=="ok" ]]; then
    return 0
    return 1

That goes into /usr/local/bin/gpg in the system I'm trying to install. This way, when I run emerge-webrsync to download the hashes, and it gets to the point where it's going to check the file's signature, it will pause so I can manually check the signature.

I spent way too much time coming up with that.

About that crash

I did eventually figure out the problem. It turned out to be a line in /etc/ld.so.preload, which is a file specifying libraries of code that the system should preload for every new program it starts up. Essentially you're patching extra computer code into the program when it runs. Extra code that it probably wasn't written to deal with. That makes an environment ripe for conflicts between the preloaded code and the original code. (The fact that this feature exists is shocking, most of the time, but it does have a few legitimate uses.)

I had tried to install the Astrill VPN client, which as part of its installation adds this line to /etc/ld.so.preload:


I'm guessing liblsp.so is Astrill's way of intercepting everything that a program tries to send or receive over the internet. It might work for most programs (might; I'm not going to keep it around to find out), but clearly, it has a major conflict with init (which doesn't even need to access the internet anyway).

It wasn't easy to find the culprit, actually; I had to boot my computer using the System Rescue CD (on a USB drive, despite the name) and find all files that were modified at the exact time I installed Astrill:

find /mnt/gentoo -mmin 537 -mmax 539

This finds all files which were last modified 538&pm;1 minutes ago. There were about 50 of them, and it wasn't hard to pick out the one that could affect how the very first program on the computer starts up.

Still, I can't believe I reinstalled my entire operating system because of one line in one file... life of a Gentoo user, I guess. :-P


There are two kinds of textbooks...

...those written in binary and — wait, that's not right!

Actually, the two kinds of textbooks, at least in physics, are educational books and reference books.

  • Educational books are good for learning a subject for the first time. They're written in a way that makes it easy to read a whole section or chapter without getting overwhelmed. They have clear summaries, walk you through simple examples, and put their topic in context to keep you interested. These books are almost telling a coherent story as much as they are imparting information.

    Examples include David Griffiths' books on quantum mechanics, electromagnetism, and particle physics, Daniel Schroeder's thermodynamics book, James Hartle's book on general relativity, and Barton Zwiebach's book on string theory.

  • Reference books are good for looking up the details of a specific procedure or situation. They're densely packed with information, but that makes it very hard to read a large section at a time, the way you would read other books. Instead, a reference book is most useful when you need to know one particular fact and can look up just the part of the book that deals with that fact. In these books, the individual sections should be more self-contained because you don't want to read through the whole book to get the information you need.

    Examples include Cohen-Tannoudji, Diu, and Laloë on quantum mechanics; Jackson on electromagnetism; Misner, Thorne, and Wheeler on general relativity; and Green, Schwartz, and Witten on superstring theory.

One of the biggest mistakes a textbook author can make is failing to decide which kind of book they want to write. While most textbooks have some features of each type, no single book can adequately fill both roles. Trying to do it just makes your book bad at both.

Of course, one of the biggest mistakes a textbook reader can make is expecting an educational book to be good as a reference, or vice-versa. A lot of trained physicists do this when they complain about Griffiths' books being insufficiently rigorous, for example. Those books aren't meant for trained physicists, they're meant for undergraduates, who would just be confused by "sufficient" rigor.