Ben Kraft

Fun With Pickles: A Pickle Quine

Sun, 21 Mar 2021 00:00:00 +0000

Pickle is Python’s all-purpose serialization format. Most Python objects can be serialized and deserialized automatically. It’s a very convenient, albeit fragile and insecure way to store data, such as when caching.

While pickle is designed to encode Python objects, it’s a quite flexible format: basically a tiny programming language. This led me to the question: can one write a quine in pickle? That is, we want to create a (byte-)string which, if read as a pickle, deserializes to itself:

quine == pickle.loads(quine)

In this post, we’ll construct such a pickle quine, and learn a bit about the internals of pickle along the way.

The pickle virtual machine

Let’s take a look at a simple pickle: the tuple (1, b'asdf').¹

>>> pickle.dumps((1, b'asdf'), protocol=3)
b'\x80\x03K\x01C\x04asdfq\x00\x86q\x01.'

A pickle file consists of a series of instructions, each consisting of an opcode and perhaps some arguments. The opcodes are described in detail in the pickletools source. Let’s take a look at our pickle, starting from the beginnning.

The first byte of our pickle is 0x80, which is the PROTO opcode: it says the next byte, 0x03, will tell us which pickle protocol we’re using (in this case version 3).² Next is K (0x4b), or BININT1. BININT1 tells us that the next byte is a small integer, in this case 0x01.

What do we do with this 1? A pickle decoder keeps its work in a stack: when we read an opcode like BININT1, we push its value onto the stack; other opcodes pop data off of the stack. At the end of the processing, the value on the top of the stack is the value we’ll return.

We can now continue on to the next opcode, C (0x43), which is SHORT_BINBYTES. As the name implies, this encodes a bytestring. The argument is a single byte 0x04, which tells us the string is the next four bytes (asdf).

Next, we get a BINPUT opcode (q or 0x71, with argument 0x00). This doesn’t affect us just yet, so we’ll revisit it when we need it.

Now our stack has two values, b'asdf' on top and 1 below it. Our next opcode is 0x86, TUPLE2, which says to take the top two values on the stack, and make them into a tuple, pushing it onto the stack. (There are other opcodes for 0-, 1-, and 3-tuples; and a pair of opcodes MARK and TUPLE that let us build arbitrary-length tuples, but we won’t need any of those today.) Then we have another BINPUT with argument 0x01, which we’ll again skip for now. Finally, we see . (0x2e) which is STOP: we’re done, and we’ll return the top value of the stack, (1, b'asdf').

The pickletools module will save us having to remember all those opcode numbers:

>>> pickletools.dis(pickle.dumps((1, b'asdf'), protocol=3))
\x80 PROTO      3
K    BININT1    1
C    SHORT_BINBYTES b'asdf'
q    BINPUT     0
\x86 TUPLE2
q    BINPUT     1
.    STOP

The layout of this dump is similar to a disassembly: and indeed we can think of a pickle decoder as a simple virtual machine, which reads and executes instructions from the pickle file.

A few more opcodes

Now that we know how a pickle decoder works, we can write the quine. We’ll need to know about a few more opcodes to do so.

The first is super simple: NONE (N or 0x4e) pushes the value None onto the stack.

The next opcode, GLOBAL (c or 0x63), is key to the pickle format: it says “look up the value from this module with this name”. This is how pickle encodes top-level functions and classes. The way it formats its argument is a bit of a relic of pickle’s original (version 0) text-based format: it has two arguments, each of which we read by looking up to the next newline. We can see an example by pickling the list builtin:

>>> pickle.dumps(list, protocol=3)
b'\x80\x03cbuiltins\nlist\nq\x00.'

After the version header, we have a GLOBAL opcode (c), then builtins\nlist\n: so we look up builtins.list and (again ignoring BINPUT) return it.

Third, we’ll need REDUCE. REDUCE is used to construct classes³ and in a few other cases. It requires two things to be on the stack: on top, a tuple of arguments, and below it, a callable; it calls callable(*arguments). This, plus TUPLE2 lets us call a function with two arguments: we push the function onto the stack, followed by the two arguments, then call TUPLE2 to make the arguments into a pair, and REDUCE to call them.

Finally, it’s time to revisit BINPUT. The pickletools module tells us that this will “store the stack top into the memo”. Up until now, we’ve been storing all our work on the stack; the memo is pickle’s other data structure. It’s just a big map of integer to object; BINPUT inserts into that map, using its argument as the key and the top of the stack as the value. Its friend BINGET does the reverse: it uses its argument as a key, and pushes the value at that key onto the stack. The memo is useful as an optimization – BINGET n lets us re-use the object that was on top of the stack when BINPUT n was called – and also allows us to pickle recursive objects.⁴ As a simplification, the pickler adds most of the objects it creates to the memo, without knowing if they’ll actually be used, which is why the above example could simply ignore the BINPUTs.

As an aside: you now know all you need to write a Very Dangerous pickle that will execute arbitrary code; try it for yourself! This is a great reminder to never depickle untrusted data.

The quine

Our basic strategy will be a typical one for writing quines:

Write code to define a string containing the entire program except for the string itself.
Write code that takes that string, splices it into itself at the right point, and returns it.

Let’s walk through the pickle. We’ll build it up as a series of bytestrings (opcodes and their arguments) which we’ll join together at the end.

First, the version header:

    PROTO, b'\x03',

Now, we push the string; this will have some placeholders we’ll figure out later.

    SHORT_BINBYTES, <1-byte length of the string>, <the string>,

Now, we have the string itself on the stack; we have to to construct the output we want, which is splicing that string into itself at index 4 (the number of bytes before where the string appears in the output pickle). In Python, this is pretty simple: we just want

return (
    the_string[:4]      # the part before where we splice
    + the_string        # the string itself
    + the_string[4:])   # the rest of the string

We just need to encode this in pickle. For this, we’ll need to know how to write everything in terms of builtin functions: pickle has no PLUS opcode. The operator module comes to the rescue: we can use operator.add(a, b) for a + b and operator.getitem(c, i) for c[i].

We do need one more trick here: we need to know how Python represents that :4 and 4:. The answer is the little-known builtin slice: the syntax c[a:b] is just syntactic sugar for c[slice(a, b)]. (Omitted values are passed as None.) So we can rewrite our code as

return (
    operator.add(
        operator.add(
            # the_string[:4]
            operator.getitem(the_string, slice(None, 4)),
            the_string),
        # the_string[4:]
        operator.getitem(the_string, slice(4, None))))

This will be easier to represent. Back to constructing our pickle, we store the string in the memo (slot 0), since we’ll need it a few times.

    BINPUT, b'\x00',

Now, we load up each of the builtins we’ll need, putting them in the memo dict.

    GLOBAL, b'builtins\nslice\n',     # push slice builtin
    BINPUT, b'\x01',                  # store it in memo

    GLOBAL, b'operator\ngetitem\n',   # push operator.getitem
    BINPUT, b'\x02',                  # store it in memo

    GLOBAL, b'operator\nadd\n',       # push operator.add
    BINPUT, b'\x03',                  # store it in memo

Now, we start computing what we need. In each case, we’re going to follow the pattern we discussed above to call a function with two arguments, and we’ll put the result in the memo.

    <code to push function>,
    <code to push first argument>,
    <code to push second argument>,
    TUPLE2, REDUCE,
    BINPUT, <memo slot number>

First, we build each slice:

    BINGET, b'\x01',                  # load slice
    NONE,                             # push None
    BININT1, b'\x04',                 # push 4
    TUPLE2, REDUCE,                   # call --> slice(None, 4)
    BINPUT, b'\x04',                  # store that in memo

    BINGET, b'\x01',                  # load slice again
    BININT1, b'\x04',                 # push 4
    NONE,                             # push None
    TUPLE2, REDUCE,                   # call --> slice(4, None)
    BINPUT, b'\x05',                  # store that in memo

Next, we use those slices to do the getitem calls:

    BINGET, b'\x02',                  # load getitem
    BINGET, b'\x00',                  # load the string
    BINGET, b'\x04',                  # load slice(None, 4)
    TUPLE2, REDUCE,                   # call --> string[:4]
    BINPUT, b'\x06',                  # store that in memo

    BINGET, b'\x02',                  # load getitem again
    BINGET, b'\x00',                  # load the string again
    BINGET, b'\x05',                  # load slice(4, None)
    TUPLE2, REDUCE,                   # call --> string[4:]
    BINPUT, b'\x07',                  # store that in memo

Finally, we call add to glue everything together:

    BINGET, b'\x03',                  # load operator.add
    BINGET, b'\x06',                  # load string[:4]
    BINGET, b'\x00',                  # load string
    TUPLE2, REDUCE,                   # call
    BINPUT, b'\x08',                  # store that in memo

    BINGET, b'\x03',                  # load operator.add again
    BINGET, b'\x08',                  # load string[:4] + string
    BINGET, b'\x07',                  # load string[4:]
    TUPLE2, REDUCE,                   # call

We’ve left the result on top of the stack, so a simple

    STOP

will finish things up.

Except we’re not quite done: we have to fill in those placeholders. The length will be the total number of opcodes we’ve printing above (counting itself, not counting the placeholder for the string), which in this case is 117. And then we just have to substitute in the string. This gives us the pickle:

quine = b'\x80\x03Cu\x80\x03Cuq\x00cbuiltins\nslice\nq\x01coperator\ngetitem\nq\x02coperator\nadd\nq\x03h\x01NK\x04\x86Rq\x04h\x01K\x04N\x86Rq\x05h\x02h\x00h\x04\x86Rq\x06h\x02h\x00h\x05\x86Rq\x07h\x03h\x06h\x00\x86Rq\x08h\x03h\x08h\x07\x86R.q\x00cbuiltins\nslice\nq\x01coperator\ngetitem\nq\x02coperator\nadd\nq\x03h\x01NK\x04\x86Rq\x04h\x01K\x04N\x86Rq\x05h\x02h\x00h\x04\x86Rq\x06h\x02h\x00h\x05\x86Rq\x07h\x03h\x06h\x00\x86Rq\x08h\x03h\x08h\x07\x86R.'

or run through pickletools.dis:

\x80 PROTO      3
C    SHORT_BINBYTES b'\x80\x03Cuq\x00cbuiltins\nslice\nq\x01coperator\ngetitem\nq\x02coperator\nadd\nq\x03h\x01NK\x04\x86Rq\x04h\x01K\x04N\x86Rq\x05h\x02h\x00h\x04\x86Rq\x06h\x02h\x00h\x05\x86Rq\x07h\x03h\x06h\x00\x86Rq\x08h\x03h\x08h\x07\x86R.'
q    BINPUT     0
c    GLOBAL     'builtins slice'
q    BINPUT     1
c    GLOBAL     'operator getitem'
q    BINPUT     2
c    GLOBAL     'operator add'
q    BINPUT     3
h    BINGET     1
N    NONE
K    BININT1    4
\x86 TUPLE2
R    REDUCE
q    BINPUT     4
h    BINGET     1
K    BININT1    4
N    NONE
\x86 TUPLE2
R    REDUCE
q    BINPUT     5
h    BINGET     2
h    BINGET     0
h    BINGET     4
\x86 TUPLE2
R    REDUCE
q    BINPUT     6
h    BINGET     2
h    BINGET     0
h    BINGET     5
\x86 TUPLE2
R    REDUCE
q    BINPUT     7
h    BINGET     3
h    BINGET     6
h    BINGET     0
\x86 TUPLE2
R    REDUCE
q    BINPUT     8
h    BINGET     3
h    BINGET     8
h    BINGET     7
\x86 TUPLE2
R    REDUCE
.    STOP

And that, truly, is it!

assert quine == pickle.loads(quine)

If you want to play with this further, I’ve posted the complete code to generate and check the pickle, as well as a version optimized to be as short as possible (suggestions welcome).

Thanks to Benjamin Tidor for comments on an earlier draft.

Throughout this post we’ll be using bytestrings, because that’s what we need for the quine. Strings work roughly the same way. ↩
We’re using version 3 because it’s a little simpler than version 4; handling framing wouldn’t complicate things much, but would add one more thing to explain. ↩
There are a few different ways to construct classes; in modern pickle versions REDUCE isn’t as common, but it’s still used in certain cases, and it’s the simplest and most flexible. For the really curious, search for REDUCE or read save_reduce in the pickle source. ↩
Exercise to the reader: figure out why a memo-dict is so important to a recursive object. Hint: think about what happens when you pickle the the value produced by d = {}; d['self'] = d. ↩

Go Reflection in 10 Minutes

Wed, 02 Oct 2019 00:00:00 +0000

This is a quick explanation of the reflect package in Go, originally written for some of my coworkers, because I find the official introduction a bit convoluted. This post assumes that you already know the basics of Go, including the usage of interface{}, and understand why you might want reflection. And a reminder: anything you can do without reflect is probably better done that way; and anything you can’t might be better not done at all.

Values and Types

There are two main types in reflect: Value and Type. These are fairly self-explanatory: they’re reflect’s representation of Go’s values and types, and we can create them by calling reflect.ValueOf and reflect.TypeOf on any Go value. Their methods are the bulk of the reflect package. For example:

s := "I'm a string!"
sValue := reflect.ValueOf(s)
sValue.Len() // 13

stringType := reflect.TypeOf(s) // or sValue.Type()
stringType.Name() // "string"

type myStruct struct{
    foo int
    bar bool
}

reflect.ValueOf(myStruct{123, false}).Field(0) // reflect.ValueOf(123)

Most of the reflect package is methods on Type and Value: for example for a map type one can get the key and value types; and for a map value one can ask for the keys as a slice of reflect.Value. (The documentation, of course, has the details.)

To convert a reflect.Value back to an ordinary Go value, one uses value.Interface(); this returns an interface{} which the caller likely must cast to the expected type. (reflect provides helpers like value.String() to do this for common types.)

Kinds

The last important type in reflect is Kind, which is used to tell which form of builtin a particular type is. (It’s unrelated to the type-theoretic notion of “kind”.) For example, a Kind tells us if we have a map or a func or an int, but not what the keys and values of the map are. It’s probably best explained by listing the enum values:

    Invalid
    Bool
    Int
    Int8
    Int16
    Int32
    Int64
    Uint
    Uint8
    Uint16
    Uint32
    Uint64
    Uintptr
    Float32
    Float64
    Complex64
    Complex128
    Array
    Chan
    Func
    Interface
    Map
    Ptr
    Slice
    String
    Struct
    UnsafePointer

Every reflect.Type fits into one of these, which we can check by type.Kind() or value.Kind().

Note that almost everything in reflect will panic if given invalid data. For example, reflect.Value(123).Field(0) (getting a field on an int) will panic.

What can it do?

In general, reflect is designed so that you can with reflect exactly that which you can do in ordinary Go, just more generally. (There are a few exceptions, like defining methods.) For example, it can:

index into a slice or map (of arbitrary element type) using reflect.Value.Index or reflect.Value.MapIndex
call a function (of arbitrary signature) using reflect.Value.Call
create a channel (of arbitrary element type) using reflect.MakeChan

Values which can be set in ordinary Go can also be set with reflect; for example:

s := []int{1, 2, 3}
v := reflect.ValueOf(s)
v.Index(1).Set(reflect.ValueOf(4))  // sets s[1] = 4

But values which are normally unsettable still are with reflect:

v := reflect.ValueOf(3)
v.Set(reflect.ValueOf(4))  // panics: can't set a literal

So you can’t use reflect to escape ordinary memory safety guarantees, for example, and access to unexported struct fields is limited.

Interfaces

Interface types in reflect are a bit weird. Basically, for most purposes, reflect never looks at the interface, only at the underlying value, because all its constructors accept interface{}. For example:

type Stringer interface{ String() string }
type Foo struct{}
func (foo Foo) String() string { return "it's a foo!" }

var v Stringer
v = Foo{}
reflect.TypeOf(v) // Foo, not Stringer

It’s possible to get interface values in reflect, such as by looking at the element type of a []Stringer, but it’s uncommon.

That’s it!

I hope this makes reflect seem a bit less scary. While the code you write with it may be very abstract, reflect itself isn’t too complicated. Use with care!

Advent of Code in 25 Languages

Tue, 26 Dec 2017 00:00:00 +0000

This year, I did Advent of Code’s 25 daily Christmas-themed programming puzzles, in a different language every day. It was a lot of fun! You can see my solutions on GitHub, or read on for (less spoilerful) thoughts on the languages I used.

Without further ado, the languages:

Bash
J
C
Logo (UCBLogo)
C++
Java
Perl 5
Sed
Vimscript
Kotlin
(Google) Spreadsheet formulas
Scala
Clojure
OCaml
Factor
R
Crystal
PHP
Go
Rust
Haskell
Ruby
Pencil and paper and a bit of Mathematica
JavaScript (ES6ish)
Python 3

General thoughts

Trying out a lot of languages quickly was a lot of fun! I didn’t really learn the languages I didn’t already know, but I at least got a quick feel for most of them. And it reminded me of just how much fun using different tools can be – I had a blast with Clojure despite little desire to use it for large, serious projects.

Seeing so many languages so fast was a great reminder of just what is out there beyond the languages I use or even would seriously consider using for work. And those differences aren’t just between the typical categories of functional and imperative, or higher and lower level. In some of those cases you can see the flow of ideas – from Python to Go, for example. And some ideas have come in over time – the default semantics around imports and namespacing seems to be a great way to date a language, in that Clojure has more in common with Python than with almost any other functional language.

The role of language design in how a language feels was also clear. I found the cleanly designed languages – Clojure, Go, and Python most clearly, and to a lesser extent Kotlin, Ruby/Crystal, and Rust – a lot easier and more fun to use than their less consistent counterparts like OCaml, R, and PHP. Just having consistent APIs, whether formally as interfaces/typeclasses/overloading or just simply as consistently laid-out modules, is a huge first step.

I don’t think I’ll do it again – suggestions welcome for next year’s Fun Constraint – but it was definitely a lot of fun.

Sed

Ok, let’s get it out of the way. I did day 8 using only sed. Not shell scripting and grep and other utils – that was day 1 – but rather a single sed script, run with #!/bin/sed -nf which output the answers.

The first thing to know is that yes, sed is easily Turing-complete: it has GOTO¹, if², and of course s³, and you’re off to the races. There’s a nice blog post with a more formal proof, and several BF interpreters for a more hands-on proof.

But as anyone who’s tried to write a turing machine by hand knows, Turing-complete doesn’t mean practical. The first problem, in this case, is that sed doesn’t have any arithmetic built in. I ended up working in unary, using tally marks, so |||| is $4$, and ----- is $-5$. (Converting to and from the decimal inputs is left as an exercise to the reader, or you can find it at the start and end of my solution.) In hindsight, I think the slightly greater effort involved in implementing (much faster) decimal arithmetic might have been worthwhile, but that will have to wait for a future project.

The second problem is that it’s pretty slow – you have only two string buffers (the pattern space and the hold space) and nearly anything you do will scan one of them.⁴ When you’re representing a few dozen 4-digit numbers in unary, this gets slow fast. The first two problems I tried to do were easier to implement, but just too slow to run. Day 5 required jumping a certain number of lines ahead or behind, which I had to do by moving my marker one line at a time, scanning the buffer each time. Day 6 required checking if the newly added line was unique, and $O\!\left(n^2\right)$ scans of the (length $n$) buffer were just too slow. (It’s possible by keeping the history sorted I could have done it; I didn’t try.) Finally, Day 8, with a somewhat smaller working set and a totally linear flow, worked; it took half an hour or so to run but that was okay with me.

I could probably write a full blog post on some of the tricks involved; you’ll just have to look at the source. Plus, Bruce Barnett’s Sed Grymoire, which I used as a reference, gives a much better explanation of the language as a whole than I ever could.

The “weird” languages

I used several other esoteric or special-purpose languages, which were a lot of fun.

Day 2 was in J – a whole 50 characters not counting input and comments. I’ve used J for some Project Euler problems before. It has a much deserved reputation for inscrutability due to the 1- and 2-character identifiers, but once you get beyond that, it’s actually a pretty interesting language: it’s array-focused (so map is often implicit: 1 + 1 2 3 evaluates to 2 3 4), and has an unusual grammar where higher-order functions are “adverbs” that modify other functions (“verbs”). I’d love to see a less terse language in this style; the only similar ones I’ve used are Mathematica and R, and they still have normal grammars. Anyway, J is just too hard to read for general use but it’s worth playing around with.

I went back to 4th grade a bit with Logo (yeah, the one with the turtle) on Day 4. It’s very primitive – input reading was a huge pain – and has some strange syntax, but in some ways it’s surprisingly modern; you can see the Lisp influence really clearly. The builtins and syntax are just too primitive for serious work, though.

As a Vim user I felt obligated to give Vimscript a try for Day 9. I expected it to be another painful special-purpose language missing normal functionality, but it turns out it’s actually a very reasonable scripting language, in the general flavor of Python or Ruby, albeit with a couple of quirks. This was actually one of the simplest solutions I wrote.

Spreadsheet formulas, like sed, took me a few tries because I had to find a suitable problem to avoid crashing my spreadsheet program, but Day 11 was perfect for it: the number of steps was manageable, and the logic was simple once you figured out how to represent the hex grid.

Knowing Benjamin Pollack meant I had to give Factor a try on Day 15. Since the problem involved a fairly linear iteration, I had no trouble handling the stack. The number of different shuffle words and combinators involved in the average script is a bit much, though – it seems like there are some conventions that would help but I haven’t really caught on to those yet. I almost think I’d love to embed it in a more normal language, so you can use traditional syntax to more conveniently express the overall control flow, but have the stack syntax to write the computations, or something.

And finally, just when I was a little worried I was coming up short a language or two, Day 23 was a great problem to do mostly by hand. I was a little miffed that I did have to resort to real code to compute the final result, but Mathematica did that just fine.

C, C++, Go, and Rust

I don’t do much lower-level programming, so languages like C, C++, Go, and Rust were a mostly new adventure for me. Luckily, the problem I chose to do in C was simple enough that I didn’t really have to think about pointers; in C++ I had to pass a few things around but didn’t really have to worry about managing memory carefully.

I found both Go and Rust really interesting; I hear them get compared a lot, but they take very different approaches and feel very different for it. I used Go first; it felt, to me, like a slightly lower-level Python. I didn’t use all the language features but I had no trouble wrapping my head around the basics quickly; I feel like I could be writing good quality real code in Go within a week. The explicit error-checking gets a little annoying, but having automatic memory management and modern build tools, even without many other modern features, was enough to make it feel plenty usable while remaining a super simple language. Meanwhile, Rust was the opposite: it felt very cutting-edge, but I had a lot of trouble wrapping my head around it. I didn’t really make sense of all the different behaviors of the borrow checker, and the differences between different types of references, and so on. Once you get past those, on the other hand, it’s a very full-featured functional language that just happens to give you very low-level capabilities. Anyway, the two of those make me excited for the future of programming languages.

What I didn’t like

Lastly, a few languages I just didn’t like.

I found Perl the hardest to understand of any of the general-purpose languages I tried. There’s so much magic: the $ vs. @ vs. % for variables (and the casts those do), for example. It felt like someone took PHP – which I actually found surprisingly reasonable, if a bit antiquated – and tried to make it much more difficult to understand without adding much useful functionality (as far as I saw). So, it keepts its place in string-processing one-liners, but I think I’ll leave it there.

Scala I managed to get in dependency hell without installing any dependencies – it didn’t like my JDK 9, and so I installed JDK 8 and it still didn’t like that, so I got a standalone install of the sbt build toolchain, but I couldn’t really figure how to get that to pass stdin to my executable, so I ended up with a somewhat messy and inconsistent workflow. I’m sure it’s a fine language once you install it correctly, although from my limited use it didn’t feel as nice as Kotlin.

OCaml I found nearly unusable. The standard library is really simple, and apparently everybody uses one of several alternative ones, which I didn’t realize until I was done. But even beyond the weak standard library, it really felt like it’s been superseded by more modern functional languages like Haskell.

The only two languages I seriously tried at and didn’t end up using were assembly and Erlang. Assembly I gave up on as soon as I realized I would need a nontrivial data structure for the problem. The combination of strange syntax and different programming model of Erlang were just too much. I had used Erlang before and found it fine, but by the time I finished parsing the input I wasn’t really excited to continue, and I ended up doing the problem in Ruby.

b label jumps to an earlier :label. ↩
/foo/ { ... } executes the commands in the braces only if the regex foo matches. ↩
s/foo/bar/g replaces all the occurrences of the regex foo with bar. Of course, with backreferences, this is much more powerful than your theory might lead you to expect. ↩
An s without the g or a /pattern/ that successfully matches will only scan as far as necessary, but that’s not much better in many cases. ↩

How I Teach Gerrymandering, v2

Sat, 17 Dec 2016 00:00:00 +0000

I wrote last year about how I’ve taught gerrymandering to students at Splash, and why. It’s been a year, I’ve made a number of updates to the class, and gerrymandering is back in the news, so it seems time for some updates. Below, find the changes I’ve made, editable versions of the maps to use yourself, and what’s next. Most of this will make more sense if you’ve read my previous post.

Changes I’ve made

As you’ll see below, the most obvious change I’ve made is to use smaller maps. It’s a little harder to fit exactly the phenomena I want into them, but it means students spend a lot less time counting up districts and hopefully some more time thinking about the underlying ideas. Using the same map (even with the same dots on it) for all the variations also removes another little bit of cognitive overhead – it’s always the same 30 precincts, 6 districts, and same geographic pattern, like so:

In the time I save from the smaller maps, I’ve added a map where the students try to draw the fairest maps they can. (It’s actually the same as the first map where they’re drawing a biased map.) This has had interesting results – usually some people go for all competitive districts, while others don’t pay as much attention to that and think about other factors.

When we discuss real maps at the end, I’ve made a stronger attempt to tie them directly to themes we’ve discussed, and pick maps that are apropos or that I can talk a bit more about. So I point to majority-minority districts in North Carolina, districts in Florida that figured in court battles there, and of course Arizona and its independent commission. Since I see this as one of the most important parts of the class there’s still a lot more I want to do here.

Maps to use

I’ve received several requests from friends to use the maps themselves, so I figured I’d put them up on GitHub. They’ll make the most sense if you’ve seen me teach the class, but I’ve also included my own notes from teaching as a reference, and some pointers in the README. You can download the entire set here or see the source on GitHub.

What’s next

This class is still a work in progress, of course. I’d like to figure out how to get students to think a bit more about the meaning of representation in the first two maps, and try to seed more discussion there, rather than just having them draw some districts and sometimes not have too much to say about them other than “well I put the city together”. I’d like to continue to better tie in real maps to the themes in the class. And I’m considering extending the class, so we can actually spend time on factors like race and incumbency, and talk more at the end about real-world implications and possible reforms and add some more discussion digging into metrics like seats/votes curves and compactness.

I’d also like to build a web version of the maps. I’m hoping that technology could save even more of the time we spend counting, while allowing students to look at a lot more fun metrics, computed automatically. And of course, then I might be able to share a bit more of the class with people who aren’t Splash students! Stay tuned.

The Federalist Papers

Sun, 26 Jun 2016 00:00:00 +0000

So I read the Federalist Papers.¹

It was an interesting exercise! While they aren’t the easiest reading, they’re still plenty comprehensible. Even if you don’t want to read the entire set (perhaps just a few at a time over several months as I did), the first fourteen are some of the best and give the flavor of the whole.

In this election cycle it’s oddly comforting to know that name-calling and intention-questioning isn’t new in our politics; Hamilton especially spills impressively much ink attacking the character of his opponents. From No. 67 (“The Executive Department”) we have

Here the writers against the Constitution seem to have taken pains to signalize their talent of misrepresentation. Calculating upon the aversion of the people to monarchy, they have endeavored to enlist all their jealousies and apprehensions in opposition to the intended President of the United States.

Hamilton would have us believe that the opponents of the constitution were reasonable people being corrupted and confused by a few malicious actors bent on destroying America. If only Trump were so grandiloquent.

The low blows aren’t the only parts of the Federalist Papers still apropos today. The discussion of the interactions between the federal and state governments in No. 46 (“The Influence of the State and Federal Governments Compared”) at times seems very dated, but sections like

On the other hand, should an unwarrantable measure of the federal government be unpopular in particular States, […] the means of opposition to it are powerful and at hand. The disquietude of the people; their repugnance and, perhaps, refusal to co-operate with the officers of the Union; […] would form, in a large State, very serious impediments

sound familiar as the federal government spars with the states over the legalization of marijuana. Some things, on the other hand, have gone in a totally different direction over the last hundred years: No. 68 (“The Mode of Electing the President”) defends the electoral college as if it were the most obvious and normal thing in the world, and No. 76 (“The Appointing Power of the Executive”) contains the choice quote

It is also not very probable that [the President’s] nomination would often be overruled [by the Senate].

The most interesting thing I got out of this project was the idealism the authors had for their new nation. In between spending pages discussing the meaning of a clause and the problems in the Articles of Confederation, Hamilton, Jay, and most eloquently Madison can’t help but wax poetic on their optimism for this American experiment. Hamilton opens No. 1 (“General Introduction”) by laying the stakes, as he sees them, on the table:

It has been frequently remarked that it seems to have been reserved to the people of this country, by their conduct and example, to decide the important question, whether societies of men are really capable or not of establishing good government from reflection and choice, or whether they are forever destined to depend for their political constitutions on accident and force.

A bit Amerocentric, sure, but it makes us step back a moment. The authors really believe that their experiment is a new one, with no equal in history to date, and that what they are doing will affect the course of history. The example from which they draw the most is, of all things, the British government and constitution, the exact structure against which they rebelled; they also mention many other examples, none quite the same as what they seek to build. Sometimes one gets the impression they thought their document would be lucky to last few decades – not unreasonable given that its predecessor lasted just 8 years. It’s not clear every clause was considered quite as carefully as we might like to believe; certainly the founders didn’t know as much as we do about political science. Yet here we are today, with barely an amendment each decade, let alone a full rewrite.

And so some of these monologues are just a joy to read. The last paragraph of No. 14 (“Objections to the Proposed Constitution From Extent of Territory Answered”) is as much a paean to the United States as it is an answer to an objection to the Constitution:

Is it not the glory of the people of America, that, whilst they have paid a decent regard to the opinions of former times and other nations, they have not suffered a blind veneration for antiquity, for custom, or for names, to overrule the suggestions of their own good sense, the knowledge of their own situation, and the lessons of their own experience? […] Had no important step been taken by the leaders of the Revolution for which a precedent could not be discovered, […] the people of the United States […] must at best have been laboring under the weight of some of those forms which have crushed the liberties of the rest of mankind. Happily for America, happily, we trust, for the whole human race, they pursued a new and more noble course. They accomplished a revolution which has no parallel in the annals of human society. They reared the fabrics of governments which have no model on the face of the globe. They formed the design of a great Confederacy, which it is incumbent on their successors to improve and perpetuate.

Here’s to improving and perpetuating it.

This all started from some iteration of listening to the Hamilton soundtrack. ↩

What Did I Learn as an Undergrad?

Fri, 03 Jun 2016 00:00:00 +0000

At some point around graduation last year, a friend asked me what I had learned in the past four years. I don’t remember what I said at the time, other than that it was surely haphazard and incomplete, but the question kept kicking around in my head. So, a whole year later, as it turns out, here is a perhaps fuller, if necessarily still incomplete, answer to that question.

On abstraction

Being a math major at MIT, I of course learned some math. I see math as the study of abstraction and abstract structure, and I learned plenty about both – especially from the perspective of algebraic topology and related fields, but also as they pertain to other fields of math, physics, and computer science. (It turns out that many of the fields that interest me have abstraction as a core idea.)

I’ve learned that maintaining these abstractions in one’s head is hard. By the time you’re thinking about something like the Steenrod Algebra everything you do relates to reality only through at least five¹ layers of abstraction. The value of the abstraction is of course that you don’t have to think about the whole stack most of the time, when you’re just doing some algebraic computation. But to put things together, you have to load it all back into your head, and I learned how hard that is.

We need better ways of thinking about these abstractions, in math but also in physics, computer science, and probably elsewhere, too, because the difficulty of maintaining abstractions prevents us from building better ones that can do more for us. I’ve learned that web development, for instance, is surprisingly time-consuming even for experienced developers; there’s just a lot of code to write for everything one does. We need to be working at a much higher level – not just a slightly more powerful programming language or framework, but something really new. But it’s not at all clear how one gets there without collapsing under the weight of the abstractions. In any case, I digress – something I definitely already knew how to do more than four years ago.

On symmetry

On a more specific note, perhaps the single most interesting vaguely academic thing I learned about probably wasn’t in math classes at all: it was the idea of gauge symmetry in physics. Lie groups keep popping up as motivation everywhere in physics and math, and their particular application in quantum field theory results in a really incredible theory of the universe with surprisingly few arbitrary choices. Gauge symmetry, among other things, caused me to realize what I wish I had known earlier: that the applications of math can be really interesting in their own right.

Even classical field theory, as simplified as it is, is already a beautiful way of thinking about things, and quantum field theory is even better with the added benefit that it appears to actually describe our universe with unparalleled accuracy. Unfortunately it’s at the top of a mountain of physics and math, and even giving a real flavor for it without a lot of background is difficult. But I learned a lot on the parts of that mountain that I climbed.

On knowledge transfer

From my first evening at MIT, I began to learn the MIT oral history – the stories of hackers past, the escapades of student groups, and the origin of dorm traditions. And they are really an oral history²: they come with their heroes’ journeys, their tired tropes, and their cultural wisdom. The four-year memory of undergraduates makes passing on knowledge and values a constant concern for student groups and dorms, and I learned a lot about that.

As chair of MIT ESP and in various other positions, I learned a fair amount about the mechanics of knowledge transfer. I learned that it’s harder than it seems: there are any number of reasons to decide not to pass something on: that it won’t actually be that useful, that it’s just your opinion, that things will change anyway; and there are just as many ways to pass on knowledge that isn’t actually useful. But I also learned that the harder thing, and the arguably more important one, is values transfer. Passing on knowledge is hard, but it’s doable, and it’s not like I’m now completely off the grid if someone has a question. But passing on values is a lot harder, and something at which I wasn’t nearly as successful as I would have liked to be.

ESP certainly continues to care about its core mission, but with other values and other organizations it’s less clear: will ESP continue to fight the battle for student independence, even when it makes some enemies and conflicts with other goals? Will my floor, Bonfire, continue to collectively value leadership in the same way? Will the ASA continue to understand just what it means to say that “it is the highest responsibility of the [ASA] to act in the best interest of the MIT Community and of the Student Activities which it represents”? And is it even right for any of those organizations to carry on my values, as new members come in with their own expectations and ideals?

In the midst of that difficulty, I’ve learned the value of trivial traditions, and of the physical objects we leave behind: the party theme that recurs every year without fail, the furniture and kitchen implements that get passed down over the years, the murals that date back to time immemorial. Perhaps it’s silly, but the fact that the chair that pkoms bequeathed to Bonfire or the fancy printers that I convinced ESP to buy are still around holds some strange measure of meaning to me.

On people

I learned that people are messy, but totally worth it. In political science classes I learned about the ridiculous factors like shark attacks that affect how people vote. But I also learned how in spite of the ignorance of the average American, voters’ choices are often better-informed than one might expect. In learning about all of the ways in which our democracy doesn’t function, what I found most impressive were the ways in which it does function.

I learned a lot about organizations, and about how people function in them – both small organizations like ESP, and large organizations like MIT. As chair of ESP, I learned about the mechanics of managing an organization: running discussions, building consensus, making policy, overseeing people, and so on. And by paying attention to student politics and sitting on various student and faculty committees I learned about how amazingly complicated large organizations can and perhaps need to be.

And I learned a lot about social dynamics and how people interact informally. I learned that small social groups are wonderfully rewarding but also incredibly complicated, that telling someone they’re welcome to join you doesn’t make you welcoming, and that your friends will do stupid things and sometimes that’s okay and sometimes you should join them. I learned about the highs and lows of romantic relationships and about both deep and shallow friendships, all of which I certainly found more of than in high school. What I learned about people was mostly squishy and abstract, but it’s been an absolutely critical part of growing up.

On what it means to care

And, as strange as it is to say as such, I realize that as an undergrad I learned to care. This isn’t to say I hadn’t valued anything before – certainly my family was important to me before, for instance – but in some sense which is difficult to describe I learned how to really, deeply care.

I learned to care about ESP. The best and the worst thing about being in charge of the organization was the amount of ownership and responsibility I felt for it. It was the worst thing because that responsibility translated pretty directly into work, and into constantly thinking about what things meant for ESP. When I screwed up as chair, I noticed, because I could see how it affected this thing I care about. But the other side of that responsibility was that I felt a stake in the success of ESP. I cared about ESP and its students, teachers, and admins, and while I could take direct credit for only a very small part of whatever they collectively did, I was proud to be a part of it. And while the responsibilities of being chair faded as my term as chair ended, I still care a lot about ESP, and I’m still happy whenever it does something awesome, which it turns out is quite often.

I learned to care about the world. In high school if someone asked I would have said that of course I wanted to improve the world, but to be honest I didn’t think much about it – I tried to be a good person when I got an opportunity but didn’t particularly seek them out. Even freshman year when I joined ESP it was in large part because it was fun and it was what my friends were doing. And sometime in the following four years, I discovered I actually cared about the world. I learned that the best place to be at the end of an ESP program is at helpdesk in Lobby 10, to see all of the smiling students, parents, teachers, and admins as they leave more excited about learning than when they arrived. I learned just how many people have vastly fewer opportunities than I, and I learned that that’s something that bothers me. I learned that I want to do something with my life that is useful to the world in whatever small way it can be, and that I hope that everyone who is able to does the same. (And I learned that not everyone is able to do so! And that it’s worth giving people the benefit of the doubt and recognizing that being able to do something about these problems is itself a privilege.) I learned that the world is a pretty shitty place, but that we can make it just a little bit less shitty if we try.

And I learned to care about people. I had had friends before, but the peers I lived with and worked with on problem sets and extracurriculars became incredibly important to me. I learned that if you end up in the hospital in a snowstorm, your friends will trek through any amount of weather to help you get back home. I learned that when you’re still staring at piles of Feynman diagrams at 6am it’s a bit more fun when it’s with friends. And I learned how, and when, and why, to shout “BONFAAR” at the top of my lungs with my floormates. Four years later, I care about those friends in a new way.

On why it all matters

I learned a lot, by experiencing it, about the value of the undergraduate experience. I already knew there was a lot more to it than classes, but I learned how small a part of my education happened in a lecture. I worry that as we can offer the more academic parts of an education more cheaply online, we’ll lose the rest, or worse, we’ll end up with a two-tiered system in which only the privileged get these more important but less quantifiable parts of an education. I don’t think Khan Academy’s mission to offer a free, world-class education for anyone, anywhere will be complete until we can provide that too.³

Thanks among others to MITOC I (re-)discovered a love of hiking. And in doing so, I discovered that one of the reasons I love standing on top of a mountain is that what I see is, in the end, so fleeting – the mountain won’t look the same the next day, nor will I be back then, or perhaps ever. There will be other mountains on other days, but none quite the same, and the pictures won’t measure up, and that makes the moment at the top what it is. At the end of my four years I learned that being an undergrad has the same fleeting beauty – I won’t again be chair of ESP, or live with exactly the same group of people, nor, necessarily, would I want to, and I learned that’s part of what makes it special.

Thanks to Miriam Gershenson for comments on a draft.

By my count, a mathematical structure like $\mathbb{R}^n$ is an abstraction over anything in reality; point-set topology abstracts those; (co)homology, then cohomology operations add two more; and then the Steenrod Algebra sits on top of all of that. ↩
With the exception of a few stories that have been written down, and of course Hack, Punt, Tool. ↩
Obligatory disclaimer that my opinions here, and elsewhere on my blog, are my own. ↩

The polar plot of sine

Thu, 18 Feb 2016 00:00:00 +0000

A couple weeks ago a friend asked for intuition as to why the (polar) graph of $r = \sin\theta$ is a circle. It’s a fairly easy fact to prove algebraically, but neither he nor I had any intuition as to why it should be true, nor did the internet come to our aid. I thought about it for a bit and came up with an explanation, and since I couldn’t find it anywhere online I’m posting it here in case it’s of interest to anyone else.

First off, this is easy enough to do algebraically – we have $x = r \cos\theta$ and $y = r \sin\theta$, so multiplying both sides of $r = \sin\theta$ by $r$ we get \[x^2 + y^2 = r^2 = r \sin\theta = y.\] We just rearrange and complete the square to get \[x^2 + \left(y - \frac12\right)^2 = \frac14,\] i.e., a circle of radius $\frac12$ centered at $\left(0, \frac12\right)$. (Note that this technically only proves that the graph lies entirely on the circle, not that it traces the whole circle, but that’s mostly what we’re interested in anyway.)

But the goal was intuition. By plugging in a few numbers it’s not too hard to find the points where $\theta$ is a multiple of $\frac\pi4$ and see that they indeed lie on the correct circle, but that’s nothing resembling a proof. The key observation turned out to be drawing the line from the point $(r, \theta)$ to the point where the circle meets the $y$-axis, or $(1, 0)$ in rectangular coordinates:

From here it’s pretty easy: the triangle is right because it’s inscribed in a semicircle; then the two marked angles are equal (and the one at the origin is by definition $\theta$). Since the hypotenuse of the triangle has length $1$, we can just use the definition of sine to get that the lower chord has length $\sin\theta$ and thus the circle forms the graph we wanted.

Thanks to Danny Ben-David for posing the question and reading a draft of this post.

How I Teach Gerrymandering

Thu, 08 Oct 2015 00:00:00 +0000

Cross-posting from the brand-new MIT ESP teacher blog: I wrote a post about how I teach gerrymandering. It of course starts with the famous cartoon.

I’m going to be teaching it again at Splash in November, so suggestions are always welcome!

My Feeds

Sun, 14 Jun 2015 00:00:00 +0000

A couple people have asked me for this, so here’s a list of all the RSS/Atom feeds I follow. It’s autogenerated from Feedly, with some manual removal of dead things. If you want to import the whole assembly into your RSS reader (which I don’t recommend), you can grab the list in OMPL format.

Note: I don’t read all of these consistently. I read all of “Comics”, and make an effort to read most of “Blogs”, and beyond that it’s just whatever headline happens to interest me.

Comics

Because who can start their day without some webcomics?

Science and Technology

And vaguely related things that have too high a volume to put in the “blogs” folder.

Wonks

News and current events.

Culture

Miscellany of interest that’s more general than the other categories.

Food

I don’t read these regularly, but sometimes if I’m looking for ideas I do.

News Firehose

I never read remotely close to all of this, it’s just a giant dump of something like 500 articles a day that I occasionally glance at.

Blogs

A very motley assortment. I usually make an effort to read all of these, although I don’t succeed.

Overflow

Things I don’t read regularly, but keep around just in case.

Physics is addicting

Tue, 30 Sep 2014 00:00:00 +0000

Something half-baked that I’ve been thinking about lately: I think physics is addicting. In particular, I think learning physics is addicting in a way that math tends not to be.

When one first learns electricity and magnetism, for example, there are a lot of things one just accepts, and places where one learns what is obviously an incomplete picture. Some – like Coulomb’s law – are reasonable to just accept as axioms, but some – like these $\mathbf{E}$s and $\mathbf{B}$s floating around and the fact that changes in fields propagate at a speed $c=\frac1{\sqrt{\mu_0\epsilon_0}}$ which is, notably, a constant – make it clear that there’s more going on. So to fill the gaps, one takes another course in E&M, and another in special relativity, in which maybe one notices that the word ``acceleration’’ is conspicuously absent – thence general relativity. And at some point the can of worms that is quantum mechanics gets opened. First it comes as differential equations, but clearly there’s something more going on, so then it becomes linear algebra. But at this point it’s obvious that to really understand things, one needs to learn quantum field theory. By the end of the first course, there are as many gaps in the theory as gaps it fills, and while the second course will (I hope) fix some of those, it’s still not the end of the road. In fact, it’s not clear that anyone knows where the end of the road even is.

Contrast to math. Calculus, as one learns it in high school or early college, is not the most self-contained, but soon thereafter most parts of math stand on their own. Certainly after taking set theory, topology, and analysis, one can claim that all the things one knows, one fully understands. That doesn’t mean there isn’t more to learn – algebra, for example, but it all in some sense builds up on top of the things already learned. Homotopy groups take algebraic topology to understand, but they are, after all, a part of algebraic topology. Where physics builds downwards, filling in the foundation of each concept with the next one, math builds upwards, one concept upon the next.

I don’t know if this means anything about the fields. Maybe it means mathematicians are better at figuring out how to learn things in a way that doesn’t leave lots of gaps, or maybe they are better at coming up with useful abstractions and suitable sets of axioms, or maybe they have an easier job since we do have a reasonable foundation for math. And I think to some extent parts of physics that I know less of may build up more – statistical mechanics, for one. CS definitely does some of each: from basic programming one can go up, to heavier software engineering, algorithms, or AI, or down, to circuits, CPUs, and compilers. But it does seem like an interesting difference to me.