When you have a MALLET, everything looks like a nail

Note: this post is part 4, section 2 of my series on whaling logs and digital history. For the full overview, click here.

One reason I'm interested in ship logs is that they give some distance to think about problems in reading digital texts. That's particularly true for machine learning techniques. In my last post, an appendix to the long whaling post, I talked about using K-means clustering and k-nearest neighbor methods to classify whaling voyages. But digital humanists working with texts hardly ever use k-means clustering; instead, they gravitate towards a more sophisticated form of clustering called topic modeling, particularly David Blei's LDA (so much so that I'm going to use 'LDA' and 'topic modeling' synonymously here). There's a whole genre of introductory posts out there encouraging humanists to try LDA: Scott Weingart's wraps a lot of them together, and Miriam Posner's is freshest off the presses.

So as an appendix to that appendix, I want to use ship's data to think about how we use LDA. I've wondered for a while why there's such a rush to make topic modeling into the machine learning tool for historians and literature scholars. It's probably true that if you only apply one algorithm to your texts, it should be LDA. But most humanists are better off applying zero clusterings, and most of the remainder should be applying several. I haven't mastered the arcana of various flavors of topic modeling to my own satisfaction, and don't feel qualified to deliver a full-on jeremiad against its uses and abuses. Suffice it to say, my basic concerns are:

1. The ease of use for LDA with basic settings means humanists are too likely to take its results as 'magic', rather than interpreting it as the output of one clustering technique among many.
2. The primary way of evaluating its result (confirming that the top words and texts in each topic 'make sense') ignores most of the model output and doesn't map perfectly onto the expectations we have for the topics. (A Gary King study, for example, that empirically ranks document clusterings based on human interpretation of 'informativeness' found Direchlet-prior based clustering the least effective of several methods.)

Ship data gives an interesting perspective on these problems. So, at the risk of descending into self-parody, I ran a couple topic models on the points in the ship's logs as a way of thinking through how that clustering works. (For those who only know LDA as a text-classification system, this isn't as loony as it sounds; in computer science, the algorithm gets thrown at all sorts of unrelated data, from images to music).

Instead of using a vocabulary of words, we can just use one of latitude-longitude points at decimal resolution. Each voyage is a text, and each day it spends in, say, Boston is one use of the word "42.4,-72.1". That gives us a vocabulary of 600,000 or so 'words' across 11,000 'texts', not far off a typical topic model (although the 'texts' are short, averaging maybe 30-50 words). Unlike k-means clustering, a topic model will divide each route up among several topics, so instead of showing paths, we can visually only look at which points fall into which 'topic'; but a single point isn't restricted to a single topic, so New York could be part of both a hypothetical 'European trade' and 'California trade' topic.

With words, it's impossible to meaningfully convey all the data in a topic model's output. Geodata has the nice feature that we can inspect all the results in a topic by simply plotting them on a map. Essentially, 'meaning' for points can be firmly reduced a two-dimensional space (although it has other ones as well), while linguistic meaning can't.

Here's the output of a model, plotted with high transparency so that a point on the map will appear black if it appears in that topic in 100 or more log entries. (The basic code to build the model and plot the code is here--dataset available on request).

Click to enlarge

At first glance, topic modeling performs far better than it has any right to on this sort of data. A 25-topic model yields a number of distinctively 'whaling' topics (see below); if I had a dataset with no metadata but a strong sense of what these map regions might mean, I might be able to separate out whaling by picking voyages heavy in topics 18, 5, 21, 16, and 11, each of which roughly corresponds to a few of the whaling grounds in the master chart.  I was frankly surprised how well this worked; I was expecting less cognizable chunks (I thought, based on my fuzzy understanding of the math, that the Atlantic crossing might be split into two topics, for example, and that most trips would draw from several smaller chunks like that. Maybe this would be true of a 100-topic model.)

The optimistic take on this is pretty obvious:

1. Topic modelling is amazing;
2. Why not topic model everything?

Since that's so straightforward, let me give the pessimistic take instead.

In short, the results are not always magic, and the way that we interpret textual topic models doesn't seem as robust as geographic analysis. For example: if I ask MALLET (the software package most people use for topic modelling) for a 9-topic model instead of a 25-topic one, things look all right on first glance: you even get some nice features, like separate clusters for the routes from and to US east coast on the trade winds. Closer inspection, though, reveals some strange decisions on the part of the clustering, such as binning the western Pacific whaling grounds in with the transatlantic clipper routes (middle right), or trade around Florida with the Bering straits (lower right).

Let me blow up that topic 5 so you can see how strange it is:

The algorithm is obviously combining two very different clusters together; whaling from Hawaii, and the eastern seaboard–England shipping route. But the standard way of interpreting topic model results, looking at the top words in each topic, might have led me to label it as "transatlantic shipping", which encompasses 9 of the ten top places in the topic. (The 10th is Guam, which isn't particularly close to the larger cluster in the Pacific; no individual point in there is all that common). Are there similar sub-threshold parasite topics in MALLET output on textual data that we don't see? I'm curious if someone who knows LDA better than I can explain why this might be a quirk of this particular source, rather than a general phenomenon of the clustering algorithm. (Topic 0 in the 25-topic model has similar problems).

This is a case where I'm really being saved by the restrictive feature space of data. If I were interpreting these MALLET results as text, I might notice it, for example, but start to tell a just-so story about how transatlantic shipping and Pacific whaling really are connected. (Which they are; but so is everything else.) The absurdity of doing that with geographic data like this is pretty clear; but interpretive leaps are extraordinarily easy to make with texts.
 
Obviously this is an extreme test of LDA; that it performs at all is to its credit, and weird clustering results like this are probably a result of the differences between this and the data MALLET is designed for. Better priors and maybe some other sounder assumptions might fix this somewhat; ship's paths are not texts, even in the shape of their data. (In this case, I bet the model could be made a lot better by fine-tuning the alpha and beta parameters to expect less breadth of documents across words; does that sound right? For the record, their distribution across geopoints isn't too far off Zipf's law after the first ten points.)

The reason I find it useful at all for understanding topic modeling is that it's also extreme, in that it's much easier to visually see the whole results of what a topic model spits out, not just the top ten words or top 20 results. One of the things I always say about Bookworm is that humanists need search interfaces that return more than just a unidimensional ordered list. We need that for interpreting clustering results much more than we need it for search results; but it's extremely hard to do. The nice thing about geographic data is that we have a ready-made contextual system for spatially situating points that let us instantly see outliers like above. That doesn't really exist for texts. (Using word clouds for topic model results, as Elijah Meeks does, is a great idea, but word clouds are just an alternate univariate way of visualizing ordered lists.)

The lesson of the whales I'd want to apply here is that we need to be able to tune and return our results in concert with outside evidence; outside metadata, visual inspection, or (ideally) both. That's not to say LDA is completely inscrutable; Matt Jockers, for example, has done just this sort of iterative improvement on the models of the Stanford literary corpus, and that lets him made some great conclusions from the data. But even those as deep into the plumbing as Jockers, Ted Underwood, or Allen Riddell will have a hard time bringing other humanistic readers along on the interpretive choices they make in tuning their topic models, and instead will have to rely on protestations of authority. Which isn't terrible—it's standard interpretive practice—but loses one of the big benefits I've seen in humanistic research, which is that it can help open up the act of humanistic inference to non-experts.

And most humanists who do what I've just done—blindly throwing data into MALLET—won't be able to give the results the pushback they deserve. I feel like I have some sense of how points cluster in high-dimensional cartesian spaces, which helps me troubleshoot k-means results, but find myself completely unable to to intuit what might go wrong in 25-dimensional simplexes. I don't think I'm alone: and I'm not sure that we should too enthusiastic about interpreting results from machine learning which we can only barely steer.

So there are cases where topic modeling can be useful for data-creation purposes; I'm satisfied with my other way of extracting whaling voyages, for instance, but topic models would work here. But as artifacts to be interpreted on their own, topic models may be less useful. The downside of too-aggressive topic modeling is that it leads occasionally leads us to treat junk as an oracle. (This isn't specific to topic modeling, obviously: other methods I've written about in the past, like principal components analysis, can be ever worse).

A poorly supervised machine learning algorithm is like a bad research assistant. It might produce some unexpected constellations that show flickers of deeper truths, but there are lots of ways to get creative rearrangements. If it's not doing the task we want, it's time to give it some clearer instructions, or to find a new one.