tag:blogger.com,1999:blog-8929346053949579231.post1881521369881006071..comments2024-03-11T02:10:31.396-04:00Comments on Sapping Attention: When you have a MALLET, everything looks like a nailBenhttp://www.blogger.com/profile/04856020368342677253noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-8929346053949579231.post-19666731276600345252013-10-08T02:00:23.790-04:002013-10-08T02:00:23.790-04:00I was very pleased to find this site. I wanted to ... I was very pleased to find this site. I wanted to thank you for this great read!<br /><a href="http://www.cert4prep.com/HP0-Y46.html" rel="nofollow">HP0-Y46</a>webexpert66https://www.blogger.com/profile/08713596422905473588noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-9424263367839821592012-11-06T01:42:41.887-05:002012-11-06T01:42:41.887-05:00I have not come across LDA before but to me (a sta...I have not come across LDA before but to me (a statistician with an interest in mixtures of distributions) it seems the same as LCA, Latent Class Analysis, a form of mixture modelling attuned to categorical data.<br />Mixture modelling can be used for clustering but it is not as straightforward and 'black-boxy' as algorithmic clustering methods.<br /><br />There is an excellent web page http://john-uebersax.com/stat/ by John Uebersax which deals at length with various aspects of LCA (?= LDA) from a social and medical science perspective.<br /><br />Passing on the link was all I wanted to do, really, but I could add that one of the difficult aspects of using LCA is choosing the number of classes, a point that Ben's post illustrates. On the positive side LCA deals quite well with overlapping clusters, which is not done very well by distance-based clustering methods. K-means, which Ben seems to like has quite a few features in common with mixture models, but is not formally based of an statistical model.<br /><br /> Anonymoushttps://www.blogger.com/profile/14586290254444971774noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-4769794581210276712012-11-05T14:41:36.844-05:002012-11-05T14:41:36.844-05:00Most of this conversation passed far over my head....Most of this conversation passed far over my head. But I thought of it again when I read this <a href="http://bostonglobe.com/ideas/2012/11/03/how-twitter-language-reveals-your-gender-your-friends/e68H6Z0Z2GAfnJ6UjU3IxO/story.html" rel="nofollow">account</a> of gender, affective language, and Twitter. It describes the work of a clever Stanford linguist named Tyler Schnoebelen, who used topic-modeling on 14k users to classify their language by gender. (<a href="http://arxiv.org/abs/1210.4567" rel="nofollow">Gender in Twitter: Styles, stances, and social networks</a>) In a separate paper, also part of his doctoral work, he looked at the use of emoticons, using a corpus of 39 million American tweets. (<a href="http://repository.upenn.edu/pwpl/vol18/iss2/14/" rel="nofollow">Do You Smile with Your Nose? Stylistic Variation in Twitter Emoticons</a>)<br /><br />What's striking about the work is that after producing apparently-robust-yet-facile conclusions from simple topic modeling, a model that was 88% predictive of a user's actual gender based on the incidence of certain words, he went back and looked further down the list. That, in turn, enabled him to identify smaller populations of users whose patterns of language use differed significantly from gendered norms. And these populations exhibited other differences, too - their interactions skewed toward users of the opposite gender. <br /><br />I'm less interested in his specific findings than in his approach. If he'd set out to find ways to classify users by gender, he might've declared success after his initial round of analysis - or refined his topic-modeling to make it even more tightly predictive. But, from his papers, it seems he was much more interested in subverting facile gender dichotomies than in reinforcing them, so he dug a little deeper. And he probably could've gone further, still. I suspect that even among the 88% of users whose gender was correctly predicted by topic modeling, there'd be some fascinating differences in incidence of specific words and constructions that reflect very different constructions of gender - much that way that if you'd been looking to identify east-west voyages, Topic 5 would've been highly predictive, even though it lumps together Hawaiian and Transatlantic routes that are fundamentally dissimilar. <br /><br />I don't know that it adds much, other than to highlight the fact that the quality of the corpus and the sophistication of the algorithm probably matter a whole hell of a lot less than the innate skepticism and curiosity of the researcher.Yoninoreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-69781179846025129972012-11-04T01:26:32.870-05:002012-11-04T01:26:32.870-05:00Sorry, er, I meant "not using a stopword list...Sorry, er, I meant "not using a stopword list *at all*" with regards to the poetry and authorship studies, and then using a stopword list for scientific articles, so reverse what you thought I'd said. <br /><br />Anyway, there are better ways to attribute authorship, it's just something I'd pulled off the top of my head, haven't actually tried to compare it to the usual methods. Jockers likes to split up stylistic and thematic similarity, but both make up an author's unique thumbprint, so I imagine LDA might still be useful in that regard.Scott B. Weingarthttp://www.scottbot.netnoreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-26131042986604042082012-11-03T22:03:58.902-04:002012-11-03T22:03:58.902-04:00Wow, I had missed the LDA-as-authorship-attributio...Wow, I had missed the LDA-as-authorship-attribution-technique papers. Obvious in retrospect… You're saying not using stopwords there is ideal? I would have thought the opposite, given the standard literature on authorship attribution; seems to be at least one paper out there suggesting that LDA on stopwords ALONE is the best as classifying Federalists papers. Which makes sense, but, yuck.<br /><br />I've got to think a bit more about this "whole model" thing, because it's not something I'm sure anybody is doing.<br />Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-42274391461415494272012-11-03T20:42:22.041-04:002012-11-03T20:42:22.041-04:00Inclusion of stopwords, and what makes it into the...Inclusion of stopwords, and what makes it into the stopword list, largely depends on your use. As Ted says, model results should be used as heuristics; in cases of poetry analysis, authorship attribution, etc., not using stopwords is generally ideal. When looking for trending topics in scientific articles, stopwords are pretty useful.Scott B. Weingarthttp://www.scottbot.netnoreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-86797339993562741152012-11-03T09:44:40.467-04:002012-11-03T09:44:40.467-04:00Can't tell you how many times I've almost ...Can't tell you how many times I've <i>almost</i> finally convinced a humanist that close reading is obsolete and only computers can truly appreciate an author's intentions, only to lose them at the last moment by carelessly saying 'corpuses' instead of 'corpora.'<br /><br />Yeah, it's the aggregate results that are so tricky. That question I was asking you at Rutgers about the super-Ngrams people tend to make with topic models falls into the same bin, I think. I've wondered if doing something on every edition of Shakespeare published over 200 years would fail in a demonstrative way. (Although yeah, it would also still work. Grr.)Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-15927905999903626272012-11-03T09:26:47.911-04:002012-11-03T09:26:47.911-04:00Thanks, that's helpful. It's true, King is...Thanks, that's helpful. It's true, King is citing a <a href="http://www.cs.princeton.edu/~blei/papers/BleiJordan2004.pdf" rel="nofollow">different Blei-Jordan DIrechlet paper</a>; I'm just guessing that for document purposes it might be similar. And then there's a tendency to cluster inside the topic space that you create; I saw Project Bamboo do something like that once with PCA, and tried it on the ships on lark. (Didn't work, probably because there's too little dimensionality; higher-k topics on this set seem to produce about 60% random scatters).Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-19814638252874459452012-11-03T08:34:46.122-04:002012-11-03T08:34:46.122-04:00But I wouldn't say that about all algorithms. ...But I wouldn't say that about all algorithms. I <em>would</em> be willing to use corpus comparison, or a classification algorithm, as evidence -- both because they're simpler and because they're answering a more sharply-defined sort of question.<br /><br />So I guess I'm agreeing with you about LDA. I think it's a great way to map a large collection that you couldn't otherwise survey. But if people want to produce evidence, they might be better off with a simpler algorithm. Ted Underwoodhttps://www.blogger.com/profile/04012428899328561750noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-70802386455402007692012-11-02T17:07:20.017-04:002012-11-02T17:07:20.017-04:00Stopwords are definitely a "pay no attention ...Stopwords are definitely a "pay no attention to the man behind the curtain" moment. My experience has been that choices about stopwords profoundly shape the results. E.g., if you include personal names when you model fiction, it just doesn't work, because you'll get an "Elizabeth" topic.<br /><br />Does this mean we can't exclude stopwords? Well, my attitude to LDA is that it's useful as a heuristic, not as evidence. I know that's not how people are imagining it right now, and I'm afraid there's going to be some disillusionment ... But if you're approaching it as a heuristic, there's no reason not to exclude whatever stopwords you want. It's not about proof, for me, it's about making the heuristic work for you.Ted Underwoodhttps://www.blogger.com/profile/04012428899328561750noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-86827906088268154362012-11-02T16:38:36.009-04:002012-11-02T16:38:36.009-04:00I wouldn't want to claim that LDA is "bet...I wouldn't want to claim that LDA is "better" than k-means at all. They do different things. K-means works by measuring similarity. It's often used to cluster documents -- but you could use it to cluster features (represented as distributions over document-space). <br /><br />To the extent that LDA can be conceived as clustering (which I think is only "sorta"), it's clustering specific <em>occurrences</em> of features. And it doesn't do that by measuring similarity between them; it just assigns them to "bins." That's a basically different approach. So I wouldn't use LDA to produce a metric of similarity at all. (I think King is actually talking about a different algorithm.)<br /><br />I also agree that the current popularity of LDA is in danger of making "everything look like a nail." Humanists are probably more commonly interested in grouping documents than they are in grouping "kinds of vocabulary." But in that case, we need classification or clustering algorithms, rather than LDA. People should also be familiar, e.g., with the hierarchical clustering algorithm you've used so effectively. Hierarchical clustering does have the advantage of not producing those arbitrary divisions.<br />Ted Underwoodhttps://www.blogger.com/profile/04012428899328561750noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-21029960258230271832012-11-02T15:34:03.143-04:002012-11-02T15:34:03.143-04:00Amazing as ever, Ben. Though I have to say in term...Amazing as ever, Ben. Though I have to say in terms of outreach to humanists, comments like "I feel like I have some sense of how points cluster in high-dimensional cartesian space...but find myself completely unable to to intuit what might go wrong in 25-dimensional simplexes" may pose trouble. (Because it's "simplices.")<br /><br />Anyway, this is a very concrete demonstration of the vague worry I have as a statistically undereducated person in looking at topic modeling: how do I evaluate the quality of the model? When I make a topic model, what can I report as an index of the fit? The introductory presentations of LDA emphasize its use as a heuristic: this might help you discover related stuff in a big archive of documents. For the heuristic use the possibility of artificially "fused" topics and nonsense isn't a big deal. But one also wants to use the classification itself--the whole model, as Ted says, not any individual topic--and the introductions I've looked at say much less about how to tell a dataset that is well modeled by LDA from a set that isn't. Inspecting top n words for a theme is dangerous, because of the Ay-very-like-a-whale problem. Anyway this is definitely chastening me and making me want to be very careful how I report any LDA result.<br /><br />It's at first surprising and then not surprising that the spatial voyage data worked: surprising because you're throwing out the time ordering of the data! how can that not matter! then not surprising, because the spatial continuity of the track of a voyage guarantees that LDA will be able to do something with the way coordinates co-occur in a document.<br /><br />Maybe it would be helpful to try to come up with a document set, textual or not, that MALLET just completely bombs on. Uh...Andrew Goldstonehttp://www.rci.rutgers.edu/~ag978noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-20385397463469750162012-11-02T14:58:25.813-04:002012-11-02T14:58:25.813-04:00In case this ends up confusing anyone, my longer c...In case this ends up confusing anyone, my longer comment below is in reply to this.Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-90992768276289360752012-11-02T14:57:38.555-04:002012-11-02T14:57:38.555-04:00To restate the end of my last comment more succinc...To restate the end of my last comment more succinctly: I don't know how to assess the validity of the following statement: <b>The apparent advantages of LDA over other clustering algorithms are primarily artifacts of the indicators of quality we like to use (top-n lists of documents and words), and it tends to be un-extraordinary on other metrics (like visual scatters across a map, or King's surveys about similarity-score usefulness).</b>Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-27279492038033367912012-11-02T14:28:13.134-04:002012-11-02T14:28:13.134-04:00Yeah, that's a good point.
Random question, f...Yeah, that's a good point.<br /><br />Random question, for you or anyone in these comments: sometimes I think we shouldn't exclude stopwords in topic model results, either; feels like a funny hack just to preserve the illusion we're looking at epistemological categories, not computerized results. Is that just curmudgeonly?Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-73549408847224645832012-11-02T14:22:07.130-04:002012-11-02T14:22:07.130-04:00Slightly incoherently:
Yeah, one problem is the a...Slightly incoherently:<br /><br />Yeah, one problem is the arbitrariness of choosing K. (Machine-oriented methods for choosing 'better' k-values comes up constantly on the topic-modeling list-serve: basically, it seems like most of the solutions to it aren't especially effective). One solution to this is 'run loads of topics, and some of them will be meaningful'. Not the worst solution, as it turns out, but the whole appeal of topics to say that they correspond to something real, and we don't have the best tools to assess whether they do. <br /><br />Still, I think it's a little tricky to say "you can't overinterpret the points of division between topics/clusters" when the whole thing that makes topic modeling appealing is that it creates hard and fast divisions. On this, K-means is definitely as problematic: it's just useful as a sort of garden-variety machine classification, less complicated than LDA and easier to understand. I should really do a k-means clustering of points like the one above, to see if it produces similar merged artifacts vs. just fuzzier boundaries.<br /><br />I guess I worry that something about topic modeling, though, is actively sidestepping the heuristics that normally lead us to reject the arbitrary results you get when setting K.<br /><br />Maybe put it this way. Hard and fast divisions are extraordinarily useful for thinking. But machine-generated ones tend not to work. So any clustering algorithm has to convince us it works. We think topic modeling works largely because it has: 1) distributions over documents and words so that you don't have to have a single bin for each word or document; in that way, it's just better.<br /><br />But at the same time, topic modeling also succeeds because it has 2) output that's difficult to falsify, because the top n documents and the top n words tend to intuitively make sense. (Except when they don't, which is when we throw it out as a 'junk topic,' which we don't have a cognitive problem with.) Again, this is also true of k-means, but we often reject k-means document results outright. And K-means results are slightly more susceptible to being interpreted visually (you can drill through plots of the top principal components on whatever vectors you're classifying on, say, to get some basic spatial sense). I think I care more about having visual fallbacks than most people, but we need <i>some</i> fallbacks.<br /><br />It's not clear to me that picking a clustering algorithm that works for top-n documents and top-n words is the best possible way. Would topic modeling succeed over its competitors equally well if we had some uncorrelated test of how well clustering algorithms succeed? The reason I keep thinking about that Gary King paper is that it suggests it might not.<br /><br />Benhttps://www.blogger.com/profile/04856020368342677253noreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-88912290139088993542012-11-02T13:26:25.936-04:002012-11-02T13:26:25.936-04:00Marvelous post as always, Ben. I'd like to ext...Marvelous post as always, Ben. I'd like to extend one of your gripes to a very limited degree. When most researchers present LDA results, they show an ordered word list. Not only is the list univariate, it's also completely ordinal. At least Elijah's word clouds display the data on an interval scale; it's still not as complex a representation as it ought to be, but it displays a heck of a lot more data than those ordered lists do.Scott B. Weingarthttp://scottbot.netnoreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-29776361944649759252012-11-02T13:15:57.602-04:002012-11-02T13:15:57.602-04:00One of the implications of the arbitrariness of k ...One of the implications of the arbitrariness of k is that, properly speaking, there's no such thing as interpreting a single cluster, or a single topic.<br /><br />You have to interpret the model as a whole. That's the only way to factor out the arbitrariness of particular boundaries. But you're right that this point hasn't been made sufficiently forcefully yet to a humanistic audience.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8929346053949579231.post-5717965658055240062012-11-02T12:49:39.977-04:002012-11-02T12:49:39.977-04:00Ben, you're absolutely right that the sort of ...Ben, you're absolutely right that the sort of problem you discover here pops up also in textual LDA. If you reduce the number of topics, you'll get things stuck together in odd ways. E.g., when Andrew Goldstone and I topic-modeled the print run of PMLA, there were some topics that combined film studies and Jewish studies. I could have constructed a big theory about that conjunction, but the reality is probably that they overlap in a couple of semantic/contextual places, and if you reduce the number of topics, things that overlap *at all* have to get fused.<br /><br />But I would say this exemplifies a broader issue about interpretation of machine learning in general. You've got to understand the algorithm. With (most forms of) k-means and LDA, the number of clusters is an arbitrary input. (There are some sophisticated ways around that with k-means, but let's hold that thought.) So you've got to start by realizing that you can't overinterpret the points of division between topics/clusters. Those boundaries are in principle, as you're observing here, arbitrary. It is not necessarily significant that "these two things got grouped together."<br /><br />But ultimately this is a potential problem for clustering as well. K-means clustering can be great, if you choose an appropriate value of k. What's an appropriate value? Well, you need to tinker ... and even when you're done tinkering, you need to realize that there's nothing absolute and inherent about those cluster boundaries. But they may still be heuristically useful.<br /><br />I'll admit that LDA is a bit harder to understand, and therefore potentially more likely to function as a mystifying black box. That's no good, and I hope the MITH workshop this weekend will help combat that danger (usefully guided by your warnings here).<br /><br />Anonymousnoreply@blogger.com