Here's the basic idea:
Friday, April 27, 2012
Publishing Libraries
[The American Antiquarian Society conference in Worcester last weekend had an interesting rider on the conference invitation--they wanted 500 words from each participant on the prospects for independent research libraries. I'm posting that response here.]
Here's the basic idea:
Here's the basic idea:
Monday, April 9, 2012
Visualizing Ocean Shipping
I saw some historians talking on Twitter about a very nice data visualization of shipping routes in the 18th and 19th centuries on Spatial Analysis. (Which is a great blog--looking through their archives, I think I've seen every previous post linked from somewhere else before).
They make a basically static visualization. I wanted to see the ships in motion. Plus, Dael Norwood made some guesses about the increasing prominence of Pacific trade in the period that I would like to see confirmed. That got me interested with the ship data that they use, which consists of detailed logbooks that have been digitized for climatological purposes. On the more technical side, I have been fiddling a bit lately with ffmpeg and ggplot (two completely unrelated systems, despite what the names imply) to make animated visualizations, and wanted to put one up. And it's an interesting case; historical data was digitized for climatological purposes, which means visualization is going to be on of the easiest ways to think about whether it might be usable for historical demonstration or analysis, as well.
So here are two visualizations.
[Update 11/12: For more of this, see my discussion of American shipping, and whaling in particular, from 1800 to 1860.]
The first one is long: it shows about 100 years of ship paths in the seas, as recorded in hundreds of ship's log books, by hand, one or several times a day. I haven't watched the whole thing at once, but skipping around gives a pretty good idea of the state of the database (if not world shipping) at any given moment.
You can watch either of these in much higher resolution by clicking around here or on YouTube--I definitely recommend 720p.
This shows mostly Spanish, Dutch, and English routes--they are surprisingly constant over the period (although some empires drop in and out of the record), but the individual voyages are fun. And there are some macro patterns--the move of British trade towards India, the effect of the American Revolution and the Napoleonic Wars, and so on.
The second has to do with seasonality: it compresses all those years onto a single span of January-December, to reveal seasonal patterns. I loop through a couple times so you can get a better sense, but the data is the same for each year.
They make a basically static visualization. I wanted to see the ships in motion. Plus, Dael Norwood made some guesses about the increasing prominence of Pacific trade in the period that I would like to see confirmed. That got me interested with the ship data that they use, which consists of detailed logbooks that have been digitized for climatological purposes. On the more technical side, I have been fiddling a bit lately with ffmpeg and ggplot (two completely unrelated systems, despite what the names imply) to make animated visualizations, and wanted to put one up. And it's an interesting case; historical data was digitized for climatological purposes, which means visualization is going to be on of the easiest ways to think about whether it might be usable for historical demonstration or analysis, as well.
So here are two visualizations.
[Update 11/12: For more of this, see my discussion of American shipping, and whaling in particular, from 1800 to 1860.]
The first one is long: it shows about 100 years of ship paths in the seas, as recorded in hundreds of ship's log books, by hand, one or several times a day. I haven't watched the whole thing at once, but skipping around gives a pretty good idea of the state of the database (if not world shipping) at any given moment.
You can watch either of these in much higher resolution by clicking around here or on YouTube--I definitely recommend 720p.
This shows mostly Spanish, Dutch, and English routes--they are surprisingly constant over the period (although some empires drop in and out of the record), but the individual voyages are fun. And there are some macro patterns--the move of British trade towards India, the effect of the American Revolution and the Napoleonic Wars, and so on.
The second has to do with seasonality: it compresses all those years onto a single span of January-December, to reveal seasonal patterns. I loop through a couple times so you can get a better sense, but the data is the same for each year.
Wednesday, April 4, 2012
Turning off the TV
I'm starting up a new blog,
People are interested in TV anachronisms, and I find the patterns it unveils really interesting for understanding language change. (A lot of my dissertation research focuses on just the sort of below-the-radar language changes). But I made this blog for working with large textual sources and posting occasional off-the-cuff rants about digital humanities, and the posts have gotten longer with time. I don't want swamp it with too much about television. Minor week-by-week rundowns of Mad Men would fall under that category, as would random Deadwood visualizations and a bunch of other things I have sitting around and may want to dole out.
I think we could have a mildly interesting discussion about the role of TV and film criticism in the digital humanities, which retains a bit of stodginess about its subject matter in order to secure acceptance for its methodologies. (I tend to think this is a wise bit of strategic positioning, but am open to the opposite perspective). Though I do have a fair amount of early broadcasting history in my dissertation, I can't bring myself to do a full-throated defense of writing about TV right now and passing it up as a somehow academic endeavor--chalk me up as part of the problem.
I'll probably follow the Andrew Gelman model and crosspost on some things with dual relevance. So whenever I get around to savaging Edith Wharton for her tin ear in The Age of Innocence, it will be here as well.
Monday, April 2, 2012
Digital Collections, Research Libraries, Collaboration
[The following is a revised version of my talk on the 'collaboration' panel at a conference about "Needs and Opportunities for the Research Library in the Digital Age" at the American Antiquarian Society in Worcester last week. Thanks to Paul Erickson for the invitation to attend, and everyone there for a fascinating weekend.]
As a few people here have suggested, there's a lot to be suspicious of in the foisting of collaboration on unsuspecting researchers. To those worries about collaboration that have already been brought up (including by myself elsewhere), I'd add the particular suspicions that early-career scholars often bear. Collaboration is often one of those ambitious things that successful scholars only seem to turn to in earnest with the security of tenure, like transnational history or raising children.
But in the last few years, I've turned more and more to working with digital sources; and in doing so, it turns out collaboration is essential. It's impossible to escape. And, as everyone says, it really is wonderful.
But the forms that digital collaboration takes, particularly when it's most helpful, are very different than the traditional forms of heady engagement around a shared codex, blackboard, or meal that tend to get us most sentimental when talking about collaborative work. And that has important implications for libraries like this, because it suggests that the way you find your collaborators may be quite different. In some cases, you may not even know who they are. And the attributes it takes to attract these invisible collaborators can be quite different from those that libraries traditionally try to display, though they remain one that a library like this may have in abundance.
As a few people here have suggested, there's a lot to be suspicious of in the foisting of collaboration on unsuspecting researchers. To those worries about collaboration that have already been brought up (including by myself elsewhere), I'd add the particular suspicions that early-career scholars often bear. Collaboration is often one of those ambitious things that successful scholars only seem to turn to in earnest with the security of tenure, like transnational history or raising children.
But in the last few years, I've turned more and more to working with digital sources; and in doing so, it turns out collaboration is essential. It's impossible to escape. And, as everyone says, it really is wonderful.
But the forms that digital collaboration takes, particularly when it's most helpful, are very different than the traditional forms of heady engagement around a shared codex, blackboard, or meal that tend to get us most sentimental when talking about collaborative work. And that has important implications for libraries like this, because it suggests that the way you find your collaborators may be quite different. In some cases, you may not even know who they are. And the attributes it takes to attract these invisible collaborators can be quite different from those that libraries traditionally try to display, though they remain one that a library like this may have in abundance.
Wednesday, March 21, 2012
Mad Men anachronism hunting
[Update: I've consolidated all of my TV anachronisms posts at a different blog, Prochronism, and new ones on Mad Men, Deadwood, Downton Abbey, and the rest are going there.]
I've got an article up today on the Atlantic's web site about how Mad Men stacks up against historical language usage. So if you're reading this blog, go read that.
Maybe I'll add some breakouts of individual episodes later today if I get some time, but here are the overall word clouds like the ones I made for Downton Abbey. Mad Men has noticeably fewer outliers towards the top:
And the ones that are are actually appropriate. (My dissertation actually has a bit on the origins of focus groups in the 1940s).
I've got an article up today on the Atlantic's web site about how Mad Men stacks up against historical language usage. So if you're reading this blog, go read that.
Maybe I'll add some breakouts of individual episodes later today if I get some time, but here are the overall word clouds like the ones I made for Downton Abbey. Mad Men has noticeably fewer outliers towards the top:
And the ones that are are actually appropriate. (My dissertation actually has a bit on the origins of focus groups in the 1940s).
Tuesday, March 6, 2012
Do women hide their gender by publishing under their initials?
A quick follow-up on this issue of author gender.
In my last post, I looked at first names as a rough gauge of author gender to see who is missing from libraries. This method has two obvious failings as a way of finding gender:
1) People use pseudonyms that can be of the opposite gender. (More often women writing as men, but sometimes men writing as women as well.)
2) People publish using initials. It's pretty widely known that women sometimes publish under their initials to avoid making their gender obvious.
The first problem is basically intractable without specific knowledge. (I can fix George Eliot by hand, but no other way). The second we can get actually get some data on, though. Authors are identified by their first initial alone in about 10% of the books I'm using (1905-1922, Open Library texts). It turns out we can actually figure out a little bit about what gender they are. If this is a really important phenomenon in the data, then it should show up in other ways.
In my last post, I looked at first names as a rough gauge of author gender to see who is missing from libraries. This method has two obvious failings as a way of finding gender:
1) People use pseudonyms that can be of the opposite gender. (More often women writing as men, but sometimes men writing as women as well.)
2) People publish using initials. It's pretty widely known that women sometimes publish under their initials to avoid making their gender obvious.
The first problem is basically intractable without specific knowledge. (I can fix George Eliot by hand, but no other way). The second we can get actually get some data on, though. Authors are identified by their first initial alone in about 10% of the books I'm using (1905-1922, Open Library texts). It turns out we can actually figure out a little bit about what gender they are. If this is a really important phenomenon in the data, then it should show up in other ways.
Evidence of absence is not absence of evidence
I just saw that various Digital Humanists on Twitter were talking about representativeness, exclusion of women from digital archives, and other Big Questions. I can only echo my general agreement about most of the comments.
But now that I see some concerns about gender biases in big digital corpora, I do have a bit to say. Partly that I have seen nothing to make me think social prejudices played into the scanning decisions at all. Rather, Google Books, Hathi Trust, the Internet Archive, and all the other similar projects are pretty much representative of the state of academic libraries. (With strange exceptions, of course). You can choose where to vaccum, but not what gets sucked up the machine; likewise the companies.
But now that I see some concerns about gender biases in big digital corpora, I do have a bit to say. Partly that I have seen nothing to make me think social prejudices played into the scanning decisions at all. Rather, Google Books, Hathi Trust, the Internet Archive, and all the other similar projects are pretty much representative of the state of academic libraries. (With strange exceptions, of course). You can choose where to vaccum, but not what gets sucked up the machine; likewise the companies.
Wednesday, February 29, 2012
Journal of Irreproduced results, vol. 1
I wanted to try to replicate and slightly expand Ted Underwood's recent discussion of genre formation over time using the Bookworm dataset of Open Library books. I couldn't, quite, but I want to just post the results and code up here for those who have been following that discussion. Warning: this is a rather dry post.
Monday, February 20, 2012
Downton Abbey Anachronisms, Season Finale edition
[Update: I've consolidated all of my TV anachronisms posts at a different blog, Prochronism, and new ones on Mad Men, Deadwood, Downton Abbey, and the rest are going there.]
It's Monday, so let's run last night's episode of Downton Abbey through the anachronism machine. I looked for Downton Abbey anachronisms for the first time last week: using the Google Ngram dataset, I can check every two-word phrase in an episode to see if it's more common today than then. This 1) lets us find completely anachronistic phrases, which is fun; and 2) lets us see how the language has evolved, and what shows do the best job at it. [Since some people care about this--don't worry, no plot spoilers below].
I'll start this with a chart of every two-word phrase that appears in the episode, just like last time. Left-to-right is overall frequency; top to bottom is over-representation. Higher up is representative of 1995 language; lower down, of 1917. Click to enlarge.
So: how does it look?
It's Monday, so let's run last night's episode of Downton Abbey through the anachronism machine. I looked for Downton Abbey anachronisms for the first time last week: using the Google Ngram dataset, I can check every two-word phrase in an episode to see if it's more common today than then. This 1) lets us find completely anachronistic phrases, which is fun; and 2) lets us see how the language has evolved, and what shows do the best job at it. [Since some people care about this--don't worry, no plot spoilers below].
I'll start this with a chart of every two-word phrase that appears in the episode, just like last time. Left-to-right is overall frequency; top to bottom is over-representation. Higher up is representative of 1995 language; lower down, of 1917. Click to enlarge.
So: how does it look?
Sunday, February 19, 2012
Second epistle to the intellectual historians
I. The new USIH blogger LD Burnett has a post up expressing ambivalence about the digital humanities because it is too eager to reject books. This is a pretty common argument, I think, familiar to me in less eloquent forms from New York Times comment threads. It's a rhetorically appealing position--to set oneself up as a defender of the book against the philistines who not only refuse to read it themselves, but want to take your books away and destroy them. I worry there's some mystification involved--conflating corporate publishers with digital humanists, lumping together books with codices with monographs, and ignoring the tension between reader and consumer. This problem ties up nicely into the big event in DH in the last week--the announcement of the first issue of the ambitiously all-digital Journal of Digital Humanities. So let me take a minute away from writing about TV shows to sort out my preliminary thoughts on books.
Monday, February 13, 2012
Making Downton more traditional
[Update: I've consolidated all of my TV anachronisms posts at a different blog, Prochronism, and new ones on Mad Men, Deadwood, Downton Abbey, and the rest are going there.]
Digital humanists like to talk about what insights about the past big data can bring. So in that spirit, let me talk about Downton Abbey for a minute. The show's popularity has led many nitpickers to draft up lists of mistakes. Language Loggers Mark Liberman and Ben Zimmer have looked at some idioms that don't belong for Language Log, NPR and the Boston Globe.) In the best British tradition, the Daily Mail even managed to cast the errors as a sort of scandal. But all of these have relied, so far as I can tell, on finding a phrase or two that sounds a bit off, and checking the online sources for earliest use. This resembles what historians do nowadays; go fishing in the online resources to confirm hypotheses, but never ever start from the digital sources. That would be, as the dowager countess, might say, untoward.
I lack such social graces. So I thought: why not just check every single line in the show for historical accuracy? Idioms are the most colorful examples, but the whole language is always changing. There must be dozens of mistakes no one else is noticing. Google has digitized so much of written language that I don't have to rely on my ear to find what sounds wrong; a computer can do that far faster and better. So I found some copies of the Downton Abbey scripts online, and fed every single two-word phrase through the Google Ngram database to see how characteristic of the English Language, c. 1917, Downton Abbey really is.
The results surprised me. There are, certainly, quite a few pure anachronisms. Asking for phrases that appear in no English-language books between 1912 and 1921 gives a list of 34 anachronistic phrases this season. Sorted from most to least common in contemporary books, we get a rather boring list:
Digital humanists like to talk about what insights about the past big data can bring. So in that spirit, let me talk about Downton Abbey for a minute. The show's popularity has led many nitpickers to draft up lists of mistakes. Language Loggers Mark Liberman and Ben Zimmer have looked at some idioms that don't belong for Language Log, NPR and the Boston Globe.) In the best British tradition, the Daily Mail even managed to cast the errors as a sort of scandal. But all of these have relied, so far as I can tell, on finding a phrase or two that sounds a bit off, and checking the online sources for earliest use. This resembles what historians do nowadays; go fishing in the online resources to confirm hypotheses, but never ever start from the digital sources. That would be, as the dowager countess, might say, untoward.
I lack such social graces. So I thought: why not just check every single line in the show for historical accuracy? Idioms are the most colorful examples, but the whole language is always changing. There must be dozens of mistakes no one else is noticing. Google has digitized so much of written language that I don't have to rely on my ear to find what sounds wrong; a computer can do that far faster and better. So I found some copies of the Downton Abbey scripts online, and fed every single two-word phrase through the Google Ngram database to see how characteristic of the English Language, c. 1917, Downton Abbey really is.
The results surprised me. There are, certainly, quite a few pure anachronisms. Asking for phrases that appear in no English-language books between 1912 and 1921 gives a list of 34 anachronistic phrases this season. Sorted from most to least common in contemporary books, we get a rather boring list:
Thursday, February 2, 2012
Poor man's sentiment analysis
Though I usually work with the Bookworm database of Open Library texts, I've been playing a bit more with the Google Ngram data sets lately, which have substantial advantages in size, quality, and time period. Largely I use it to check or search for patterns I can then analyze in detail with text-length data; but there's also a lot more that could be coming out of the Ngrams set than what I've seen in the last year.
Most humanists respond to the raw frequency measures in Google Ngrams with some bafflement. There's a lot to get excited about internally to those counts that can help answer questions we already have, but the base measure is a little foreign. If we want to know about the history of capitalism, the punctuated ascent of its Ngram only tells us so much:
It's certainly interesting that the steepest rises, in the 1930s and the 1970s, are associated with systematic worldwide crises--but that's about all I can glean from this, and it's one more thing than I get from most Ngrams. Usually, the game is just tracing individual peaks to individual events; a solitary quiz on historical events in front of the screen. Is this all the data can tell us?
Most humanists respond to the raw frequency measures in Google Ngrams with some bafflement. There's a lot to get excited about internally to those counts that can help answer questions we already have, but the base measure is a little foreign. If we want to know about the history of capitalism, the punctuated ascent of its Ngram only tells us so much:
Monday, January 30, 2012
Fixing the job market in two modest steps
Another January, another set of hand-wringing about the humanities job market. So, allow me a brief departure from the digital humanities. First, in four paragraphs, the problem with our current understanding of the history job market; and then, in several more, the solution.
Tony Grafton and Jim Grossman launched the latest exchange with what they call a "modest proposal" for expanding professional opportunities for historians. Jesse Lemisch counters that we need to think bigger and mobilize political action. There's a big and productive disagreement there, but also a deep similarity: both agree there isn't funding inside the academy for history PhDs to find work, but think we ought to be able to get our hands on money controlled by someone else. Political pressure and encouraging words will unlock vast employment opportunities in the world of museums, archives, and other public history (Grafton) or government funded jobs programs (Lemisch). These are funny places to look for growth in a 21st-century OECD country (perhaps Bill Cronon could take the more obvious route, and make his signature initiative as AHA president creating new tenure-track jobs in the BRICs?) but the higher levels of the profession don't see much choice but to change the world.
Tony Grafton and Jim Grossman launched the latest exchange with what they call a "modest proposal" for expanding professional opportunities for historians. Jesse Lemisch counters that we need to think bigger and mobilize political action. There's a big and productive disagreement there, but also a deep similarity: both agree there isn't funding inside the academy for history PhDs to find work, but think we ought to be able to get our hands on money controlled by someone else. Political pressure and encouraging words will unlock vast employment opportunities in the world of museums, archives, and other public history (Grafton) or government funded jobs programs (Lemisch). These are funny places to look for growth in a 21st-century OECD country (perhaps Bill Cronon could take the more obvious route, and make his signature initiative as AHA president creating new tenure-track jobs in the BRICs?) but the higher levels of the profession don't see much choice but to change the world.
Thursday, January 5, 2012
Practices, the periphery, and Pittsburg(h)
[This is not what I'll be saying at the AHA on Sunday morning, since I'm participating in a panel discussion with Stefan Sinclair, Tim Sherrat, and Fred Gibbs, chaired by Bill Turkel. Do come! But if I were to toss something off today to show how text mining can contribute to historical questions and what sort of issues we can answer, now, using simple tools and big data, this might be the story I'd start with to show how much data we have, and how little things can have different meanings at big scales...]
Spelling variations are not a bread-and-butter historical question, and with good reason. There is nothing at stake in whether someone writes "Pittsburgh" or "Pittsburg." But precisely because spelling is so arbitrary, we only change it for good reason. And so it can give insights into power, center and periphery, and transmission. One of the insights of cultural history is that the history of practices, however mundane, can be deeply rooted in the history of power and its use. So bear with me through some real arcana here; there's a bit of a payoff. Plus a map.
The set-up: until 1911, the proper spelling of Pittsburg/Pittsburgh was in flux. Wikipedia (always my go-to source for legalistic minutia) has an exhaustive blow-by-blow, but basically, it has to do with decisions in Washington DC, not Pittsburgh itself (which has usually used the 'h'). The city was supposedly mostly "Pittsburgh" to 1891, when the new US Board on Geographic Names made it firmly "Pittsburg;" then they changed their minds, and made it and once again and forevermore "Pittsburgh" from 1911 on. This is kind of odd, when you think about it: the government changed the name of the eighth-largest city in the country twice in twenty years. (Harrison and Taft are not the presidents you usually think of as kings of over-reach). But it happened; people seem to have changed the addresses on their envelopes, the names on their baseball uniforms, and everything else right on cue.
Thanks to about 500,000 books from the Open Library, though, we don't have to accept this prescriptive account as the whole story; what did people actually do when they had to write about Pittsburgh?
Here's the usage in American books:
What does this tell us about how practices change?
Spelling variations are not a bread-and-butter historical question, and with good reason. There is nothing at stake in whether someone writes "Pittsburgh" or "Pittsburg." But precisely because spelling is so arbitrary, we only change it for good reason. And so it can give insights into power, center and periphery, and transmission. One of the insights of cultural history is that the history of practices, however mundane, can be deeply rooted in the history of power and its use. So bear with me through some real arcana here; there's a bit of a payoff. Plus a map.
The set-up: until 1911, the proper spelling of Pittsburg/Pittsburgh was in flux. Wikipedia (always my go-to source for legalistic minutia) has an exhaustive blow-by-blow, but basically, it has to do with decisions in Washington DC, not Pittsburgh itself (which has usually used the 'h'). The city was supposedly mostly "Pittsburgh" to 1891, when the new US Board on Geographic Names made it firmly "Pittsburg;" then they changed their minds, and made it and once again and forevermore "Pittsburgh" from 1911 on. This is kind of odd, when you think about it: the government changed the name of the eighth-largest city in the country twice in twenty years. (Harrison and Taft are not the presidents you usually think of as kings of over-reach). But it happened; people seem to have changed the addresses on their envelopes, the names on their baseball uniforms, and everything else right on cue.
Thanks to about 500,000 books from the Open Library, though, we don't have to accept this prescriptive account as the whole story; what did people actually do when they had to write about Pittsburgh?
Here's the usage in American books:
What does this tell us about how practices change?
Friday, December 16, 2011
Genre similarities
When data exploration produces Christmas-themed charts, that's a sign it's time to post again. So here's a chart and a problem.
First, the problem. One of the things I like about the posts I did on author age and vocabulary change in the spring is that they have two nice dimensions we can watch changes happening in. This captures the fact that language as a whole doesn't just up and change--things happen among particular groups of people, and the change that results has shape not just in time (it grows, it shrinks) but across those other dimensions as well.
There's nothing fundamental about author age for this--in fact, I think it probably captures what, at least at first, I would have thought were the least interesting types of vocabulary change. But author age has two nice characteristics.
1) It's straightforwardly linear, and so can be set against publication year cleanly.
2) Librarians have been keeping track of it, pretty much accidentally, by noting the birth year of every book's author.
Neither of these attributes are that remarkable; but the combination is.
Subscribe to:
Posts (Atom)