The Google Ngram Viewer<\/a> is a database browser used to chart the relative frequency of words or phrases. The data source is the Google Books database (sort of) and the graphic engine is Google Charts<\/a>. It’s cool. It’s pretty. It’s hard to use for academic work because it doesn’t easily give up the raw data.<\/p>\n This page explains how to run a script \u2014 ngram.sh \u2014 on a *NIX shell account and extract your keywords into spreadsheet-ready files, without<\/em> buying a terabyte hard drive.<\/p>\n Before mapping out your research project, you should spend some time testing out search terms in the Google Ngram Viewer<\/a>, reading this page’s bibliography<\/a> and selecting one of the Ngram datasets<\/a>.<\/p>\n By default, ngram.sh is configured for 1gram searches of the English Version 20090715 dataset. It will download ten ~210mb ZIP files, one at a time, unpack the ~1000mb CSV inside, and (grep) search for each keyword. It will write the output to a keyword-named CSV, and then delete the source file. You must have a MINIMUM 2gb to run the default script (sorry, your SFU diskshare isn’t big enough).<\/p>\n This script can take quite a while to run. In part that’s because it’s searching large amounts of data, but mostly it’s because it’s downloading<\/em> lots of data. Unless you’re running on a very fast Internet connection, bandwidth is your bottleneck. You can speed things up for future searches by removing the lines to delete source (or CSV) files. You could then modify the script to run off your hard drive without downloading anew. However, you must have you must have upwards of 10gb available to do this with 1grams. If you change the script to process 2-grams or higher, WATCH OUT! A multi-keyword search of 5-grams without deletes can easily top a terabyte of data!<\/p>\n It is really, REALLY easy to fill every last sector of your hard drive \u2014 or bust your bandwidth cap! \u2014 with this script. Be cautious and pay attention<\/strong>.<\/p>\n You can run this script on any UNIX or *NIX system, including your OSX or higher Apple\/Mac personal computer<\/a> (YouTube).<\/p>\n Download ngram.sh<\/a><\/strong>. The script will download after you click through the registration.<\/p>\n Permissions: Some people like to run their scripts with “bash ngram.sh<\/em>” and file permissions unchanged, others prefer to set executable permissions with chmod 755 and run with “.\/ngram.sh<\/em>“.<\/p>\n Configuration: The very first line will need to be edited with your path to bash. Replace the 1grams on line 41 (beginning “for word in”) with your search keywords. Save and run the script.<\/p>\n Cleaning the results: I like to manipulate the CSVs in MS Excel, but any spreadsheet application will do (even GoogleDocs). When reading the results of your query, you’re likely to discover that you grabbed a bunch of words you didn’t intend to, will have to collapse a bunch that are similar, and might have missed a few that you wanted. Keyword selection is a science and an art. Just modify your script, set it to run, make yourself some tea and hope you don’t cause your ISP to blow a gasket.<\/p>\n Happy counting!<\/p>\n <\/a><\/p>\n Bentley, J., & Knuth, D. (1986). Programming pearls: literate programming<\/a>. Communications of the Association of Computing Machinery<\/em>, 29<\/em>(5), 384-369. doi:10.1145\/5689.315644<\/p>\n Bentley, J., Knuth, D., & McIlroy, D. (1986). Programming pearls: a literate program<\/a>. Communications of the Association of Computing Machinery<\/em>, 29<\/em>(6), 471-483. doi:10.1145\/5948.315654<\/p>\n Lieberman, E., Michel, J.-B., Jackson, J., Tang, T., & Nowak, M. A. (2007). Quantifying the evolutionary dynamics of language<\/a>. Nature, 449(7163), 713-716. doi:10.1038\/nature06137<\/p>\n Michel, J.-B., Shen, Y. K., Aiden, A. P., Veres, A., Gray, M. K., The Google Books Team, Pickett, J. P., et al. (2010). Quantitative Analysis of Culture Using Millions of Digitized Books<\/a>. Science. doi:10.1126\/science.1199644<\/p>\nHow ngram.sh works<\/h2>\n
How to run ngram.sh<\/h2>\n
bibliography<\/h2>\n