Importing¶
We import the nltk (natural languages toolkit) package with import, like the other Python packages we use in the SWC lesson
import nltk
Using nltk¶
Downloading example datasets¶
We almost immediately depart from the SWC lesson, because we need to deal with specific functions of nltk (as opposed to general programming principles). However, what we learned in the SWC lesson is still relevant, here.
We access functions in the nltk package with dotted notation, just like the functions we saw in matplotlib. The first function we'll use is one that downloads text corpora, so we have some examples to work with.
This function is nltk.download(), and we can pass it the name of a specific corpus, such as gutenberg.
Downloads may take a little time…
nltk.download('gutenberg')
Working with corpus file lists¶
The corpus examples from nltk are accessed using dotted notation in the same way as in the lesson, like the pyplot package from matplotlib - matplotlib.pyplot. One important difference is that we need to use nltk-specific functions.
We can use the dotted notation to get a list of nltk's example Gutenberg corpus datasets, with the fileids() function:
files = nltk.corpus.gutenberg.fileids()
print(files)
This returns a list, which behaves in the same way as the other lists we saw in the lesson, so we can select individual files or slice the list as in the cells below:
# Use indexing to get the fourth item in the list
print(files[3])
# Use slicing to get the first three files in the list
print(files[:3])
We can also use a for loop to identify all the Chesterton texts:
# Get a list of texts by Chesterton
ch_files = [] # empty list to hold filenames
for file in files: # look at each filename in turn
if 'chesterton' in file: # test if it's a Chesterton
ch_files.append(file) # add file to the list
print(ch_files)
Working with file contents¶
We can identify and subset lists of files, but at some point we want to work with the text itself. The way that nltk does this is specific to the package, and so not suitable for the general SWC lesson, but we can use what we learned in SWC to carry out common tasks.
It is common in nltk to import a corpus directly - this helps reduce typing. For the Gutenberg corpus, this would be:
from nltk.corpus import gutenberg
Now we can refer to items in the gutenberg corpus with dotted notation, as below:
print(gutenberg.fileids())
Each corpus also provides functions, again accessed by dotted notation, which can be used, for example to get data from or analyse each text.
So, to get all the words in Jane Austen's Emma, we can use the gutenberg.words() function:
gutenberg.words('austen-emma.txt')
This looks like a list, but isn't. We can use the type() function that we met in the SWC lesson to see what datatype it is:
emma_words = gutenberg.words('austen-emma.txt')
type(emma_words)
This is a datatype that is specific to nltk (and so not in the scope of SWC), but we can do many of the things we would for lists, as in the cell below:
# How many tokens in the text:
print("Token count:", len(emma_words))
# What is the token at index 1000?
print("token at index 1000:", emma_words[1000])
# Slice from token 1400 to 1500
print("slice from 1400 to 1500:", emma_words[1400:1500])
Visualisation¶
We can plot statistics relating to the text, using matplotlib and seaborn, as we did in the SWC lesson.
Firstly, we'll take the length of all tokens in the text and put that information in a list called token_len (and print the first few values):
token_len = []
for token in emma_words:
token_len.append(len(token))
print("token_len:", token_len[:20])
Next, we'll import modules for visualisation:
%matplotlib inline
import matplotlib.pyplot
import seaborn
Now we can plot summary statistics of the data - such as a distribution plot, to see what the most common token length is:
seaborn.distplot(token_len)
By using for loops, we can make comparisons over all texts in a corpus - here, text length:
text_len = []
for text in gutenberg.fileids():
text_len.append(len(gutenberg.words(text)))
seaborn.distplot(text_len)
seaborn.plt.xlim(0, 1e6) # set x-axis limits
An nltk analysis example¶
This sort of 'analysis' is rather simplistic, and nltk provides more meaningful analyses that are accessed through the nltk functions. These have particular syntax and expect a specific kind of input.
For example, nltk will calculate the conditional frequency distribution of target words, using the class (which we don't cover in SWC!) nltk.ConditionalFreqDist.
To see what it requires to work, we can use the help() function, as below - luckily, the nltk documentation is quite thorough, if a little technical:
#help(nltk.ConditionalFreqDist)
What nltk.ConditionalFreqDist needs is a list of tuples (which we don't cover in SWC!).
Tuples are ordered immutable collections - a little like lists, except they cannot be modified once created. We make them in the same way as we make lists, except that we use parentheses instead of square brackets:
# list
my_list = [1, 2, 3]
# tuple
my_tuple = (1, 2, 3)
print("list:", my_list)
print("tuple:", my_tuple)
More specifically, what nltk.ConditionalFreqDist needs is a list of tuples where each tuple is of the form:
(<word>, <text identifier>)
such as
('Vogon', 'Adams')
or
('Vogon', 'Sci-Fi')
The identifier is presumably going to depend on the focus of your study, and a list of these could look like:
[('Vogon', 'Sci-Fi'), ('Orc', 'Fantasy'), ('Marlowe', 'Crime'), …]
Let's say we want to compare occurrences of man or woman (but not men or women) in the Gutenberg texts.
To get our list of tuples, we need to preprocess the data a little.
We're going to analyse all the texts in the Gutenberg corpus, so we loop over each text, and get the words with the .words() function:
for text in gutenberg.fileids():
words = gutenberg.words(text)
To buid this up for our analysis - for each text, we're going to filter for words that are in a list we define: ['man', 'woman']. These words are lower-case, so we'll have to force the words from the text to lower-case, too.
We'll catch the words in a list called filtered, and we'll catch them as tuples, of the form (<word>, <fileid>):
targets = ['man', 'woman']
filtered = []
for text in gutenberg.fileids():
words = gutenberg.words(text)
for word in words:
word_lc = word.lower() # word.lower() converts the word to lower case
if word_lc in targets:
filtered.append((word_lc, text))
targets = ['man', 'woman']
filtered = []
for text in gutenberg.fileids():
words = gutenberg.words(text)
for word in words:
word_lc = word.lower() # word.lower() converts the word to lower case
if word_lc in targets:
filtered.append((word_lc, text))
Let's look at our data:
print(filtered[:20])
This looks to be correct, so we can use it as input to nltk.ConditionalFreqDist as below:
data = nltk.ConditionalFreqDist(filtered)
print(data)
This kind of object is unique to nltk, but we can access it using square bracket notation like other objects in Python.
For this kind of object, we put the word we're interested in into the square brackets, like:
data['man']
# Get frequency count of a word
data['woman']
The nltk developers have been kind, and provided a plot() function in that object which, like so much else in Python, we can access with dotted notation to get a graphical representation:
# Plot conditional frequency data
data.plot()
Making this into a function¶
We can wrap this nltk analysis into a function, where we pass the target words we'd like to plot, and the function produces a graph, as above.
We'll call the function plot_cfd(), and have it take as its first argument a list of words. As a second optional argument, we can pass a corpus - but we'll make this default to gutenberg.
One subtlety here is that we want to match the case of words from the text, and the targets, so we convert all the target words to lower-case, too.
def plot_cfd(targets, corpus=gutenberg):
"""Plot conditional frequency graph of target words.
Plots a conditional frequency graph of the target words in
a corpus.
- targets a list of target words
- corpus the corpus module from nltk (optional)
default value: gutenberg
"""
filtered = []
targets_lc = []
for target in targets: # make targets lower-case
targets_lc.append(target.lower())
for text in corpus.fileids():
words = corpus.words(text)
for word in words:
word_lc = word.lower()
if word_lc in targets_lc:
filtered.append((word_lc, text))
data = nltk.ConditionalFreqDist(filtered)
data.plot()
Now we have the function, we can use it with arbitrary lists of target words, as below:
plot_cfd(['I', 'you', 'he', 'she', 'they', 'it'])
plot_cfd(['from', 'to'])
plot_cfd(['love', 'hate'])
# Download Indian corpus
nltk.download('indian')
Unicode representation is native in Python3, and viewable in Jupyter:
# Import Indian corpus
from nltk.corpus import indian
# Show words from each text
for text in indian.fileids():
print(text, indian.words(text)[:30])