This is a Unix-style workflow for topic modeling a personal library of PDFs using MALLET. It should work on macOS and Linux.
If you find something in these instructions unclear, or something doesn't work for you, or you have a suggestion for improving this workflow, please file an issue.
These instructions assume some basic familiarity with the macOS terminal or Linux shell.
Depending on how many PDFs you have, you may need a computer that has a good amount of RAM and a CPU with multiple cores. On my 2019 laptop with a 8 core Intel CPU and 16GB of RAM, building and visualizing a 50-topic model of around 5,000 PDFs, having a total of about 20 million words, takes about 15 minutes. Building and visualizing a 200-topic model takes about an hour. (Most of that time is spent generating the visualization, not building the topic model.)
If you have more PDFs than that, or don't have that many cores or that much RAM, you might want to run this on an on-demand cloud compute server—but doing that is beyond the scope of these instructions.
Wherever you run this, you will need recent versions of the following software: GNU make (version 4.3 or later), Python (version 3.8 or later), pdftotext, git, and ant.
(It probably will work with an earlier version of Python 3, but I
haven't tested this. If you want to try it with an earlier version,
set the value of NEED_PYTHON_VERSION in the Makefile to that
version, and then please let me know if it works by filing an
issue.)
On macOS, all of these can be installed using Homebrew. After installing Homebrew, execute the following command in the terminal to install all the prerequisites:
brew install make python3 poppler git ant
(poppler is the name of the package that installs pdftotext.)
On Ubuntu Linux 20.10 (Groovy Gorilla) or later, these prerequisites
can be installed using apt-get:
sudo apt-get install make python3 python3-venv poppler-utils git ant
Note that on Ubuntu you must install python3-venv in addition to
python3.
Either clone this repository, or download the ZIP file of
this repository and unzip it someplace. Either way you should end up
with a directory called topics on your computer.
Visit the topics directory in your terminal or shell:
cd /wherever/you/put/topics
Create a subdirectory called pdf:
mkdir pdf
Then you can put all your PDFs in this subdirectory. If you don't want
to disturb the organization of your PDFs, you can leave them where
they are and use symbolic links to point to them. For example, if
you keep your PDFs in ~/Documents/Papers and ~/Desktop/TO-READ
cd pdf
ln -s ~/Documents/Papers
ln -s ~/Desktop/Papers
The directory structure inside pdf (or inside the directories
pointed to by symbolic links within pdf) doesn't matter. Any files
ending in .pdf, no matter how deeply nested, will be found and
indexed and all other files will be ignored.
If there are PDFs you want to exclude from your topic models, you can
create a file called exclude.txt in the topics directory, and list
patterns there matching any filenames you want to exclude. For example
if your exclude.txt file contained:
michel foucault
draft
… then any PDFs with filenames containing the name “Michel Foucault” or the word “draft” will be excluded. You can also use basic regular expressions. Matching is case-insensitive.
Assuming you’re still visiting the topics directory in your terminal
or shell, you can build a topic model with 50 topics with the command:
./build 50-topics
This will
- convert all your PDFs to plain text files in a subdirectory called
txt - download and install the MALLET topic modeling software
- build a topic model with 50 topics
- create an interactive visualization of the topics using pyLDAvis
- create a set of web pages for browsing the model
50 can be changed to whatever you want; to build a model with 111
topics use the command:
./build 111-topics
“Hyperparameter optimization” is a MALLET option that allows some
topics to be more prominent than others, which can sometimes result in
better models. To build an optimized model, put the word -optimized
after the number of topics you want:
./build 50-optimized-topics
You can build multiple topic models with one command (each will be built sequentially):
./build 50-topics 50-optimized-topics 100-topics 100-optimized-topics
By default MALLET will use 12G of RAM; if you don't have that much (or you have more) you can specify how much RAM to use:
MEMORY=8g ./build 50-topics
Note that the less RAM MALLET uses, the more time topic modeling will take (and the more it uses, the less time it will take).
Run a local web server for browsing the topic models with the command:
make serve
This will run a server on http://localhost:5555, which you can visit with a web browser. If something else is already running on port 5555, you can change the port:
PORT=12345 make serve
When you visit http://localhost:5555 you'll see links to each of the models you've built. For each model, you can see
- lists of the top documents per topic
- a visualization of the topic structure
- diagnostics tool for troubleshooting the model
The lists of the top documents per topic show the strength of association with the topic in a column along the left side of the page. Clicking on a document in the list will open it in the browser. For documents that are in the list of top documents for more than one topic, links to the other topics appear after the link to the document.
You can find a specific document in the models you've built by using
the topics shell script in the topics directory. Give it the path
to a PDF, and it will print the URL of and (if it can) open in a
browser every top documents list in which that document appears:
./topics pdf/Desktop/TO-READ/an-interesting-article.pdf
Note that if you change the port on which the local web server runs
from the default of 5555, you'll need to rebuild the shell scripts
for finding specific documents. Assuming you've built topic models
with 50, 100, and 200 topics, you can delete and rebuild the shell
scripts as follows:
rm -f topics 50-topics 100-topics 200-topics
./build 50-topics 100-topics 200-topics
For an explanation of the topic structure visualization, see the pyLDAvis documentation.
For an explanation of the diagnostics tool, see the MALLET documentation.
If you add new PDFs to your PDF directories, you'll need to rebuild your topic models. To do this use the command:
./rebuild 50-optimized-topics
This will delete all existing models and then build an optimized 50-topic model.
If you only want to rebuild a specific model, without deleting all existing models, you can delete just that model and then build it:
./delete 50-topics
./build 50-topics
Note that if you've added new PDFs, only the models that are rebuilt
will incorporate them, so most of the time you're going to want to use
rebuild to rebuild all of your models.
These commands (rebuild or delete followed by build) will only
convert the new PDFs to plain text, skipping any PDFs that have
already been converted. If you want to start completely from scratch,
including re-converting all your PDFs and re-installing MALLET and
pyLDAvis, use the command:
make superclean 50-topics
Note that you will not get the same set of topics when you rebuild your models! This is expected: topic modeling involves random sampling, which produces different (but comparable) results each time.
For exploring a library of PDFs, this is a feature, not a bug: you can rebuild your topic models several times, looking to see what kinds of interesting clusters are turned up.
However, if you need reproducibility, when you build your topic models you can specify a random seed for MALLET to use as follows:
RANDOM_SEED=7 ./build 50-topics
The value of RANDOM_SEED can be any positive integer (not zero).
If you use the same RANDOM_SEED value every time you build the
models, you should get the same results each time.
Some PDFs, such as scanned PDFs that have not had OCR run on them, may not have any readable text in them, meaning that the plain text versions of these files will be empty. You can check if this is the case for any of your PDF files by running this command:
make wordcounts.csv
This will generate a CSV file listing the word count for each plain text file created from your PDFs. Open the CSV in the spreadsheet software of your choice and sort by word count. Files with zero or very few words should be checked to see if the PDF needs to have OCR run on it.
You may find that one or more of your topics seems to consist of gibberish. This is usually a sign that OCR has failed, producing a bunch of unreadable symbols instead of readable text. This is often due to a problem with embedded fonts in the PDF file. Using Adobe Acrobat Pro DC to convert fonts to outlines before running OCR can fix this.
You can use the diagnostics tool to find other kinds of “problematic” topics—see the MALLET documentation.