# Read text files with readtext()

    # Load readtext package
    library(readtext)

# 1. Introduction

The vignette walks you through importing a variety of different text files into R using the readtext package. Currently, readtext supports plain text files (.txt), data in some form of JavaScript Object Notation (.json), comma-or tab-separated values (.csv, .tab, .tsv), XML documents (.xml), as well as PDF and Microsoft Word formatted files (.pdf, .doc, .docx).

readtext also handles multiple files and file types using for instance a “glob” expression, files from a URL or an archive file (.zip, .tar, .tar.gz, .tar.bz). Usually, you do not have to determine the format of the files explicitly - readtext takes this information from the file ending.

The readtext package comes with a data directory called `extdata` that contains examples of all files listed above. In the vignette, we use this data directory.

    # Get the data directory from readtext
    DATA_DIR <- system.file("extdata/", package = "readtext")

The `extdata` directory contains several subfolders that include different text files. In the following examples, we load one or more files stored in each of these folders. The `paste0` command is used to concatenate the `extdata` folder from the readtext package with the subfolders. When reading in custom text files, you will need to determine your own data directory (see `?setwd()`).

# 2. Reading one or more text files

## 2.1 Plain text files (.txt)

The folder “txt” contains a subfolder named UDHR with .txt files of the Universal Declaration of Human Rights in 13 languages.

    # Read in all files from a folder
    readtext(paste0(DATA_DIR, "/txt/UDHR/*"))
    ## readtext object consisting of 13 documents and 0 docvars.
    ## # data.frame [13 × 2]
    ##   doc_id            text
    ##   <chr>             <chr>
    ## 1 UDHR_chinese.txt  "\"世界人权宣言\n联合国\"..."
    ## 2 UDHR_czech.txt    "\"VŠEOBECNÁ \"..."
    ## 3 UDHR_danish.txt   "\"Den 10. de\"..."
    ## 4 UDHR_english.txt  "\"Universal \"..."
    ## 5 UDHR_french.txt   "\"Déclaratio\"..."
    ## 6 UDHR_georgian.txt "\"FLFVBFYBC \"..."
    ## # ... with 7 more rows

We can specify document-level metadata (`docvars`) based on the file names or on a separate data.frame. Below we take the docvars from the filenames (`docvarsfrom = "filenames"`) and set the names for each variable (`docvarnames = c("unit", "context", "year", "language", "party")`). The command `dvsep = "_"` determines the separator (a regular expression character string) included in the filenames to delimit the `docvar` elements.

    # Manifestos with docvars from filenames
    readtext(paste0(DATA_DIR, "/txt/EU_manifestos/*.txt"),
             docvarsfrom = "filenames",
             docvarnames = c("unit", "context", "year", "language", "party"),
             dvsep = "_",
             encoding = "ISO-8859-1")
    ## readtext object consisting of 17 documents and 5 docvars.
    ## # data.frame [17 × 7]
    ##   doc_id                  text          unit  context  year language party
    ##   <chr>                   <chr>         <chr> <chr>   <int> <chr>    <chr>
    ## 1 EU_euro_2004_de_PSE.txt "\"PES · PSE… EU    euro     2004 de       PSE
    ## 2 EU_euro_2004_de_V.txt   "\"Gemeinsam… EU    euro     2004 de       V
    ## 3 EU_euro_2004_en_PSE.txt "\"PES · PSE… EU    euro     2004 en       PSE
    ## 4 EU_euro_2004_en_V.txt   "\"Manifesto… EU    euro     2004 en       V
    ## 5 EU_euro_2004_es_PSE.txt "\"PES · PSE… EU    euro     2004 es       PSE
    ## 6 EU_euro_2004_es_V.txt   "\"Manifesto… EU    euro     2004 es       V
    ## # ... with 11 more rows

readtext can also curse through subdirectories. In our example, the folder `txt/movie_reviews` contains two subfolders (called `neg` and `pos`). We can load all texts included in both folders.

    # Recurse through subdirectories
    readtext(paste0(DATA_DIR, "/txt/movie_reviews/*"))
    ## readtext object consisting of 10 documents and 0 docvars.
    ## # data.frame [10 × 2]
    ##   doc_id              text
    ##   <chr>               <chr>
    ## 1 neg_cv000_29416.txt "\"plot : two\"..."
    ## 2 neg_cv001_19502.txt "\"the happy \"..."
    ## 3 neg_cv002_17424.txt "\"it is movi\"..."
    ## 4 neg_cv003_12683.txt "\" \" quest f\"..."
    ## 5 neg_cv004_12641.txt "\"synopsis :\"..."
    ## 6 pos_cv000_29590.txt "\"films adap\"..."
    ## # ... with 4 more rows

## 2.2 Comma- or tab-separated values (.csv, .tab, .tsv)

Read in comma separted values (.csv files) that contain textual data. We determine the `texts` variable in our .csv file as the `text_field`. This is the column that contains the actual text. The other columns of the original csv file (`Year`, `President`, `FirstName`) are by default treated as document-level variables.

    # Read in comma-separated values
    readtext(paste0(DATA_DIR, "/csv/inaugCorpus.csv"), text_field = "texts")
    ## readtext object consisting of 5 documents and 3 docvars.
    ## # data.frame [5 × 5]
    ##   doc_id            text                 Year President  FirstName
    ##   <chr>             <chr>               <int> <chr>      <chr>
    ## 1 inaugCorpus.csv.1 "\"Fellow-Cit\"..."  1789 Washington George
    ## 2 inaugCorpus.csv.2 "\"Fellow cit\"..."  1793 Washington George
    ## 3 inaugCorpus.csv.3 "\"When it wa\"..."  1797 Adams      John
    ## 4 inaugCorpus.csv.4 "\"Friends an\"..."  1801 Jefferson  Thomas
    ## 5 inaugCorpus.csv.5 "\"Proceeding\"..."  1805 Jefferson  Thomas

The same procedure applies to tab-separated values.

    # Read in tab-separated values
    readtext(paste0(DATA_DIR, "/tsv/dailsample.tsv"), text_field = "speech")
    ## readtext object consisting of 33 documents and 9 docvars.
    ## # data.frame [33 × 11]
    ##   doc_id  text   speechID memberID partyID constID title date  member_name
    ##   <chr>   <chr>     <int>    <int>   <int>   <int> <chr> <chr> <chr>
    ## 1 dailsa… "\"Mo…        1      977      22     158 1. C… 1919… Count Geor…
    ## 2 dailsa… "\"Is…        2     1603      22     103 1. C… 1919… Mr. Pádrai…
    ## 3 dailsa… "\"' …        3      116      22     178 1. C… 1919… Mr. Cathal…
    ## 4 dailsa… "\"Tá…        4      116      22     178 2. C… 1919… Mr. Cathal…
    ## 5 dailsa… "\"Lé…        5      116      22     178 3. A… 1919… Mr. Cathal…
    ## 6 dailsa… "\"-B…        6      116      22     178 3. A… 1919… Mr. Cathal…
    ## # ... with 27 more rows, and 2 more variables: party_name <chr>,
    ## #   const_name <chr>

## 2.3 JSON data (.json)

You can also read .json data. Again you need to specify the `text_field`.

    ## Read in JSON data
    readtext(paste0(DATA_DIR, "/json/inaugural_sample.json"), text_field = "texts")
    ## readtext object consisting of 3 documents and 3 docvars.
    ## # data.frame [3 × 5]
    ##   doc_id                  text                 Year President  FirstName
    ##   <chr>                   <chr>               <int> <chr>      <chr>
    ## 1 inaugural_sample.json.1 "\"Fellow-Cit\"..."  1789 Washington George
    ## 2 inaugural_sample.json.2 "\"Fellow cit\"..."  1793 Washington George
    ## 3 inaugural_sample.json.3 "\"When it wa\"..."  1797 Adams      John

## 2.4 PDF files

readtext can also read in and convert .pdf files.

In the example below we load all .pdf files stored in the `UDHR` folder, and determine that the `docvars` shall be taken from the filenames. We call the document-level variables `document` and `language`, and specify the delimiter (`dvsep`).

    ## Read in Universal Declaration of Human Rights pdf files
    (rt_pdf <- readtext(paste0(DATA_DIR, "/pdf/UDHR/*.pdf"),
                        docvarsfrom = "filenames",
                        docvarnames = c("document", "language"),
                        sep = "_"))
    ## readtext object consisting of 11 documents and 2 docvars.
    ## # data.frame [11 × 4]
    ##   doc_id           text                          document language
    ##   <chr>            <chr>                         <chr>    <chr>
    ## 1 UDHR_chinese.pdf "\"世界人权宣言\n联合国\"..." UDHR     chinese
    ## 2 UDHR_czech.pdf   "\"VŠEOBECNÁ \"..."           UDHR     czech
    ## 3 UDHR_danish.pdf  "\"Den 10. de\"..."           UDHR     danish
    ## 4 UDHR_english.pdf "\"Universal \"..."           UDHR     english
    ## 5 UDHR_french.pdf  "\"Déclaratio\"..."           UDHR     french
    ## 6 UDHR_greek.pdf   "\"ΟΙΚΟΥΜΕΝΙΚ\"..."           UDHR     greek
    ## # ... with 5 more rows

## 2.5 Microsoft Word files (.doc, .docx)

Microsoft Word formatted files are converted through the package antiword for older `.doc` files, and using XML for newer `.docx` files.

    ## Read in Word data (.docx)
    readtext(paste0(DATA_DIR, "/word/*.docx"))
    ## readtext object consisting of 2 documents and 0 docvars.
    ## # data.frame [2 × 2]
    ##   doc_id                      text
    ##   <chr>                       <chr>
    ## 1 UK_2015_EccentricParty.docx "\"The Eccent\"..."
    ## 2 UK_2015_LoonyParty.docx     "\"The Offici\"..."

## 2.6 Text from URLs

You can also read in text directly from a URL.

    # Note: Example required: which URL should we use?

## 2.7 Text from archive files (.zip, .tar, .tar.gz, .tar.bz)

Finally, it is possible to inclue text from archives.

    # Note: Archive file required. The only zip archive included in readtext has
    # different encodings and is difficult to import (see section 4.2).

# 3. Inter-operability with quanteda

readtext was originally developed in early versions of the quanteda package for the quantitative analysis of textual data. It was spawned from the `textfile()` function from that package, and now lives exclusively in readtext. Because quanteda’s corpus constructor recognizes the data.frame format returned by `readtext()`, it can construct a corpus directly from a `readtext` object, preserving all docvars and other meta-data.

    require(quanteda)

You can easily contruct a corpus from a readtext object.

    # read in comma-separated values with readtext
    rt_csv <- readtext(paste0(DATA_DIR, "/csv/inaugCorpus.csv"), text_field = "texts")

    # create quanteda corpus
    corpus_csv <- corpus(rt_csv)
    summary(corpus_csv, 5)
    ## Corpus consisting of 5 documents, showing 5 documents:
    ##
    ##               Text Types Tokens Sentences Year  President FirstName
    ##  inaugCorpus.csv.1   625   1540        23 1789 Washington    George
    ##  inaugCorpus.csv.2    96    147         4 1793 Washington    George
    ##  inaugCorpus.csv.3   826   2578        37 1797      Adams      John
    ##  inaugCorpus.csv.4   717   1927        41 1801  Jefferson    Thomas
    ##  inaugCorpus.csv.5   804   2381        45 1805  Jefferson    Thomas
    ##
    ## Source: /private/var/folders/46/zfn6gwj15d3_n6dhyy1cvwc00000gp/T/RtmpbFlOkN/Rbuild20971d2a3cb8/readtext/vignettes/* on x86_64 by kbenoit
    ## Created: Thu May 10 22:43:14 2018
    ## Notes:

# 4. Solving common problems

## 4.1 Remove page numbers using regular expressions

When a document contains page numbers, they are imported as well. If you want to remove them, you can use a regular expression. We strongly recommend using the stringi package. For the most common regular expressions you can look at this cheatsheet.

You first need to check in the original file in which format the page numbers occur (e.g., “1”, “-1-”, “page 1” etc.). We can make use of the fact that page numbers are almost always preceded and followed by a linebreak (`\n`). After loading the text with readtext, you can replace the page numbers.

    # Load stringi package
    require(stringi)

In the first example, the page numbers have the format “page X”.

    # Make some text with page numbers
    sample_text_a <- "The quick brown fox named Seamus jumps over the lazy dog also named Seamus,
    page 1
    with the newspaper from a boy named quick Seamus, in his mouth.
    page 2
    The quicker brown fox jumped over 2 lazy dogs."

    sample_text_a
    ## [1] "The quick brown fox named Seamus jumps over the lazy dog also named Seamus, \npage 1 \nwith the newspaper from a boy named quick Seamus, in his mouth.\npage 2\nThe quicker brown fox jumped over 2 lazy dogs."

    # Remove "page" and respective digit
    sample_text_a2 <- unlist(stri_split_fixed(sample_text_a, '\n'), use.names = FALSE)
    sample_text_a2 <- stri_replace_all_regex(sample_text_a2, "page \\d*", "")
    sample_text_a2 <- stri_trim_both(sample_text_a2)
    sample_text_a2 <- sample_text_a2[sample_text_a2 != '']
    stri_paste(sample_text_a2, collapse = '\n')
    ## [1] "The quick brown fox named Seamus jumps over the lazy dog also named Seamus,\nwith the newspaper from a boy named quick Seamus, in his mouth.\nThe quicker brown fox jumped over 2 lazy dogs."

In the second example we remove page numbers which have the format “- X -”.

    sample_text_b <- "The quick brown fox named Seamus
    - 1 -
    jumps over the lazy dog also named Seamus, with
    - 2 -
    the newspaper from a boy named quick Seamus, in his mouth.
    - 33 -
    The quicker brown fox jumped over 2 lazy dogs."

    sample_text_b
    ## [1] "The quick brown fox named Seamus \n- 1 - \njumps over the lazy dog also named Seamus, with \n- 2 - \nthe newspaper from a boy named quick Seamus, in his mouth. \n- 33 - \nThe quicker brown fox jumped over 2 lazy dogs."

    sample_text_b2 <- unlist(stri_split_fixed(sample_text_b, '\n'), use.names = FALSE)
    sample_text_b2 <- stri_replace_all_regex(sample_text_b2, "[-] \\d* [-]", "")
    sample_text_b2 <- stri_trim_both(sample_text_b2)
    sample_text_b2 <- sample_text_b2[sample_text_b2 != '']
    stri_paste(sample_text_b2, collapse = '\n')
    ## [1] "The quick brown fox named Seamus\njumps over the lazy dog also named Seamus, with\nthe newspaper from a boy named quick Seamus, in his mouth.\nThe quicker brown fox jumped over 2 lazy dogs."

Such stringi functions can also be applied to readtext objects.

## 4.2 Read files with different encodings

Sometimes files of the same type have different encodings. If the encoding of a file is included in the file name, we can extract this information and import the texts correctly.

    # create a temporary directory to extract the .zip file
    FILEDIR <- tempdir()
    # unzip file
    unzip(system.file("extdata", "data_files_encodedtexts.zip", package = "readtext"), exdir = FILEDIR)

Here, we will get the encoding from the filenames themselves.

    # get encoding from filename
    filenames <- list.files(FILEDIR, "^(Indian|UDHR_).*\\.txt$")

    head(filenames)
    ## [1] "IndianTreaty_English_UTF-16LE.txt"
    ## [2] "IndianTreaty_English_UTF-8-BOM.txt"
    ## [3] "UDHR_Arabic_ISO-8859-6.txt"
    ## [4] "UDHR_Arabic_UTF-8.txt"
    ## [5] "UDHR_Arabic_WINDOWS-1256.txt"
    ## [6] "UDHR_Chinese_GB2312.txt"

    # Strip the extension
    filenames <- gsub(".txt$", "", filenames)
    parts <- strsplit(filenames, "_")
    fileencodings <- sapply(parts, "[", 3)

    head(fileencodings)
    ## [1] "UTF-16LE"     "UTF-8-BOM"    "ISO-8859-6"   "UTF-8"
    ## [5] "WINDOWS-1256" "GB2312"

    # Check whether certain file encodings are not supported
    notAvailableIndex <- which(!(fileencodings %in% iconvlist()))
    fileencodings[notAvailableIndex]
    ## [1] "UTF-8-BOM"

If we read the text files without specifying the encoding, we get erroneously formatted text. To avoid this, we determine the `encoding` using the character object `fileencoding` created above.

We can also add `docvars` based on the filenames.

    txts <- readtext(paste0(DATA_DIR, "/data_files_encodedtexts.zip"),
                     encoding = fileencodings,
                     docvarsfrom = "filenames",
                     docvarnames = c("document", "language", "input_encoding"))
    print(txts, n = 50)
    ## readtext object consisting of 36 documents and 3 docvars.
    ## # data.frame [36 × 5]
    ##    doc_id              text             document   language input_encoding
    ##    <chr>               <chr>            <chr>      <chr>    <chr>
    ##  1 IndianTreaty_Engli… "\"WHEREAS, t\"… IndianTre… English  UTF-16LE
    ##  2 IndianTreaty_Engli… "\"ARTICLE 1.\"… IndianTre… English  UTF-8-BOM
    ##  3 UDHR_Arabic_ISO-88… "\"الديباجة\nل\… UDHR       Arabic   ISO-8859-6
    ##  4 UDHR_Arabic_UTF-8.… "\"الديباجة\nل\… UDHR       Arabic   UTF-8
    ##  5 UDHR_Arabic_WINDOW… "\"الديباجة\nل\… UDHR       Arabic   WINDOWS-1256
    ##  6 UDHR_Chinese_GB231… "\"世界人权宣言\n联合国\… UDHR       Chinese  GB2312
    ##  7 UDHR_Chinese_GBK.t… "\"世界人权宣言\n联合国\… UDHR       Chinese  GBK
    ##  8 UDHR_Chinese_UTF-8… "\"世界人权宣言\n联合国\… UDHR       Chinese  UTF-8
    ##  9 UDHR_English_UTF-1… "\"Universal \"… UDHR       English  UTF-16BE
    ## 10 UDHR_English_UTF-1… "\"Universal \"… UDHR       English  UTF-16LE
    ## 11 UDHR_English_UTF-8… "\"Universal \"… UDHR       English  UTF-8
    ## 12 UDHR_English_WINDO… "\"Universal \"… UDHR       English  WINDOWS-1252
    ## 13 UDHR_French_ISO-88… "\"Déclaratio\"… UDHR       French   ISO-8859-1
    ## 14 UDHR_French_UTF-8.… "\"Déclaratio\"… UDHR       French   UTF-8
    ## 15 UDHR_French_WINDOW… "\"Déclaratio\"… UDHR       French   WINDOWS-1252
    ## 16 UDHR_German_ISO-88… "\"Die Allgem\"… UDHR       German   ISO-8859-1
    ## 17 UDHR_German_UTF-8.… "\"Die Allgem\"… UDHR       German   UTF-8
    ## 18 UDHR_German_WINDOW… "\"Die Allgem\"… UDHR       German   WINDOWS-1252
    ## 19 UDHR_Greek_CP1253.… "\"ΟΙΚΟΥΜΕΝΙΚ\"… UDHR       Greek    CP1253
    ## 20 UDHR_Greek_ISO-885… "\"ΟΙΚΟΥΜΕΝΙΚ\"… UDHR       Greek    ISO-8859-7
    ## 21 UDHR_Greek_UTF-8.t… "\"ΟΙΚΟΥΜΕΝΙΚ\"… UDHR       Greek    UTF-8
    ## 22 UDHR_Hindi_UTF-8.t… "\"मानव अधिका\"… UDHR       Hindi    UTF-8
    ## 23 UDHR_Icelandic_ISO… "\"Mannréttin\"… UDHR       Iceland… ISO-8859-1
    ## 24 UDHR_Icelandic_UTF… "\"Mannréttin\"… UDHR       Iceland… UTF-8
    ## 25 UDHR_Icelandic_WIN… "\"Mannréttin\"… UDHR       Iceland… WINDOWS-1252
    ## 26 UDHR_Japanese_CP93… "\"『世界人権宣言』\n \… UDHR       Japanese CP932
    ## 27 UDHR_Japanese_ISO-… "\"『世界人権宣言』\n \… UDHR       Japanese ISO-2022-JP
    ## 28 UDHR_Japanese_UTF-… "\"『世界人権宣言』\n \… UDHR       Japanese UTF-8
    ## 29 UDHR_Japanese_WIND… "\"『世界人権宣言』\n \… UDHR       Japanese WINDOWS-936
    ## 30 UDHR_Korean_ISO-20… "\"세 계 인 권 선 \"… UDHR       Korean   ISO-2022-KR
    ## 31 UDHR_Korean_UTF-8.… "\"세 계 인 권 선 \"… UDHR       Korean   UTF-8
    ## 32 UDHR_Russian_ISO-8… "\"Всеобщая д\"… UDHR       Russian  ISO-8859-5
    ## 33 UDHR_Russian_KOI8-… "\"Всеобщая д\"… UDHR       Russian  KOI8-R
    ## 34 UDHR_Russian_UTF-8… "\"Всеобщая д\"… UDHR       Russian  UTF-8
    ## 35 UDHR_Russian_WINDO… "\"Всеобщая д\"… UDHR       Russian  WINDOWS-1251
    ## 36 UDHR_Thai_UTF-8.txt "\"ปฏิญญาสากล\"…  UDHR       Thai     UTF-8

From this file we can easily create a quanteda `corpus` object.

    corpus_txts <- corpus(txts)
    summary(corpus_txts, 5)
    ## Corpus consisting of 36 documents, showing 5 documents:
    ##
    ##                                Text Types Tokens Sentences     document
    ##   IndianTreaty_English_UTF-16LE.txt   618   2578       155 IndianTreaty
    ##  IndianTreaty_English_UTF-8-BOM.txt   646   3090       154 IndianTreaty
    ##          UDHR_Arabic_ISO-8859-6.txt   753   1555        86         UDHR
    ##               UDHR_Arabic_UTF-8.txt   753   1555        86         UDHR
    ##        UDHR_Arabic_WINDOWS-1256.txt   753   1555        86         UDHR
    ##  language input_encoding
    ##   English       UTF-16LE
    ##   English      UTF-8-BOM
    ##    Arabic     ISO-8859-6
    ##    Arabic          UTF-8
    ##    Arabic   WINDOWS-1256
    ##
    ## Source: /private/var/folders/46/zfn6gwj15d3_n6dhyy1cvwc00000gp/T/RtmpbFlOkN/Rbuild20971d2a3cb8/readtext/vignettes/* on x86_64 by kbenoit
    ## Created: Thu May 10 22:43:14 2018
    ## Notes: