---
title: '[en] Introduction to rainette'
author: "Julien Barnier"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{[en] Introduction to rainette}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```


# Reinert textual clustering method

`rainette` is an R implementation of the Reinert textual clustering algorithm.
This algorithm is already available in softwares like
[Iramuteq](http://www.iramuteq.org/) or Alceste, the goal of `rainette` is
to provide it as an R package.

This is not a new algorithm, as the first articles describing it date back
to 1983. Here are some features of the method :

- each text is only assigned to one cluster, unlike methods like LDA.
- it is better suited to cluster small "homogeneous" documents : in fact the
  good practice is first to split texts into segments before applying the
  method.
- as the algorithm uses a singular value decomposition of a document-term
  matrix, the algorithm may not work on very large corpus.


## Description of the algorithm

The Reinert method is a divisive hierarchical clustering algorithm whose aim
is to maximise the inter-cluster Chi-squared distance.

The algorithm is applied to the document-term matrix computed on the corpus.
Documents are called `uce` (elementary context units). If an `uce` doesn't
include enough terms, it is merged with the following one into an `uc`
(context unit). The resulting matrix is then weighted as binary, so only the
presence/absence of terms are taken into account, not their frequencies.

The aim is to split this matrix into two clusters by maximizing the
Chi-squared distance between those. As an exhaustive search would be too
compute-intensive, the following method is used to get a good approximation :

- first, `uc` are ordered according to their coordinates on the first axis of
  the correspondance analysis of the binary matrix.
- `uc` are grouped in two clusters based on this order, and the grouping with
  the maximum inter-cluster Chi-squared distance is kept.
- based on this grouping, each `uc` is in turn assigned to the other cluster.
  If this new assignment gives a higher inter-cluster Chi-squared value, it is
  kept. The operation is repeated until no new assignment gives a higher
  Chi-squared.
- finally, on the resulting clusters binary matrices, features are selected
  based on their frequency and on a contingency coefficient minimum value.
- the biggest of the two resulting clusters is then split with the same algorithm.


## Double clustering

The Reinert method suggests to do a double clustering to get more robust
clusters. This method is also implemented in `rainette`.

The principle is to run two simple clusterings by varying the minimum `uc`
size. For example, the first one will be run with a minimum size of 10 terms,
and the second one with a minimum size of 15.

The two sets of clusters are then "crossed" : every pair of clusters of each
clustering are crossed together, even if they are not on the same hierarchical
level. We then compute the number of `uc` present in both clusters, and a
Chi-squared value of association between them.

Only a subset of all these "crossed-clusters" are kept : those with different
elements, with a minimum number of `uc` or with a minimum association value.
Then, for a given number of clusters `k`, the algorithm looks for the optimal
partition of crossed-clusters, *ie* it keeps the set of crossed-clusters with no
common elements, and with either the higher total number of elements, or the
higher sum of Chi-squared association coefficients.

Then, this optimal partition is used either as the final clustering (with
potentially quite a high proportion of `NA`), or as a starting point for a
k-nearest-neighbour clustering for non-assigned documents.


## References (in french)

- Reinert M, Une méthode de classification descendante hiérarchique :
  application à l'analyse lexicale par contexte, Cahiers de l'analyse des
  données, Volume 8, Numéro 2, 1983.
  <http://www.numdam.org/item/?id=CAD_1983__8_2_187_0>
- Reinert M., Alceste une méthodologie d'analyse des données textuelles et une
  application: Aurelia De Gerard De Nerval, Bulletin de Méthodologie
  Sociologique, Volume 26, Numéro 1, 1990.
  <https://doi.org/10.1177/075910639002600103>


# `rainette` usage

## Corpus preparation

### `split_segments`

As explained before, as it doesn't take into account terms frequencies and
assign each document to only one cluster, the Reinert method must be applied
to short and "homogeneous" documents. It could be ok if your work on short
comments on a website, but on longer documents the corpus must first be split
into small textual segments.

The `split_segments` function does just that, and can be applied on a `tm` or
`quanteda` corpus. 

On this article we will apply it to the sample `data_corpus_inaugural`
`quanteda` corpus :

```{r warnings = FALSE, message  = FALSE}
library(quanteda)
library(rainette)
corpus <- split_segments(data_corpus_inaugural)
```

`split_segments` will split the original texts into smaller chunks, attempting to
respect sentences and punctuation when possible. The function takes three
arguments :

- `segment_size` : the preferred segment size, in words
- `segment_size_window` : the "window" into which looking for the best segment
  split, in words. If `NULL`, it is set to 0.4*`segment_size`.
- `force_single_core` : by default, multithreading is used when the corpus is
  big. Set this to `TRUE` to disable multithreading completely.

The result of the function is always a `quanteda` corpus, which keeps the
original corpus metadata :

```{r}
corpus
```

```{r}
head(docvars(corpus))
```

### dfm computation

Next step is to compute the document-feature matrix. As our `corpus` object is
a `quanteda` corpus, we can use its `dfm` function :

```{r}
dtm <- dfm(corpus, remove = stopwords("en"), tolower = TRUE, remove_punct = TRUE)
```

We only keep the terms that appear at least in 10 segments by using `dfm_trim`
:

```{r}
dtm <- dfm_trim(dtm, min_termfreq = 10)
```

## Simple clustering

We are now ready to compute a simple Reinert clustering by using the
`rainette` function. Its main arguments are :

- `k` : the number of clusters to compute.
- `min_uc_size` : the minimum number of terms in each context unit at startup
  (default : 10). If a `dfm` document contains less than this number of terms,
  it will be merged with the following one.
- `min_split_members` : if a cluster is smaller than this value, it won't be
  split afterwards (default : 5).
  
Here we will compute 5 clusters with a `minimum_uc_size` of 15 :

```{r message = FALSE}
res <- rainette(dtm, k = 5, min_uc_size = 15)
```

To help exploring the clustering results, `rainette` offers a small *shiny
gadget* which can be launched with `rainette_explor` :

```{r eval = FALSE}
rainette_explor(res, dtm)
```

The interactive interface should look something like this :

![](rainette_explor_en.png)

You can change the number of clusters, the keyness statistic used, etc., and
see the result in real time. The most characteristics terms are displayed with
a blue bar or a red one for those with a negative keyness.

It's also possible to display terms keyness as word clouds :

![](rainette_explor_en_cloud.png)

At any time, you can click on the *Get R code* button to get the R code to
reproduce the current plot and to compute cluster membership.

You can also directly use `cutree_rainette` to get each document cluster at
level `k` :

```{r}
groups <- cutree_rainette(res, k = 3)
```
This vector can be used, for example, as a new corpus metadata variable :

```{r}
corpus$group <- groups
head(docvars(corpus))
```

Finally, you can also directly compute each cluster keyness statistics with the
`rainette_stats` function :

```{r}
rainette_stats(corpus$group, dtm, n_terms = 5)
```

## Double clustering

`rainette` also offers a "double clustering" algorithm, as described above : two
simple clusterings are computed with varying `min_uc_size`, and then combined
to get a better partition and more robust clusters.

This can be done with the `rainette2` function

`rainette2` can be applied to two already computed simple clusterings. Here,
we compute them with `min_uc_size` at 10 and 15 :

```{r message=FALSE}
res1 <- rainette(dtm, k = 5, min_uc_size = 10, min_split_members = 10)
res2 <- rainette(dtm, k = 5, min_uc_size = 15, min_split_members = 10)
```

We then use `rainette2` to combine them. The main function arguments are
`max_k`, the maximum number of clusters, and `min_members`, the minimum
cluster size :

```{r message=FALSE}
res <- rainette2(res1, res2, max_k = 5, min_members = 10)
```

Another way is to call `rainette2` directly on our `dtm` matrix by giving it
two `uc_size1` and `uc_size2` arguments :

```{r eval=FALSE}
res <- rainette2(dtm, uc_size1 = 10, uc_size2 = 15, max_k = 5, min_members = 10)
```

The resulting object is a *tibble* with, for each level *k*, the optimal
partitions and their characteristics. Another interactive interface is
available to explore the results. It is launched with `rainette2_explor` :

```{r eval=FALSE}
rainette2_explor(res, dtm)
```

![](rainette2_explor_en.png)

The interface is very similar to the previous one, except there is no
dendrogram anymore, but a single barplot with each cluster size instead. Be
careful of the number of `NA` (not assigned documents), as it can be quite
high. You can still use the *Get R code* button to get the current plot code,
and use `cutree_rainette` to get clusters membership.

`rainette_stats` can also be used to get keyness statistics for each cluster :

```{r}
groups <- cutree(res, 3)
rainette_stats(groups, dtm, n_terms = 5)
```

Finally, if some points are not assigned to any cluster, you can use
`rainette2_complete_groups` to assign them to the nearest one by using a
*k-nearest-neighbors* algorithm :

```{r}
groups_completed <- rainette2_complete_groups(dtm, groups)
```
