This vignette shows how to use the massiveGST package.

Installation

Get the stable release from CRAN

install.packages("massiveGST")

On going bug correction and future improvements are in github.

if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
BiocManager::install("stefanoMP/massiveGST")

Setup

suppressPackageStartupMessages(library(massiveGST, quietly = TRUE))

getting gene-profile from a file

In any cases, the names of the list have to match the gene names in any of their coding (gene-symbol, entrez, ensembl, etc.). To explain some details of the functions, we assume that positive values of the gene-profile are up-regulated in the treatment group, while the negative ones are up-regulated in the control group. In other words, the positive genes are associated with the treatment group, while the others are associated with the control samples.

Here, we consider the working gene-profile from Frattini et al, 2018, stored as external data in the package.

fname <- system.file("extdata", package="massiveGST")
fname <- file.path(fname, "pre_ranked_list.txt")
geneProfile <- get_geneProfile(fname)
class(geneProfile)
## [1] "numeric"
head(geneProfile)
##    WASF3     VAV3     PFKM     LGR6  SLC6A11 KIAA1147 
## 4.819656 4.332110 4.244893 4.182461 4.088991 4.045542
tail(geneProfile)
##      SLC47A2        STON1 DKFZp434K191      PLEKHO1        PDE6B       IFNGR2 
##    -2.728522    -2.748892    -2.881312    -2.942627    -3.141299    -3.158739

getting gene-sets from msigdbr

msigdbr package is an R distribution of the most recent release of the MSigDB collection of gene-sets. The function get_geneSets_from_msigdbr is a wrapper allowing the creation of a data structure for following gene-set enrichment analysis. Function parameters are those of the msigdbr function, except for ‘what’, allowing to choose the coding of the gene names.

This example extracts gene-sets where the gene-names are the gene symbols. Other possibilities have been specified in the ‘man’ fo the functions, and essentially they are the names of the columns of the data frame retrieved by the msigdbr function.

geneSets <- get_geneSets_from_msigdbr(category = "H", what = "gene_symbol")
## msigdbr: R package version 7.4.1
class(geneSets)
## [1] "list"
head(names(geneSets))
## [1] "HALLMARK_ADIPOGENESIS"        "HALLMARK_ALLOGRAFT_REJECTION"
## [3] "HALLMARK_ANDROGEN_RESPONSE"   "HALLMARK_ANGIOGENESIS"       
## [5] "HALLMARK_APICAL_JUNCTION"     "HALLMARK_APICAL_SURFACE"

Optionally, the gene-sets collection can be stored in gmt formatted file

fname <- file.path(tempdir(), "hallmarks.gmt")
write_geneSets_to_gmt(geneSets, fileName = fname)

and retrieved with the code

fname <- file.path(tempdir(), "hallmarks.gmt")
tmp <- get_geneSets_from_local_files(fname)

class(geneSets)
## [1] "list"
head(names(geneSets))
## [1] "HALLMARK_ADIPOGENESIS"        "HALLMARK_ALLOGRAFT_REJECTION"
## [3] "HALLMARK_ANDROGEN_RESPONSE"   "HALLMARK_ANGIOGENESIS"       
## [5] "HALLMARK_APICAL_JUNCTION"     "HALLMARK_APICAL_SURFACE"

Running the analysis with alternative = “two.sided”

system.time({ans <- massiveGST(geneProfile, geneSets, alternative = "two.sided")})
##    user  system elapsed 
##   0.041   0.000   0.041
class(ans)
## [1] "mGST"       "data.frame"
ans[1:6,]
##                                    size actualSize       NES      odd logit2NES
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5866509 1.419262 0.5051413
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907006 2.233114 1.1590566
## HALLMARK_ADIPOGENESIS               200        176 0.5946204 1.466824 0.5526955
## HALLMARK_MITOTIC_SPINDLE            199        187 0.5508155 1.226257 0.2942609
## HALLMARK_BILE_ACID_METABOLISM       112        112 0.5856262 1.413280 0.4990471
## HALLMARK_FATTY_ACID_METABOLISM      158        149 0.5658001 1.303087 0.3819331
##                                    abs_logit2NES      p.value     BH.value
## HALLMARK_ESTROGEN_RESPONSE_EARLY       0.5051413 5.111425e-05 2.555713e-04
## HALLMARK_OXIDATIVE_PHOSPHORYLATION     1.1590566 2.234530e-18 2.793163e-17
## HALLMARK_ADIPOGENESIS                  0.5526955 1.513137e-05 8.406317e-05
## HALLMARK_MITOTIC_SPINDLE               0.2942609 1.664580e-02 4.380475e-02
## HALLMARK_BILE_ACID_METABOLISM          0.4990471 1.753213e-03 6.224599e-03
## HALLMARK_FATTY_ACID_METABOLISM         0.3819331 5.595257e-03 1.748518e-02
##                                         B.value relevance
## HALLMARK_ESTROGEN_RESPONSE_EARLY   2.555713e-03        59
## HALLMARK_OXIDATIVE_PHOSPHORYLATION 1.117265e-16        59
## HALLMARK_ADIPOGENESIS              7.565685e-04        56
## HALLMARK_MITOTIC_SPINDLE           8.322902e-01        52
## HALLMARK_BILE_ACID_METABOLISM      8.766064e-02        48
## HALLMARK_FATTY_ACID_METABOLISM     2.797628e-01        48

The analysis result is essentially a data frame; the first class ‘mGST’ allows hooking other methods to handle the output.

The results can be saved in tab-separated value (tsv) format or an XLS format file.

fname <- file.path(tempdir(), "massiveGST_results.tsv")
save_as_tsv(ans, file_name = fname)

fname <- file.path(tempdir(), "massiveGST_results.xls")
save_as_xls(ans, file_name = fname)

The summary method.

The summary function allows customizing the result of the analysis.

summary(ans)[1:10,]
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_MITOTIC_SPINDLE            199        187 0.5508    0.2943 1.7e-02
## HALLMARK_BILE_ACID_METABOLISM       112        112 0.5856    0.4990 1.8e-03
## HALLMARK_FATTY_ACID_METABOLISM      158        149 0.5658    0.3819 5.6e-03
## HALLMARK_MTORC1_SIGNALING           200        188 0.5372    0.2152 7.9e-02
## HALLMARK_G2M_CHECKPOINT             200        178 0.5475    0.2748 2.9e-02
## HALLMARK_UV_RESPONSE_DN             144        132 0.5528    0.3059 3.6e-02
## HALLMARK_HEME_METABOLISM            200        180 0.5325    0.1876 1.3e-01
##                                    BH.value B.value relevance
## HALLMARK_ESTROGEN_RESPONSE_EARLY    2.6e-04 2.6e-03      59.0
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  2.8e-17 1.1e-16      59.0
## HALLMARK_ADIPOGENESIS               8.4e-05 7.6e-04      56.0
## HALLMARK_MITOTIC_SPINDLE            4.4e-02 8.3e-01      52.0
## HALLMARK_BILE_ACID_METABOLISM       6.2e-03 8.8e-02      48.0
## HALLMARK_FATTY_ACID_METABOLISM      1.7e-02 2.8e-01      48.0
## HALLMARK_MTORC1_SIGNALING           1.6e-01 1.0e+00      47.0
## HALLMARK_G2M_CHECKPOINT             6.9e-02 1.0e+00      46.5
## HALLMARK_UV_RESPONSE_DN             8.2e-02 1.0e+00      42.0
## HALLMARK_HEME_METABOLISM            2.5e-01 1.0e+00      41.0

With it:

  • The ordering of the table can be chosen.
summary(ans, order_by = "NES")[1:10,]
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05
## HALLMARK_BILE_ACID_METABOLISM       112        112 0.5856    0.4990 1.8e-03
## HALLMARK_FATTY_ACID_METABOLISM      158        149 0.5658    0.3819 5.6e-03
## HALLMARK_NOTCH_SIGNALING             32         30 0.5572    0.3316 2.8e-01
## HALLMARK_UV_RESPONSE_DN             144        132 0.5528    0.3059 3.6e-02
## HALLMARK_MITOTIC_SPINDLE            199        187 0.5508    0.2943 1.7e-02
## HALLMARK_G2M_CHECKPOINT             200        178 0.5475    0.2748 2.9e-02
## HALLMARK_PROTEIN_SECRETION           96         93 0.5459    0.2656 1.3e-01
##                                    BH.value B.value
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  2.8e-17 1.1e-16
## HALLMARK_ADIPOGENESIS               8.4e-05 7.6e-04
## HALLMARK_ESTROGEN_RESPONSE_EARLY    2.6e-04 2.6e-03
## HALLMARK_BILE_ACID_METABOLISM       6.2e-03 8.8e-02
## HALLMARK_FATTY_ACID_METABOLISM      1.7e-02 2.8e-01
## HALLMARK_NOTCH_SIGNALING            4.2e-01 1.0e+00
## HALLMARK_UV_RESPONSE_DN             8.2e-02 1.0e+00
## HALLMARK_MITOTIC_SPINDLE            4.4e-02 8.3e-01
## HALLMARK_G2M_CHECKPOINT             6.9e-02 1.0e+00
## HALLMARK_PROTEIN_SECRETION          2.4e-01 1.0e+00
summary(ans, order_by = "p.value")[1:10,]
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  200        188 0.2472   -1.6070 6.8e-33
## HALLMARK_INTERFERON_ALPHA_RESPONSE   97         87 0.1688   -2.3000 1.3e-26
## HALLMARK_ALLOGRAFT_REJECTION        200        189 0.2816   -1.3512 4.4e-25
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_INFLAMMATORY_RESPONSE      200        192 0.3517   -0.8825 1.4e-12
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    200        186 0.3554   -0.8591 1.1e-11
## HALLMARK_IL6_JAK_STAT3_SIGNALING     87         83 0.3064   -1.1788 1.1e-09
## HALLMARK_COMPLEMENT                 200        190 0.3869   -0.6643 7.8e-08
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05
##                                    BH.value B.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  3.4e-31 3.4e-31
## HALLMARK_INTERFERON_ALPHA_RESPONSE  3.3e-25 6.6e-25
## HALLMARK_ALLOGRAFT_REJECTION        7.3e-24 2.2e-23
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  2.8e-17 1.1e-16
## HALLMARK_INFLAMMATORY_RESPONSE      1.4e-11 7.2e-11
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    8.9e-11 5.3e-10
## HALLMARK_IL6_JAK_STAT3_SIGNALING    7.8e-09 5.4e-08
## HALLMARK_COMPLEMENT                 4.8e-07 3.9e-06
## HALLMARK_ADIPOGENESIS               8.4e-05 7.6e-04
## HALLMARK_ESTROGEN_RESPONSE_EARLY    2.6e-04 2.6e-03
summary(ans, order_by = "bonferroni")[1:10,]
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  200        188 0.2472   -1.6070 6.8e-33
## HALLMARK_INTERFERON_ALPHA_RESPONSE   97         87 0.1688   -2.3000 1.3e-26
## HALLMARK_ALLOGRAFT_REJECTION        200        189 0.2816   -1.3512 4.4e-25
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_INFLAMMATORY_RESPONSE      200        192 0.3517   -0.8825 1.4e-12
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    200        186 0.3554   -0.8591 1.1e-11
## HALLMARK_IL6_JAK_STAT3_SIGNALING     87         83 0.3064   -1.1788 1.1e-09
## HALLMARK_COMPLEMENT                 200        190 0.3869   -0.6643 7.8e-08
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05
##                                    BH.value B.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  3.4e-31 3.4e-31
## HALLMARK_INTERFERON_ALPHA_RESPONSE  3.3e-25 6.6e-25
## HALLMARK_ALLOGRAFT_REJECTION        7.3e-24 2.2e-23
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  2.8e-17 1.1e-16
## HALLMARK_INFLAMMATORY_RESPONSE      1.4e-11 7.2e-11
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    8.9e-11 5.3e-10
## HALLMARK_IL6_JAK_STAT3_SIGNALING    7.8e-09 5.4e-08
## HALLMARK_COMPLEMENT                 4.8e-07 3.9e-06
## HALLMARK_ADIPOGENESIS               8.4e-05 7.6e-04
## HALLMARK_ESTROGEN_RESPONSE_EARLY    2.6e-04 2.6e-03
  • Some columns can be removed.
(tmp <- summary(ans, order_by = "p.value", cols_to_remove = c("BH.value", "B.value"))[1:10,])
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  200        188 0.2472   -1.6070 6.8e-33
## HALLMARK_INTERFERON_ALPHA_RESPONSE   97         87 0.1688   -2.3000 1.3e-26
## HALLMARK_ALLOGRAFT_REJECTION        200        189 0.2816   -1.3512 4.4e-25
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_INFLAMMATORY_RESPONSE      200        192 0.3517   -0.8825 1.4e-12
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    200        186 0.3554   -0.8591 1.1e-11
## HALLMARK_IL6_JAK_STAT3_SIGNALING     87         83 0.3064   -1.1788 1.1e-09
## HALLMARK_COMPLEMENT                 200        190 0.3869   -0.6643 7.8e-08
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05
summary(ans, as.formattable = TRUE)
size actualSize NES logit2NES p.value BH.value B.value relevance
HALLMARK_ESTROGEN_RESPONSE_EARLY 200 184 0.5867 0.5051 5.1e-05 2.6e-04 2.6e-03 59.0
HALLMARK_OXIDATIVE_PHOSPHORYLATION 200 177 0.6907 1.1591 2.2e-18 2.8e-17 1.1e-16 59.0
HALLMARK_ADIPOGENESIS 200 176 0.5946 0.5527 1.5e-05 8.4e-05 7.6e-04 56.0
HALLMARK_MITOTIC_SPINDLE 199 187 0.5508 0.2943 1.7e-02 4.4e-02 8.3e-01 52.0
HALLMARK_BILE_ACID_METABOLISM 112 112 0.5856 0.4990 1.8e-03 6.2e-03 8.8e-02 48.0
HALLMARK_FATTY_ACID_METABOLISM 158 149 0.5658 0.3819 5.6e-03 1.7e-02 2.8e-01 48.0
HALLMARK_MTORC1_SIGNALING 200 188 0.5372 0.2152 7.9e-02 1.6e-01 1.0e+00 47.0
HALLMARK_G2M_CHECKPOINT 200 178 0.5475 0.2748 2.9e-02 6.9e-02 1.0e+00 46.5
HALLMARK_UV_RESPONSE_DN 144 132 0.5528 0.3059 3.6e-02 8.2e-02 1.0e+00 42.0
HALLMARK_HEME_METABOLISM 200 180 0.5325 0.1876 1.3e-01 2.5e-01 1.0e+00 41.0
HALLMARK_MYC_TARGETS_V1 200 178 0.5228 0.1318 2.9e-01 4.3e-01 1.0e+00 33.5
HALLMARK_PROTEIN_SECRETION 96 93 0.5459 0.2656 1.3e-01 2.4e-01 1.0e+00 32.0
HALLMARK_NOTCH_SIGNALING 32 30 0.5572 0.3316 2.8e-01 4.2e-01 1.0e+00 29.0
HALLMARK_ESTROGEN_RESPONSE_LATE 200 193 0.5037 0.0216 8.6e-01 8.9e-01 1.0e+00 28.0
HALLMARK_UNFOLDED_PROTEIN_RESPONSE 113 101 0.5285 0.1646 3.2e-01 4.6e-01 1.0e+00 26.0
HALLMARK_E2F_TARGETS 200 171 0.5113 0.0651 6.1e-01 6.9e-01 1.0e+00 22.0
HALLMARK_PEROXISOME 104 100 0.5181 0.1047 5.3e-01 6.2e-01 1.0e+00 20.0
HALLMARK_WNT_BETA_CATENIN_SIGNALING 42 41 0.5309 0.1783 4.9e-01 6.0e-01 1.0e+00 20.0
HALLMARK_ANGIOGENESIS 36 36 0.5321 0.1856 5.0e-01 6.0e-01 1.0e+00 19.0
HALLMARK_ANDROGEN_RESPONSE 100 91 0.5133 0.0766 6.6e-01 7.2e-01 1.0e+00 14.0
HALLMARK_PI3K_AKT_MTOR_SIGNALING 105 99 0.5016 0.0090 9.6e-01 9.8e-01 1.0e+00 11.0
HALLMARK_CHOLESTEROL_HOMEOSTASIS 74 70 0.5008 0.0049 9.8e-01 9.8e-01 1.0e+00 6.0
HALLMARK_MYC_TARGETS_V2 58 50 0.4819 -0.1042 6.6e-01 7.2e-01 1.0e+00 -9.0
HALLMARK_HEDGEHOG_SIGNALING 36 35 0.4661 -0.1959 4.9e-01 6.0e-01 1.0e+00 -11.0
HALLMARK_APICAL_SURFACE 44 41 0.4641 -0.2073 4.3e-01 5.5e-01 1.0e+00 -16.0
HALLMARK_PANCREAS_BETA_CELLS 40 40 0.4560 -0.2544 3.4e-01 4.6e-01 1.0e+00 -21.0
HALLMARK_KRAS_SIGNALING_DN 200 181 0.4795 -0.1181 3.4e-01 4.6e-01 1.0e+00 -25.5
HALLMARK_XENOBIOTIC_METABOLISM 200 190 0.4947 -0.0303 8.0e-01 8.5e-01 1.0e+00 -26.5
HALLMARK_REACTIVE_OXYGEN_SPECIES_PATHWAY 49 44 0.4502 -0.2886 2.5e-01 4.1e-01 1.0e+00 -28.0
HALLMARK_TGF_BETA_SIGNALING 54 52 0.4511 -0.2829 2.2e-01 3.7e-01 1.0e+00 -30.0
HALLMARK_DNA_REPAIR 150 134 0.4646 -0.2048 1.6e-01 2.8e-01 1.0e+00 -31.0
HALLMARK_SPERMATOGENESIS 135 123 0.4582 -0.2419 1.1e-01 2.2e-01 1.0e+00 -32.0
HALLMARK_P53_PATHWAY 200 181 0.4716 -0.1640 1.9e-01 3.2e-01 1.0e+00 -32.5
HALLMARK_APICAL_JUNCTION 200 187 0.4765 -0.1357 2.7e-01 4.2e-01 1.0e+00 -33.5
HALLMARK_MYOGENESIS 200 195 0.4813 -0.1081 3.7e-01 4.8e-01 1.0e+00 -36.0
HALLMARK_UV_RESPONSE_UP 158 148 0.4567 -0.2504 6.9e-02 1.5e-01 1.0e+00 -37.0
HALLMARK_KRAS_SIGNALING_UP 200 185 0.4523 -0.2763 2.5e-02 6.3e-02 1.0e+00 -45.0
HALLMARK_COAGULATION 138 132 0.4100 -0.5250 3.6e-04 1.4e-03 1.8e-02 -50.0
HALLMARK_APOPTOSIS 161 157 0.4170 -0.4835 3.3e-04 1.4e-03 1.7e-02 -52.0
HALLMARK_GLYCOLYSIS 200 187 0.4489 -0.2961 1.6e-02 4.4e-02 8.0e-01 -52.5
HALLMARK_HYPOXIA 200 186 0.4338 -0.3842 1.9e-03 6.2e-03 9.3e-02 -54.5
HALLMARK_IL2_STAT5_SIGNALING 199 180 0.4164 -0.4872 1.1e-04 5.0e-04 5.5e-03 -55.0
HALLMARK_IL6_JAK_STAT3_SIGNALING 87 83 0.3064 -1.1788 1.1e-09 7.8e-09 5.4e-08 -55.0
HALLMARK_EPITHELIAL_MESENCHYMAL_TRANSITION 200 191 0.4449 -0.3191 8.7e-03 2.6e-02 4.4e-01 -60.0
HALLMARK_INTERFERON_ALPHA_RESPONSE 97 87 0.1688 -2.3000 1.3e-26 3.3e-25 6.6e-25 -63.0
HALLMARK_TNFA_SIGNALING_VIA_NFKB 200 186 0.3554 -0.8591 1.1e-11 8.9e-11 5.3e-10 -65.5
HALLMARK_COMPLEMENT 200 190 0.3869 -0.6643 7.8e-08 4.8e-07 3.9e-06 -68.5
HALLMARK_ALLOGRAFT_REJECTION 200 189 0.2816 -1.3512 4.4e-25 7.3e-24 2.2e-23 -76.0
HALLMARK_INFLAMMATORY_RESPONSE 200 192 0.3517 -0.8825 1.4e-12 1.4e-11 7.2e-11 -76.0
HALLMARK_INTERFERON_GAMMA_RESPONSE 200 188 0.2472 -1.6070 6.8e-33 3.4e-31 3.4e-31 -76.0 
summary(ans, order_by = "p.value", cols_to_remove = c("BH.value", "B.value"), as.formattable = TRUE)
size actualSize NES logit2NES p.value
HALLMARK_INTERFERON_GAMMA_RESPONSE 200 188 0.2472 -1.6070 6.8e-33
HALLMARK_INTERFERON_ALPHA_RESPONSE 97 87 0.1688 -2.3000 1.3e-26
HALLMARK_ALLOGRAFT_REJECTION 200 189 0.2816 -1.3512 4.4e-25
HALLMARK_OXIDATIVE_PHOSPHORYLATION 200 177 0.6907 1.1591 2.2e-18
HALLMARK_INFLAMMATORY_RESPONSE 200 192 0.3517 -0.8825 1.4e-12
HALLMARK_TNFA_SIGNALING_VIA_NFKB 200 186 0.3554 -0.8591 1.1e-11
HALLMARK_IL6_JAK_STAT3_SIGNALING 87 83 0.3064 -1.1788 1.1e-09
HALLMARK_COMPLEMENT 200 190 0.3869 -0.6643 7.8e-08
HALLMARK_ADIPOGENESIS 200 176 0.5946 0.5527 1.5e-05
HALLMARK_ESTROGEN_RESPONSE_EARLY 200 184 0.5867 0.5051 5.1e-05
HALLMARK_IL2_STAT5_SIGNALING 199 180 0.4164 -0.4872 1.1e-04
HALLMARK_APOPTOSIS 161 157 0.4170 -0.4835 3.3e-04
HALLMARK_COAGULATION 138 132 0.4100 -0.5250 3.6e-04
HALLMARK_BILE_ACID_METABOLISM 112 112 0.5856 0.4990 1.8e-03
HALLMARK_HYPOXIA 200 186 0.4338 -0.3842 1.9e-03
HALLMARK_FATTY_ACID_METABOLISM 158 149 0.5658 0.3819 5.6e-03
HALLMARK_EPITHELIAL_MESENCHYMAL_TRANSITION 200 191 0.4449 -0.3191 8.7e-03
HALLMARK_GLYCOLYSIS 200 187 0.4489 -0.2961 1.6e-02
HALLMARK_MITOTIC_SPINDLE 199 187 0.5508 0.2943 1.7e-02
HALLMARK_KRAS_SIGNALING_UP 200 185 0.4523 -0.2763 2.5e-02
HALLMARK_G2M_CHECKPOINT 200 178 0.5475 0.2748 2.9e-02
HALLMARK_UV_RESPONSE_DN 144 132 0.5528 0.3059 3.6e-02
HALLMARK_UV_RESPONSE_UP 158 148 0.4567 -0.2504 6.9e-02
HALLMARK_MTORC1_SIGNALING 200 188 0.5372 0.2152 7.9e-02
HALLMARK_SPERMATOGENESIS 135 123 0.4582 -0.2419 1.1e-01
HALLMARK_PROTEIN_SECRETION 96 93 0.5459 0.2656 1.3e-01
HALLMARK_HEME_METABOLISM 200 180 0.5325 0.1876 1.3e-01
HALLMARK_DNA_REPAIR 150 134 0.4646 -0.2048 1.6e-01
HALLMARK_P53_PATHWAY 200 181 0.4716 -0.1640 1.9e-01
HALLMARK_TGF_BETA_SIGNALING 54 52 0.4511 -0.2829 2.2e-01
HALLMARK_REACTIVE_OXYGEN_SPECIES_PATHWAY 49 44 0.4502 -0.2886 2.5e-01
HALLMARK_APICAL_JUNCTION 200 187 0.4765 -0.1357 2.7e-01
HALLMARK_NOTCH_SIGNALING 32 30 0.5572 0.3316 2.8e-01
HALLMARK_MYC_TARGETS_V1 200 178 0.5228 0.1318 2.9e-01
HALLMARK_UNFOLDED_PROTEIN_RESPONSE 113 101 0.5285 0.1646 3.2e-01
HALLMARK_PANCREAS_BETA_CELLS 40 40 0.4560 -0.2544 3.4e-01
HALLMARK_KRAS_SIGNALING_DN 200 181 0.4795 -0.1181 3.4e-01
HALLMARK_MYOGENESIS 200 195 0.4813 -0.1081 3.7e-01
HALLMARK_APICAL_SURFACE 44 41 0.4641 -0.2073 4.3e-01
HALLMARK_HEDGEHOG_SIGNALING 36 35 0.4661 -0.1959 4.9e-01
HALLMARK_WNT_BETA_CATENIN_SIGNALING 42 41 0.5309 0.1783 4.9e-01
HALLMARK_ANGIOGENESIS 36 36 0.5321 0.1856 5.0e-01
HALLMARK_PEROXISOME 104 100 0.5181 0.1047 5.3e-01
HALLMARK_E2F_TARGETS 200 171 0.5113 0.0651 6.1e-01
HALLMARK_MYC_TARGETS_V2 58 50 0.4819 -0.1042 6.6e-01
HALLMARK_ANDROGEN_RESPONSE 100 91 0.5133 0.0766 6.6e-01
HALLMARK_XENOBIOTIC_METABOLISM 200 190 0.4947 -0.0303 8.0e-01
HALLMARK_ESTROGEN_RESPONSE_LATE 200 193 0.5037 0.0216 8.6e-01
HALLMARK_PI3K_AKT_MTOR_SIGNALING 105 99 0.5016 0.0090 9.6e-01
HALLMARK_CHOLESTEROL_HOMEOSTASIS 74 70 0.5008 0.0049 9.8e-01

The value of the summary method is invisible, but it is a data frame.

tmp
##                                    size actualSize    NES logit2NES p.value
## HALLMARK_INTERFERON_GAMMA_RESPONSE  200        188 0.2472   -1.6070 6.8e-33
## HALLMARK_INTERFERON_ALPHA_RESPONSE   97         87 0.1688   -2.3000 1.3e-26
## HALLMARK_ALLOGRAFT_REJECTION        200        189 0.2816   -1.3512 4.4e-25
## HALLMARK_OXIDATIVE_PHOSPHORYLATION  200        177 0.6907    1.1591 2.2e-18
## HALLMARK_INFLAMMATORY_RESPONSE      200        192 0.3517   -0.8825 1.4e-12
## HALLMARK_TNFA_SIGNALING_VIA_NFKB    200        186 0.3554   -0.8591 1.1e-11
## HALLMARK_IL6_JAK_STAT3_SIGNALING     87         83 0.3064   -1.1788 1.1e-09
## HALLMARK_COMPLEMENT                 200        190 0.3869   -0.6643 7.8e-08
## HALLMARK_ADIPOGENESIS               200        176 0.5946    0.5527 1.5e-05
## HALLMARK_ESTROGEN_RESPONSE_EARLY    200        184 0.5867    0.5051 5.1e-05

Trimming the table fo results

Three functions allow to remove rows from the results table as need. The output of these function can be formatted with the summary method.

The first function removes the non significant gene-sets. By default, a 5% level of significance is applied to BH.values (Benijamini and Hockberg adjustment of the p.values)

summary(cut_by_significance(ans), as.formattable = TRUE)
size actualSize NES logit2NES p.value BH.value B.value relevance
HALLMARK_ESTROGEN_RESPONSE_EARLY 200 184 0.5867 0.5051 5.1e-05 2.6e-04 2.6e-03 59.0
HALLMARK_OXIDATIVE_PHOSPHORYLATION 200 177 0.6907 1.1591 2.2e-18 2.8e-17 1.1e-16 59.0
HALLMARK_ADIPOGENESIS 200 176 0.5946 0.5527 1.5e-05 8.4e-05 7.6e-04 56.0
HALLMARK_MITOTIC_SPINDLE 199 187 0.5508 0.2943 1.7e-02 4.4e-02 8.3e-01 52.0
HALLMARK_BILE_ACID_METABOLISM 112 112 0.5856 0.4990 1.8e-03 6.2e-03 8.8e-02 48.0
HALLMARK_FATTY_ACID_METABOLISM 158 149 0.5658 0.3819 5.6e-03 1.7e-02 2.8e-01 48.0
HALLMARK_COAGULATION 138 132 0.4100 -0.5250 3.6e-04 1.4e-03 1.8e-02 -50.0
HALLMARK_APOPTOSIS 161 157 0.4170 -0.4835 3.3e-04 1.4e-03 1.7e-02 -52.0
HALLMARK_GLYCOLYSIS 200 187 0.4489 -0.2961 1.6e-02 4.4e-02 8.0e-01 -52.5
HALLMARK_HYPOXIA 200 186 0.4338 -0.3842 1.9e-03 6.2e-03 9.3e-02 -54.5
HALLMARK_IL2_STAT5_SIGNALING 199 180 0.4164 -0.4872 1.1e-04 5.0e-04 5.5e-03 -55.0
HALLMARK_IL6_JAK_STAT3_SIGNALING 87 83 0.3064 -1.1788 1.1e-09 7.8e-09 5.4e-08 -55.0
HALLMARK_EPITHELIAL_MESENCHYMAL_TRANSITION 200 191 0.4449 -0.3191 8.7e-03 2.6e-02 4.4e-01 -60.0
HALLMARK_INTERFERON_ALPHA_RESPONSE 97 87 0.1688 -2.3000 1.3e-26 3.3e-25 6.6e-25 -63.0
HALLMARK_TNFA_SIGNALING_VIA_NFKB 200 186 0.3554 -0.8591 1.1e-11 8.9e-11 5.3e-10 -65.5
HALLMARK_COMPLEMENT 200 190 0.3869 -0.6643 7.8e-08 4.8e-07 3.9e-06 -68.5
HALLMARK_ALLOGRAFT_REJECTION 200 189 0.2816 -1.3512 4.4e-25 7.3e-24 2.2e-23 -76.0
HALLMARK_INFLAMMATORY_RESPONSE 200 192 0.3517 -0.8825 1.4e-12 1.4e-11 7.2e-11 -76.0
HALLMARK_INTERFERON_GAMMA_RESPONSE 200 188 0.2472 -1.6070 6.8e-33 3.4e-31 3.4e-31 -76.0

As a toy example, …

summary(cut_by_significance(ans, level_of_significance = 0.01, where = "bonferroni"),
        cols_to_remove = c("BH.value", "NES", "size"), 
        order_by = "logit2NES",
        as.formattable = TRUE)
actualSize logit2NES p.value B.value
HALLMARK_OXIDATIVE_PHOSPHORYLATION 177 1.1591 2.2e-18 1.1e-16
HALLMARK_ADIPOGENESIS 176 0.5527 1.5e-05 7.6e-04
HALLMARK_ESTROGEN_RESPONSE_EARLY 184 0.5051 5.1e-05 2.6e-03
HALLMARK_IL2_STAT5_SIGNALING 180 -0.4872 1.1e-04 5.5e-03
HALLMARK_COMPLEMENT 190 -0.6643 7.8e-08 3.9e-06
HALLMARK_TNFA_SIGNALING_VIA_NFKB 186 -0.8591 1.1e-11 5.3e-10
HALLMARK_INFLAMMATORY_RESPONSE 192 -0.8825 1.4e-12 7.2e-11
HALLMARK_IL6_JAK_STAT3_SIGNALING 83 -1.1788 1.1e-09 5.4e-08
HALLMARK_ALLOGRAFT_REJECTION 189 -1.3512 4.4e-25 2.2e-23
HALLMARK_INTERFERON_GAMMA_RESPONSE 188 -1.6070 6.8e-33 3.4e-31
HALLMARK_INTERFERON_ALPHA_RESPONSE 87 -2.3000 1.3e-26 6.6e-25

The functions cut_by_NES and cut_by_logit2NES remove the rows having a NES/logit2NES below a given threshold. They are equivalent, in fact

\[logit2NES = \log_2 \frac{NES}{1-NES},\] and back \[NES = \frac{2^{logit2NES}}{1+2^{logit2NES}}.\]

The advantage of representing the Normalized Enrichment Score (\(NES\)) as \(logit2NES\) is that this last is a signed value: positive means association with the genes of the treatment group, while a negative value signals the association with the control samples. The default thresholds are set to 0.6 and 0.58 for \(NES\) and \(logit2NES\). These values say that the probability of association of the gene-set with the treatment group is 1.5 higher than the case of association with the control group.

Trimming the table of results according to the \(NES\)/\(logit2NES\) means giving much more attention to the descriptive interpretation of the NES as a measure of strongness of association. Given a gene-set, the \(NES\) is the percentile rank associated with the gene-set, seen as a single value (the average of the ranks), in the universe of the genes outside the gene-set.

tmp <- cut_by_significance(ans)
summary(cut_by_logit2NES(tmp), as.formattable = TRUE, order_by = "NES")
size actualSize NES logit2NES p.value BH.value B.value
HALLMARK_OXIDATIVE_PHOSPHORYLATION 200 177 0.6907 1.1591 2.2e-18 2.8e-17 1.1e-16
HALLMARK_COMPLEMENT 200 190 0.3869 -0.6643 7.8e-08 4.8e-07 3.9e-06
HALLMARK_TNFA_SIGNALING_VIA_NFKB 200 186 0.3554 -0.8591 1.1e-11 8.9e-11 5.3e-10
HALLMARK_INFLAMMATORY_RESPONSE 200 192 0.3517 -0.8825 1.4e-12 1.4e-11 7.2e-11
HALLMARK_IL6_JAK_STAT3_SIGNALING 87 83 0.3064 -1.1788 1.1e-09 7.8e-09 5.4e-08
HALLMARK_ALLOGRAFT_REJECTION 200 189 0.2816 -1.3512 4.4e-25 7.3e-24 2.2e-23
HALLMARK_INTERFERON_GAMMA_RESPONSE 200 188 0.2472 -1.6070 6.8e-33 3.4e-31 3.4e-31
HALLMARK_INTERFERON_ALPHA_RESPONSE 97 87 0.1688 -2.3000 1.3e-26 3.3e-25 6.6e-25 
summary(cut_by_NES(tmp), as.formattable = TRUE, order_by = "NES")
size actualSize NES logit2NES p.value BH.value B.value
HALLMARK_OXIDATIVE_PHOSPHORYLATION 200 177 0.6907 1.1591 2.2e-18 2.8e-17 1.1e-16
HALLMARK_COMPLEMENT 200 190 0.3869 -0.6643 7.8e-08 4.8e-07 3.9e-06
HALLMARK_TNFA_SIGNALING_VIA_NFKB 200 186 0.3554 -0.8591 1.1e-11 8.9e-11 5.3e-10
HALLMARK_INFLAMMATORY_RESPONSE 200 192 0.3517 -0.8825 1.4e-12 1.4e-11 7.2e-11
HALLMARK_IL6_JAK_STAT3_SIGNALING 87 83 0.3064 -1.1788 1.1e-09 7.8e-09 5.4e-08
HALLMARK_ALLOGRAFT_REJECTION 200 189 0.2816 -1.3512 4.4e-25 7.3e-24 2.2e-23
HALLMARK_INTERFERON_GAMMA_RESPONSE 200 188 0.2472 -1.6070 6.8e-33 3.4e-31 3.4e-31
HALLMARK_INTERFERON_ALPHA_RESPONSE 97 87 0.1688 -2.3000 1.3e-26 3.3e-25 6.6e-25

Plotting the analysis

The results can be displayed two-way: a bar plot and a network graph. The corresponding functions provide a graphical rendering of the table in input in both cases. Then, eventually, the table has to be trimmed according to significance or NES.

plot(ans)

A meaningful display follows.

plot(cut_by_significance(ans), top = 30)

Here, the maximum number of bars has been restricted to 30.

The horizontal axis (signed-NES) is a linear transformation of the NES: \[signed\mbox-NES = 2\cdot NES - 1\] necessary to a) signal the direction of the association, and b) to bound the bars between -1.0 and 1.0

The network plot needs to specify ‘as.network = TRUE’ and provide the gene-sets collection.

plot(cut_by_significance(ans), gene_sets = geneSets, as.network = TRUE)

The similarity \(S(A, B)\) between two gene-sets \(A\) and \(B\) comes from the convex combination \[S(A,B)= \epsilon \cdot \delta_1(A, B)+ (1-\epsilon)\cdot \delta_0(A, B),\] with \(0\leq \epsilon \leq 1\), and \(\delta_0\) is the Jaccard similarity, while \(\delta_1\) is the overlap index. \(logit2NES\) controls the color of the balls: red for those with positive values (associated with the treatment), green otherwise. \(actualSize\) controls the dimension of the balls.

A more intensive analysis: playing with more than 10,000 gene-sets

Here, we show a more serious analysis. A larger collection of gene-sets have be considered.

The nature of the gene profile included in the package is strictly according to the treatment versus control logic. In this case, 9 fusion FGFR3-TACC3 positive samples have been compared to 535 other samples. The interest is in the treatment, and then the analysis requires the alternative hypothesis “greater”.

system.time({C5BP_gs <- get_geneSets_from_msigdbr(category = "C5", 
                                                  subcategory = "BP", 
                                                  what = "gene_symbol")})
## msigdbr: R package version 7.4.1
##    user  system elapsed 
##  30.050   0.084  30.134
system.time({C5MF_gs <- get_geneSets_from_msigdbr(category = "C5", 
                                                  subcategory = "MF", 
                                                  what = "gene_symbol")})
## msigdbr: R package version 7.4.1
##    user  system elapsed 
##   1.314   0.092   1.406
system.time({C5CC_gs <- get_geneSets_from_msigdbr(category = "C5", 
                                                  subcategory = "CC", 
                                                  what = "gene_symbol")})
## msigdbr: R package version 7.4.1
##    user  system elapsed 
##   0.837   0.052   0.890
system.time({H_gs <- get_geneSets_from_msigdbr(category = "H", 
                                               what = "gene_symbol")})
## msigdbr: R package version 7.4.1
##    user  system elapsed 
##   0.104   0.008   0.112
# merging gene-sets collections
geneSets <- c(C5MF_gs, C5CC_gs, C5BP_gs, H_gs)
length(geneSets)
## [1] 10235
# running the analysis
system.time({ans <- massiveGST(geneProfile, geneSets, 
                               alternative = "greater")})
##    user  system elapsed 
##   6.241   0.004   6.245
# removing non significant results
ans <- cut_by_significance(ans, 
                           level_of_significance = 0.05, 
                           where = "bonferroni")

# Tabular results
summary(ans, as.formattable = TRUE, cols_to_remove = "BH.value")
collection size actualSize NES logit2NES p.value B.value relevance
HALLMARK_OXIDATIVE_PHOSPHORYLATION H 200 177 0.6907 1.1591 1.1e-18 1.1e-14 14604.5
GOBP_CELLULAR_RESPIRATION C5 BP 187 155 0.6948 1.1868 3.0e-17 3.1e-13 14571.0
GOBP_RESPIRATORY_ELECTRON_TRANSPORT_CHAIN C5 BP 117 93 0.7335 1.4605 3.7e-15 3.7e-11 14434.5
GOBP_ENERGY_DERIVATION_BY_OXIDATION_OF_ORGANIC_COMPOUNDS C5 BP 278 234 0.6480 0.8801 3.4e-15 3.5e-11 14399.0
GOCC_INNER_MITOCHONDRIAL_MEMBRANE_PROTEIN_COMPLEX C5 CC 142 87 0.7392 1.5029 6.3e-15 6.5e-11 14398.5
GOBP_ATP_SYNTHESIS_COUPLED_ELECTRON_TRANSPORT C5 BP 100 77 0.7450 1.5464 5.4e-14 5.6e-10 14333.5
GOBP_OXIDATIVE_PHOSPHORYLATION C5 BP 148 96 0.7013 1.2310 4.8e-12 4.9e-08 14326.0
GOCC_RESPIRASOME C5 CC 103 76 0.7300 1.4348 2.1e-12 2.1e-08 14272.5
GOCC_MITOCHONDRIAL_PROTEIN_CONTAINING_COMPLEX C5 CC 265 197 0.6412 0.8375 4.4e-12 4.4e-08 14265.0
GOBP_AEROBIC_RESPIRATION C5 BP 86 71 0.7338 1.4630 4.8e-12 4.9e-08 14236.0
GOCC_RESPIRATORY_CHAIN_COMPLEX C5 CC 87 65 0.7426 1.5288 6.7e-12 6.9e-08 14194.0
GOCC_MITOCHONDRIAL_MATRIX C5 CC 476 399 0.6137 0.6678 3.7e-15 3.7e-11 14123.5
GOCC_ORGANELLE_INNER_MEMBRANE C5 CC 552 413 0.6102 0.6466 8.8e-15 9.0e-11 14054.0
GOCC_OXIDOREDUCTASE_COMPLEX C5 CC 111 92 0.6582 0.9454 7.9e-08 8.1e-04 14003.5
GOBP_MITOCHONDRIAL_RESPIRATORY_CHAIN_COMPLEX_ASSEMBLY C5 BP 101 63 0.6899 1.1539 9.3e-08 9.5e-04 13952.5
GOBP_MITOCHONDRIAL_ELECTRON_TRANSPORT_NADH_TO_UBIQUINONE C5 BP 55 46 0.7415 1.5201 7.3e-09 7.5e-05 13910.0
GOBP_NADH_DEHYDROGENASE_COMPLEX_ASSEMBLY C5 BP 65 46 0.7138 1.3185 2.6e-07 2.7e-03 13811.0
GOCC_MITOCHONDRIAL_ENVELOPE C5 CC 778 603 0.5923 0.5386 6.1e-15 6.2e-11 13799.0
GOBP_ELECTRON_TRANSPORT_CHAIN C5 BP 178 147 0.6143 0.6713 8.8e-07 9.0e-03 13758.0
GOMF_OXIDOREDUCTASE_ACTIVITY_ACTING_ON_NAD_P_H_QUINONE_OR_SIMILAR_COMPOUND_AS_ACCEPTOR C5 MF 60 50 0.6901 1.1548 1.7e-06 1.7e-02 13757.5
GOBP_FATTY_ACID_BETA_OXIDATION C5 BP 75 66 0.6582 0.9452 4.4e-06 4.5e-02 13751.0
GOMF_NAD_P_H_DEHYDROGENASE_QUINONE_ACTIVITY C5 MF 48 39 0.7368 1.4848 1.6e-07 1.6e-03 13732.5
GOCC_NADH_DEHYDROGENASE_COMPLEX C5 CC 50 40 0.7211 1.3704 6.5e-07 6.7e-03 13701.0
GOBP_TRICARBOXYLIC_ACID_CYCLE C5 BP 34 32 0.8193 2.1808 2.0e-10 2.1e-06 13690.5
GOBP_ATP_METABOLIC_PROCESS C5 BP 313 240 0.5988 0.5780 6.9e-08 7.0e-04 13689.5
GOCC_MITOCHONDRION C5 CC 1627 1269 0.5787 0.4582 3.8e-21 3.9e-17 13601.0
GOBP_ACETYL_COA_METABOLIC_PROCESS C5 BP 38 35 0.7281 1.4212 1.5e-06 1.5e-02 13592.0
GOBP_VASCULAR_PROCESS_IN_CIRCULATORY_SYSTEM C5 BP 245 229 0.5924 0.5391 7.6e-07 7.7e-03 13543.0
GOBP_GENERATION_OF_PRECURSOR_METABOLITES_AND_ENERGY C5 BP 524 428 0.5823 0.4792 2.8e-09 2.9e-05 13505.5
GOBP_INNER_MITOCHONDRIAL_MEMBRANE_ORGANIZATION C5 BP 56 27 0.7649 1.7022 9.4e-07 9.6e-03 13406.5
GOCC_ENVELOPE C5 CC 1214 954 0.5683 0.3967 5.8e-13 5.9e-09 13308.0
GOBP_MITOCHONDRION_ORGANIZATION C5 BP 548 407 0.5686 0.3986 1.1e-06 1.1e-02 13163.0 
# Saving the table
fname <- file.path(tempdir(), "massiveGST_results.tsv")
save_as_xls(ans, file_name = fname)

# Inspecting the network of gene-sets
plot(ans, gene_sets = geneSets, as.network = TRUE)

Session info

sessionInfo()
## R version 4.1.3 (2022-03-10)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Linux Mint 20.3
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=it_IT.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=it_IT.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=it_IT.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=it_IT.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] massiveGST_1.0    visNetwork_2.0.9  igraph_1.2.6      WriteXLS_6.3.0   
## [5] msigdbr_7.4.1     formattable_0.2.1
## 
## loaded via a namespace (and not attached):
##  [1] highr_0.9         pillar_1.6.2      bslib_0.3.1       compiler_4.1.3   
##  [5] jquerylib_0.1.4   tools_4.1.3       digest_0.6.27     jsonlite_1.7.2   
##  [9] evaluate_0.14     lifecycle_1.0.0   tibble_3.1.3      pkgconfig_2.0.3  
## [13] rlang_0.4.11      DBI_1.1.1         yaml_2.2.1        xfun_0.25        
## [17] fastmap_1.1.0     stringr_1.4.0     dplyr_1.0.7       knitr_1.33       
## [21] generics_0.1.0    vctrs_0.3.8       htmlwidgets_1.5.3 sass_0.4.0       
## [25] tidyselect_1.1.1  glue_1.4.2        babelgene_21.4    R6_2.5.1         
## [29] fansi_0.5.0       rmarkdown_2.10    purrr_0.3.4       magrittr_2.0.1   
## [33] ellipsis_0.3.2    htmltools_0.5.2   assertthat_0.2.1  utf8_1.2.2       
## [37] stringi_1.7.3     crayon_1.4.1