# Text Tokenization using Byte Pair Encoding and Unigram Modelling

## Documentation for package ‘sentencepiece’ version 0.1.2

- DESCRIPTION file.

## Help Pages

read_word2vec | Read a word2vec embedding file
sentencepiece | Construct a Sentencepiece model
sentencepiece_decode | Decode encoded sequences back to text
sentencepiece_download_model | Download a Sentencepiece model
sentencepiece_encode | Tokenise text alongside a Sentencepiece model
sentencepiece_load_model | Load a Sentencepiece model
wordpiece_encode | Wordpiece encoding