These functions are wrappers around unnest_tokens( token = "ngrams" )
and unnest_tokens( token = "skip_ngrams" )
.
unnest_ngrams(
tbl,
output,
input,
n = 3L,
n_min = n,
ngram_delim = " ",
format = c("text", "man", "latex", "html", "xml"),
to_lower = TRUE,
drop = TRUE,
collapse = NULL,
...
)
unnest_skip_ngrams(
tbl,
output,
input,
n_min = 1,
n = 3,
k = 1,
format = c("text", "man", "latex", "html", "xml"),
to_lower = TRUE,
drop = TRUE,
collapse = NULL,
...
)
A data frame
Output column to be created as string or symbol.
Input column that gets split as string or symbol.
The output/input arguments are passed by expression and support quasiquotation; you can unquote strings and symbols.
The number of words in the n-gram. This must be an integer greater than or equal to 1.
The minimum number of words in the n-gram. This must be an
integer greater than or equal to 1, and less than or equal to n
.
The separator between words in an n-gram.
Either "text", "man", "latex", "html", or "xml". When the format is "text", this function uses the tokenizers package. If not "text", this uses the hunspell tokenizer, and can tokenize only by "word".
Whether to convert tokens to lowercase.
Whether original input column should get dropped. Ignored if the original input and new output column have the same name.
A character vector of variables to collapse text across,
or NULL
.
For tokens like n-grams or sentences, text can be collapsed across rows
within variables specified by collapse
before tokenization. At tidytext
0.2.7, the default behavior for collapse = NULL
changed to be more
consistent. The new behavior is that text is not collapsed for NULL
.
Grouping data specifies variables to collapse across in the same way as
collapse
but you cannot use both the collapse
argument and
grouped data. Collapsing applies mostly to token
options of "ngrams",
"skip_ngrams", "sentences", "lines", "paragraphs", or "regex".
Extra arguments passed on to tokenizers
For the skip n-gram tokenizer, the maximum skip distance between
words. The function will compute all skip n-grams between 0
and
k
.
library(dplyr)
library(janeaustenr)
d <- tibble(txt = prideprejudice)
d %>%
unnest_ngrams(word, txt, n = 2)
#> # A tibble: 114,045 × 1
#> word
#> <chr>
#> 1 pride and
#> 2 and prejudice
#> 3 NA
#> 4 by jane
#> 5 jane austen
#> 6 NA
#> 7 NA
#> 8 NA
#> 9 chapter 1
#> 10 NA
#> # ℹ 114,035 more rows
d %>%
unnest_skip_ngrams(word, txt, n = 3, k = 1)
#> # A tibble: 700,179 × 1
#> word
#> <chr>
#> 1 pride
#> 2 pride and
#> 3 pride prejudice
#> 4 pride and prejudice
#> 5 and
#> 6 and prejudice
#> 7 prejudice
#> 8 NA
#> 9 by
#> 10 by jane
#> # ℹ 700,169 more rows