unnest_ngrams(
  tbl,
  output,
  input,
  n = 3L,
  n_min = n,
  ngram_delim = " ",
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

unnest_skip_ngrams(
  tbl,
  output,
  input,
  n_min = 1,
  n = 3,
  k = 1,
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

Arguments

tbl

A data frame

output

Output column to be created as string or symbol.

input

Input column that gets split as string or symbol.

The output/input arguments are passed by expression and support quasiquotation; you can unquote strings and symbols.

n

The number of words in the n-gram. This must be an integer greater than or equal to 1.

n_min

This must be an integer greater than or equal to 1, and less than or equal to n.

ngram_delim

The separator between words in an n-gram.

format

Either "text", "man", "latex", "html", or "xml". If not text, this uses the hunspell tokenizer, and can tokenize only by "word"

to_lower

Whether to convert tokens to lowercase. If tokens include URLS (such as with token = "tweets"), such converted URLs may no longer be correct.

drop

Whether original input column should get dropped. Ignored if the original input and new output column have the same name.

collapse

Whether to combine text with newlines first in case tokens (such as sentences or paragraphs) span multiple lines. If NULL, collapses when token method is "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", or "regex".

...

Extra arguments passed on to tokenizers

k

For the skip n-gram tokenizer, the maximum skip distance between words. The function will compute all skip n-grams between 0 and k.

See also

Examples

library(dplyr) library(janeaustenr) d <- tibble(txt = prideprejudice) d %>% unnest_ngrams(word, txt, n = 2)
#> # A tibble: 122,203 x 1 #> word #> <chr> #> 1 pride and #> 2 and prejudice #> 3 prejudice by #> 4 by jane #> 5 jane austen #> 6 austen chapter #> 7 chapter 1 #> 8 1 it #> 9 it is #> 10 is a #> # … with 122,193 more rows
d %>% unnest_skip_ngrams(word, txt, n = 3, k = 1)
#> # A tibble: 855,413 x 1 #> word #> <chr> #> 1 pride #> 2 pride and #> 3 pride prejudice #> 4 pride and prejudice #> 5 pride and by #> 6 pride prejudice by #> 7 pride prejudice jane #> 8 and #> 9 and prejudice #> 10 and by #> # … with 855,403 more rows