unnest_sentences(
  tbl,
  output,
  input,
  strip_punct = FALSE,
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

unnest_lines(
  tbl,
  output,
  input,
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

unnest_paragraphs(
  tbl,
  output,
  input,
  paragraph_break = "\n\n",
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

Arguments

tbl

A data frame

output

Output column to be created as string or symbol.

input

Input column that gets split as string or symbol.

The output/input arguments are passed by expression and support quasiquotation; you can unquote strings and symbols.

strip_punct

Should punctuation be stripped?

format

Either "text", "man", "latex", "html", or "xml". If not text, this uses the hunspell tokenizer, and can tokenize only by "word"

to_lower

Whether to convert tokens to lowercase. If tokens include URLS (such as with token = "tweets"), such converted URLs may no longer be correct.

drop

Whether original input column should get dropped. Ignored if the original input and new output column have the same name.

collapse

Whether to combine text with newlines first in case tokens (such as sentences or paragraphs) span multiple lines. If NULL, collapses when token method is "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", or "regex".

...

Extra arguments passed on to tokenizers

paragraph_break

A string identifying the boundary between two paragraphs.

See also

Examples

library(dplyr) library(janeaustenr) d <- tibble(txt = prideprejudice) d %>% unnest_sentences(word, txt)
#> # A tibble: 7,066 x 1 #> word #> <chr> #> 1 "pride and prejudice by jane austen chapter 1 it is a truth universall… #> 2 "however little known the feelings or views of such a man may be on his firs… #> 3 "\"my dear mr." #> 4 "bennet,\" said his lady to him one day, \"have you heard that netherfield p… #> 5 "mr." #> 6 "bennet replied that he had not." #> 7 "\"but it is,\" returned she; \"for mrs." #> 8 "long has just been here, and she told me all about it.\"" #> 9 "mr." #> 10 "bennet made no answer." #> # … with 7,056 more rows