Wrapper around unnest_tokens for characters and character shingles

Source: R/unnest_char.R

These functions are a wrapper around unnest_tokens( token = "characters" ) and unnest_tokens( token = "character_shingles" ).

unnest_characters(
  tbl,
  output,
  input,
  strip_non_alphanum = TRUE,
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

unnest_character_shingles(
  tbl,
  output,
  input,
  n = 3L,
  n_min = n,
  strip_non_alphanum = TRUE,
  format = c("text", "man", "latex", "html", "xml"),
  to_lower = TRUE,
  drop = TRUE,
  collapse = NULL,
  ...
)

Arguments

tbl

A data frame

output

Output column to be created as string or symbol.

input

Input column that gets split as string or symbol.

The output/input arguments are passed by expression and support quasiquotation; you can unquote strings and symbols.

strip_non_alphanum

Should punctuation and white space be stripped?

format

Either "text", "man", "latex", "html", or "xml". When the format is "text", this function uses the tokenizers package. If not "text", this uses the hunspell tokenizer, and can tokenize only by "word".

to_lower

Whether to convert tokens to lowercase.

drop

Whether original input column should get dropped. Ignored if the original input and new output column have the same name.

collapse

A character vector of variables to collapse text across, or NULL.

For tokens like n-grams or sentences, text can be collapsed across rows within variables specified by collapse before tokenization. At tidytext 0.2.7, the default behavior for collapse = NULL changed to be more consistent. The new behavior is that text is not collapsed for NULL.

Grouping data specifies variables to collapse across in the same way as collapse but you cannot use both the collapse argument and grouped data. Collapsing applies mostly to token options of "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", or "regex".

...

Extra arguments passed on to tokenizers

n

The number of characters in each shingle. This must be an integer greater than or equal to 1.

n_min

This must be an integer greater than or equal to 1, and less than or equal to n.

See also

Examples

library(dplyr)
library(janeaustenr)

d <- tibble(txt = prideprejudice)

d %>%
  unnest_characters(word, txt)
#> # A tibble: 536,529 × 1
#>    word 
#>    <chr>
#>  1 p    
#>  2 r    
#>  3 i    
#>  4 d    
#>  5 e    
#>  6 a    
#>  7 n    
#>  8 d    
#>  9 p    
#> 10 r    
#> # ℹ 536,519 more rows

d %>%
  unnest_character_shingles(word, txt, n = 3)
#> # A tibble: 517,422 × 1
#>    word 
#>    <chr>
#>  1 pri  
#>  2 rid  
#>  3 ide  
#>  4 dea  
#>  5 ean  
#>  6 and  
#>  7 ndp  
#>  8 dpr  
#>  9 pre  
#> 10 rej  
#> # ℹ 517,412 more rows