In Files

Methods

Class Index [+]

Quicksearch

Ferret::Analysis::AsciiStandardTokenizer

Summary

The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.

Example

  "Dave's résumé, at http://www.davebalmain.com/ 1234"
    => ["Dave's", "r", "sum", "at", "http://www.davebalmain.com", "1234"]

Public Class Methods

new() → tokenizer click to toggle source

Create a new AsciiStandardTokenizer

static VALUE
frb_a_standard_tokenizer_init(VALUE self, VALUE rstr) 
{
    return get_wrapped_ts(self, rstr, standard_tokenizer_new());
}

Disabled; run with --debug to generate this.

[Validate]

Generated with the Darkfish Rdoc Generator 1.1.6.