text.cleaners Namespace Reference

Functions
def	expand_abbreviations (text)
def	expand_numbers (text)
def	lowercase (text)
def	collapse_whitespace (text)
def	convert_to_ascii (text)
def	basic_cleaners (text)
def	transliteration_cleaners (text)
def	english_cleaners (text)

Variables
	_whitespace_re = re.compile(r'\s+')
list	_abbreviations

Function Documentation

expand_abbreviations()

def text.cleaners.expand_abbreviations

(

text

)

Here is the caller graph for this function:

expand_numbers()

def text.cleaners.expand_numbers

(

text

)

Here is the caller graph for this function:

lowercase()

def text.cleaners.lowercase

(

text

)

Here is the caller graph for this function:

collapse_whitespace()

def text.cleaners.collapse_whitespace

(

text

)

Here is the caller graph for this function:

convert_to_ascii()

def text.cleaners.convert_to_ascii

(

text

)

Here is the caller graph for this function:

basic_cleaners()

def text.cleaners.basic_cleaners

(

text

)

Basic pipeline that lowercases and collapses whitespace without transliteration.

Here is the call graph for this function:

transliteration_cleaners()

def text.cleaners.transliteration_cleaners

(

text

)

Pipeline for non-English text that transliterates to ASCII.

Here is the call graph for this function:

english_cleaners()

def text.cleaners.english_cleaners

(

text

)

Pipeline for English text, including number and abbreviation expansion.

Here is the call graph for this function:

Variable Documentation

_whitespace_re

text.cleaners._whitespace_re = re.compile(r'\s+')

private

_abbreviations

list text.cleaners._abbreviations

private

Initial value:

=  [(re.compile('\\b%s\\.' % x[0], re.IGNORECASE), x[1]) for x in [
  ('mrs', 'misess'),
  ('mr', 'mister'),
  ('dr', 'doctor'),
  ('st', 'saint'),
  ('co', 'company'),
  ('jr', 'junior'),
  ('maj', 'major'),
  ('gen', 'general'),
  ('drs', 'doctors'),
  ('rev', 'reverend'),
  ('lt', 'lieutenant'),
  ('hon', 'honorable'),
  ('sgt', 'sergeant'),
  ('capt', 'captain'),
  ('esq', 'esquire'),
  ('ltd', 'limited'),
  ('col', 'colonel'),
  ('ft', 'fort'),
]]