helpers.tokenization.FullTokenizer Class Reference

Inheritance diagram for helpers.tokenization.FullTokenizer:

Collaboration diagram for helpers.tokenization.FullTokenizer:

Public Member Functions
def	__init__ (self, vocab_file, do_lower_case=True)
def	tokenize (self, text)
def	convert_tokens_to_ids (self, tokens)
def	convert_ids_to_tokens (self, ids)

Public Attributes
	vocab
	inv_vocab
	basic_tokenizer
	wordpiece_tokenizer

Detailed Description

Runs end-to-end tokenziation.

Constructor & Destructor Documentation

__init__()

def helpers.tokenization.FullTokenizer.__init__	(		self,
			vocab_file,
			do_lower_case = True
	)

Member Function Documentation

tokenize()

def helpers.tokenization.FullTokenizer.tokenize	(		self,
			text
	)

convert_tokens_to_ids()

def helpers.tokenization.FullTokenizer.convert_tokens_to_ids	(		self,
			tokens
	)

Here is the call graph for this function:

convert_ids_to_tokens()

def helpers.tokenization.FullTokenizer.convert_ids_to_tokens	(		self,
			ids
	)

Here is the call graph for this function:

Member Data Documentation

vocab

helpers.tokenization.FullTokenizer.vocab

inv_vocab

helpers.tokenization.FullTokenizer.inv_vocab

basic_tokenizer

helpers.tokenization.FullTokenizer.basic_tokenizer

wordpiece_tokenizer

helpers.tokenization.FullTokenizer.wordpiece_tokenizer

---

The documentation for this class was generated from the following file:

• tokenization.py