helpers.tokenization.BertTokenizer Class Reference

Inheritance diagram for helpers.tokenization.BertTokenizer:

Collaboration diagram for helpers.tokenization.BertTokenizer:

Public Member Functions
def	__init__ (self, vocab_file, do_lower_case=True)
def	tokenize (self, text)
def	convert_tokens_to_ids (self, tokens)
def	convert_ids_to_tokens (self, ids)

Public Attributes
	vocab
	ids_to_tokens
	basic_tokenizer
	wordpiece_tokenizer

Detailed Description

Runs end-to-end tokenization: punctuation splitting + wordpiece

Constructor & Destructor Documentation

__init__()

def helpers.tokenization.BertTokenizer.__init__	(		self,
			vocab_file,
			do_lower_case = True
	)

Member Function Documentation

tokenize()

def helpers.tokenization.BertTokenizer.tokenize	(		self,
			text
	)

convert_tokens_to_ids()

def helpers.tokenization.BertTokenizer.convert_tokens_to_ids	(		self,
			tokens
	)

Converts a sequence of tokens into ids using the vocab.

convert_ids_to_tokens()

def helpers.tokenization.BertTokenizer.convert_ids_to_tokens	(		self,
			ids
	)

Converts a sequence of ids in wordpiece tokens using the vocab.

Member Data Documentation

vocab

helpers.tokenization.BertTokenizer.vocab

ids_to_tokens

helpers.tokenization.BertTokenizer.ids_to_tokens

basic_tokenizer

helpers.tokenization.BertTokenizer.basic_tokenizer

wordpiece_tokenizer

helpers.tokenization.BertTokenizer.wordpiece_tokenizer

---

The documentation for this class was generated from the following file:

• tokenization.py