• 원본 링크 : https://keras.io/api/keras_nlp/tokenizers/
• 최종 수정일 : 2024-03-29

KerasNLP Tokenizers

목차

---

---

Table of contents

• Tokenizer base class
• WordPieceTokenizer
• SentencePieceTokenizer
• BytePairTokenizer
• ByteTokenizer
• UnicodeCodepointTokenizer
• compute_word_piece_vocabulary function
• compute_sentence_piece_proto function