Download as PDF, PPTX
![Language Modeling
Objectives - MLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
['The', ‘pipeline’ 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
Tokenization
Masking
['The', ‘pipeline’, 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
‘Natural’
‘Artificial’
‘Machine’
‘Processing’
‘Speech’
Prediction](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-22-2048.jpg)
![Language Modeling
Objectives - CLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
Tokenization
Prediction
['Process', '##ing', '(', 'NL', '##P', ')', 'software', 'which', 'will', 'allow', 'a', 'user', 'to', 'develop']](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-23-2048.jpg)
![The tokenization pipeline
Inner workings
Normalization Pre-tokenization Tokenization Post-processing
▪ Add special tokens: for example [CLS], [SEP] with BERT
▪ Truncate to match the maximum length of the model
▪ Pad all sequence in a batch to the same length
▪ ...](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-38-2048.jpg)
![Transformers
Tokenization to prediction
transformers.PreTrainedTokenizer
transformers.PreTrainedModel
The pipeline for State-of-the-Art Natural Language Processing
[[464, 11523, 329, 1812, 12, ..., 15417, 28403]]
Tensor(batch_size, sequence_length, hidden_size) Task-specific prediction
Base model With task-specific head](https://image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-49-2048.jpg)
![Language Modeling
Objectives - MLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
['The', ‘pipeline’ 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
Tokenization
Masking
['The', ‘pipeline’, 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing']
‘Natural’
‘Artificial’
‘Machine’
‘Processing’
‘Speech’
Prediction](https://crownmelresort.com/image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-22-2048.jpg)
![Language Modeling
Objectives - CLM
['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing']
The pipeline for State-of-the-Art Natural Language Processing
Tokenization
Prediction
['Process', '##ing', '(', 'NL', '##P', ')', 'software', 'which', 'will', 'allow', 'a', 'user', 'to', 'develop']](https://crownmelresort.com/image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-23-2048.jpg)
![The tokenization pipeline
Inner workings
Normalization Pre-tokenization Tokenization Post-processing
▪ Add special tokens: for example [CLS], [SEP] with BERT
▪ Truncate to match the maximum length of the model
▪ Pad all sequence in a batch to the same length
▪ ...](https://crownmelresort.com/image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-38-2048.jpg)
![Transformers
Tokenization to prediction
transformers.PreTrainedTokenizer
transformers.PreTrainedModel
The pipeline for State-of-the-Art Natural Language Processing
[[464, 11523, 329, 1812, 12, ..., 15417, 28403]]
Tensor(batch_size, sequence_length, hidden_size) Task-specific prediction
Base model With task-specific head](https://crownmelresort.com/image.slidesharecdn.com/661lysandredebutanthonymoi-200629032103/75/Building-a-Pipeline-for-State-of-the-Art-Natural-Language-Processing-Using-Hugging-Face-Tools-49-2048.jpg)
Uploaded byDatabricks
Building a Pipeline for State-of-the-Art Natural Language Processing Using Hugging Face Tools
The document summarizes a presentation about state-of-the-art natural language processing (NLP) techniques. It discusses how transformer networks have achieved state-of-the-art results in many NLP tasks using transfer learning from large pre-trained models. It also describes how Hugging Face's Transformers library and Tokenizers library provide tools for tokenization and using pre-trained transformer models through a simple interface.
More Related Content
![[Paper Reading] Attention is All You Need](https://cdn.slidesharecdn.com/ss_thumbnails/reading20181228-190111054908-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Building a Pipeline for State-of-the-Art Natural Language Processing Using Hugging Face Tools
Building a Pipeline for State-of-the-Art Natural Language Processing Using Hugging Face Tools
- 2. The pipeline for State-of-the-ArtNLP Hugging Face
- 3. Agenda Lysandre DEBUT Machine LearningEngineer @ Hugging Face, maintainer and core contributor of huggingface/transformers Anthony MOI Technical Lead @ Hugging Face, maintainer and core contributor of huggingface/tokenizers Some slides were adapted from previous HuggingFace talk by Thomas Wolf, Victor Sanh and Morgan Funtowicz
- 4. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.
- 5.
- 6. Hugging Face Most popularopen source NLP library ▪ 1,000+ Research paper mentions ▪ Used in production by 1000+ companies
- 7.
- 8.
- 9. Subjects we’ll divein today ● NLP: Transfer learning, transformer networks ● Tokenizers: from text to tokens ● Transformers: from tokens to predictions
- 10. Transfer Learning -Transformer networks One big training to rule them all
- 11. NLP took aturn in 2018 Self-supervised Training & Transfer Learning Large Text Datasets Compute Power The arrival of the transformer architecture
- 12. Transfer learning In afew diagrams
- 13. Sequential transfer learning Learnon one task/dataset, transfer to another task/dataset word2vec GloVe skip-thought InferSent ELMo ULMFiT GPT BERT DistilBERT Text classification Word labeling Question-Answering .... Pre-training Adaptation Computationally intensive step General purpose model
- 14. Transformer Networks Very largemodels - State of the Art in several tasks
- 15. Transformer Networks ● Verylarge networks ● Can be trained on very big datasets ● Better than previous architectures at maintaining long-term dependencies ● Require a lot of compute to be trained Source: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova.In NACCL, 2019.
- 16. Transformer Networks Pre-training Base model Pre-trainedlanguage model Very large corpus $$$ in compute Days of training
- 17. Transformer Networks Fine-tuning Pre-trained language model Fine-tunedlanguage model Training can be done on single GPU Small dataset Easily reproducible
- 18. Model Sharing Reduced compute,cost, energy footprint From 🏎 Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT, by Victor Sanh
- 19. A deeper lookat the inner mechanisms Pipeline, pre-training, fine-tuning
- 20. Adaptation Head Pre-trained model Tokenizer Transfer Learning pipelinein NLP From text to tokens, from tokens to prediction Jim Henson was a puppet ##eer 11067 5567 245 120 7756 9908 1.2 2.7 0.6 -0.2 3.7 9.1 -2.1 3.1 1.5 -4.7 2.4 6.7 6.1 2.4 7.3 -0.6 -3.1 2.5 1.9 -0.1 0.7 2.1 4.2 -3.1 True 0.7886 False 0.223 Jim Henson was a puppeteer Tokenization Convert to vocabulary indices Pre-trained model Task-specificmodel
- 21. Pre-training Many currently successfulpre-training approaches are based on language modeling: learning to predict Pϴ (text) or Pϴ (text | other text) Advantages: - Doesn’t require human annotation - self-supervised - Many languages have enough text to learn high capacity models - Versatile - can be used to learn both sentence and word representations with a variety of objective functions The rise of language modeling pre-training
- 22. Language Modeling Objectives -MLM ['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing'] The pipeline for State-of-the-Art Natural Language Processing ['The', ‘pipeline’ 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing'] Tokenization Masking ['The', ‘pipeline’, 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', [MASK], 'Language', 'Process', '##ing'] ‘Natural’ ‘Artificial’ ‘Machine’ ‘Processing’ ‘Speech’ Prediction
- 23. Language Modeling Objectives -CLM ['The', 'pipeline', 'for', 'State', '-', 'of', '-', 'the', '-', 'Art', 'Natural', 'Language', 'Process', '##ing'] The pipeline for State-of-the-Art Natural Language Processing Tokenization Prediction ['Process', '##ing', '(', 'NL', '##P', ')', 'software', 'which', 'will', 'allow', 'a', 'user', 'to', 'develop']
- 24. Tokenization It doesn’t haveto be slow
- 25. Tokenization - Convert inputstrings to a set of numbers Its role in the pipeline Jim Henson was a puppet ##eer 11067 5567 245 120 7756 9908 Jim Henson was a puppeteer - Goal: Find the most meaningful and smallest possible representation
- 26. Some examples Let’s divein the nitty-gritty
- 27. Word-based Word by wordtokenization Let’s do tokenization! Let ‘s do tokenization ! Split on punctuation: Split on spaces: ▪ Split on spaces, or following specific rules to obtain words ▪ What to do with punctuation? ▪ Requires large vocabularies: dog != dogs, run != running ▪ Out-of-vocabulary (aka <UNK>) tokens for unknown words
- 28. Character Character by charactertokenization ▪ Split on characters individually ▪ Do we include spaces or not? ▪ Smaller vocabularies ▪ But lack of meaning -> Characters don’t necessarily have a meaning separately ▪ End up with a huge amount of tokens to be processed by the model L e t ‘ s d o t o k e n i z a t i o n !
- 29. Byte Pair Encoding Welcomesubword tokenization ▪ First introduced by Philip Gage in 1994, as a compression algorithm ▪ Applied to NLP by Rico Sennrich et al. in “Neural Machine Translation of Rare Words with Subwords Units”. ACL 2016.
- 30. Byte Pair Encoding Welcomesubword tokenization A B C ... a b c ... ? ! ... Initial alphabet: ▪ Start with a base vocabulary using Unicode characters seen in the data ▪ Most frequent pairs get merged to a new token: 1. T + h => Th 2. Th + e => The
- 31. Byte Pair Encoding Welcomesubword tokenization ▪ Less out-of-vocabulary tokens ▪ Smaller vocabularies Let’s</w> do</w> token ization</w> !</w>
- 32. And a lotmore So many algorithms... ▪ Byte-level BPE as used in GPT-2 (Alec Radford et al. OpenAI) ▪ WordPiece as used in BERT (Jacob Devlin et al. Google) ▪ SentencePiece (Unigram model) (Taku Kudo et al. Google)
- 33. Tokenizers Why did webuild it? ▪ Performance ▪ One API for all the different tokenizers ▪ Easy to share and reproduce your work ▪ Easy to use any tokenizer, and re-train it on a new language/dataset
- 34. The tokenization pipeline Innerworkings Normalization Pre-tokenization Tokenization Post-processing
- 35. The tokenization pipeline Innerworkings Normalization Pre-tokenization Tokenization Post-processing ▪ Strip ▪ Lowercase ▪ Removing diacritics ▪ Deduplication ▪ Unicode normalization (NFD, NFC, NFKC, NFKD)
- 36. The tokenization pipeline Innerworkings Normalization Pre-tokenization Tokenization Post-processing ▪ Set of rules to split: - Whitespace use - Punctuation use - Something else?
- 37. The tokenization pipeline Innerworkings Normalization Pre-tokenization Tokenization Post-processing ▪ Actual tokenization algorithm: - BPE - Unigram - Word level
- 38. The tokenization pipeline Innerworkings Normalization Pre-tokenization Tokenization Post-processing ▪ Add special tokens: for example [CLS], [SEP] with BERT ▪ Truncate to match the maximum length of the model ▪ Pad all sequence in a batch to the same length ▪ ...
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46. Transformers Using complex modelsshouldn’t be complicated
- 47. Transformers An explosion ofTransformer architectures ▪ Wordpiece tokenization ▪ MLM & NSP BERT ALBERT GPT-2 ▪ SentencePiece tokenization ▪ MLM & SOP ▪ Repeating layers ▪ Byte-level BPE tokenization ▪ CLM Same API
- 48. Transformers As flexible aspossible Runs and trains on: ▪ CPU ▪ GPU ▪ TPU With optimizations: ▪ XLA ▪ TorchScript ▪ Half-precision ▪ Others All models BERT & RoBERTa More to come!
- 49. Transformers Tokenization to prediction transformers.PreTrainedTokenizer transformers.PreTrainedModel Thepipeline for State-of-the-Art Natural Language Processing [[464, 11523, 329, 1812, 12, ..., 15417, 28403]] Tensor(batch_size, sequence_length, hidden_size) Task-specific prediction Base model With task-specific head
- 50. Transformers Available pre-trained models transformers.PreTrainedTokenizer transformers.PreTrainedModel ▪We publicly host pre-trained tokenizer vocabularies and model weights ▪ 1611 model/tokenizer pairs at the time of writing
- 51. Transformers Pipelines transformers.Pipeline ▪ Pipelines handleboth the tokenization and prediction ▪ Reasonable defaults ▪ SOTA models ▪ Customizable
- 52. A few use-cases That’swhere it gets interesting
- 53.
- 54.
- 55.
- 56. Transformers Sequence Classification -Under the hood
- 57. Transformers Sequence Classification -Under the hood
- 58. Transformers Sequence Classification -Under the hood
- 59. Transformers Sequence Classification -Under the hood
- 60. Transformers Sequence Classification -Under the hood
- 61. Transformers Sequence Classification -Under the hood
- 62. Transformers Training models Example scripts(TensorFlow & PyTorch) - Named Entity Recognition - Sequence Classification - Question Answering - Language modeling (fine-tuning & from scratch) - Multiple Choice Trains on TPU, CPU, GPU Example scripts for PyTorch Lightning
- 63. Transformers Just grazed thesurface The transformers library covers a lot more ground: - ELECTRA - Reformer - Longformer - Encoder-decoder architectures - Translation & Summarization
- 64. Transformers + Tokenizers Thefull pipeline? Data Tokenization Prediction 🤗 nlp Tokenizers Transformers Metrics 🤗 nlp
- 65. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.