From the course: Introduction to Large Language Models

Unlock the full course today

Join today to access over 24,900 courses taught by industry experts.

PaLM and PaLM 2

PaLM and PaLM 2

From the course: Introduction to Large Language Models

Start my 1-month free trial Buy for my team

PaLM and PaLM 2

“

- [Instructor] We're finally at one of the largest language models to date. Google released PaLM, or to give it its full name, the Pathways Language Model, in April 2022. There are a couple of key takeaways from the PaLM model. PaLM is the largest model with 540 billion parameters. Google uses the Pathway systems, which meant more hardware accelerators could be used for model training, and they could be trained more efficiently. PaLM was trained on an enormous 780 billion tokens multilingual corpus with text from over 100 languages. Now, just over 3/4 of this training data was in English. Another really interesting phenomena that the Google team picked up on was on scaling. It looked like the models could only perform certain tasks once a certain scale was reached. Here, 8 billion parameter models could perform certain tasks such as question answering, language understanding, and arithmetic. It was only when the model was…

Contents