Gemini 1.5 Pro Developer Insights

Re: Gemini 1.5’s 1M token window, I saw Mike Kaput test it on the Marketing AI Podcast with Paul Roetzer using a 500-page (possibly boring) government document. It worked so well I needed to try it. This week Gemini expanded its capabilities to have it ingest video, so I started thinking of ways to experiment. My test: summarizing a movie from the Public Domain - the Night of the Living Dead.   First, I could only get about 40 minutes of the movie in – a little over 715,000 tokens. Gemini 1.5 Pro seemed to balk at higher amounts even though the token window should have been able to handle it.  Still, it did an excellent job of summarizing the video. Really excellent.  The use cases:  1. If you have a large repository of videos from webinars, you could summarize them for better engagement and more engagement.  2. If you are producing video content today, this gives you another and better path for doing summaries. Sure, recorders can get a transcript and build a summary from the transcript, but so far, I’ve found these summaries to be just okay.  The irony is that Gemini will let you do this with a video you upload, but not by pointing it at a YouTube video. Once this happens, think of the applications.    Also, I think Gemini 1.5 Pro seems to handle transcript summaries better than the other LLMs even though the token count is low.  

I was fortunate to receive an invitation for early access to Google's new Gemini 1.5 pro model, which boasts a 1 million token context window. If you want to experiment with it, here are a few things you need to know to get started. It was released yesterday to the public in a low-key announcement primarily aimed at developers. 1. You can access it in AI Studio. (Link in comments) 2. AI Studio is free. 3. In AI Studio, the interface doesn't natively save your chat history. (It is designed for developers to test prompts in different ways with models.) However, you can save your prompts to a library. (Note: Officially, it doesn't save chat history...But I have noticed my last few saved prompts include the chat history, so I hope that is a newly upgraded feature since they are improving it continuously.) 4. You can test prompts in different models in three ways: a chat interface, freeform prompts, and structured prompts. You can learn how each type works using their tutorials. 5. With the Gemini 1.5 Pro model, you can, for the first time, upload video to an LLM as an input 🤯 6. The video, however, does not have audio modality - for now. Technically, the AI is ingesting the video frame by frame as stills, but it can read timestamps in the video. 7. For any response, you can use the "get code" button to get the lines of code vs text, which you can copy and paste. 8. Expect responses (especially with video inputs) to take a bit longer than you are used to with smaller context text only or text plus images inputs. This early peek at Gemini 1.5 pro is mind-blowing, especially considering it is still in its most primitive state. Iterative releases will only improve it from here.  Using it over these last few weeks has already changed my perspective on much of the progress made in AI in the past several years. I will share more of my thoughts about that soon, but for now, I wanted to share the tips on access and how to use it so that you can also get a peek into it and try it out over the weekend. #ai #google #gemini

The release of Google's Gemini Pro 1.5 is, IMO, the biggest piece of A.I. news yet this year. The LLM has a gigantic million-token context window, multimodal inputs (text, code, image, audio, video) and GPT-4-like capabilities despite being much smaller and faster. Key Features 1. Despite being a mid-size model (so much faster and cheaper), its capabilities rival the full-size models Gemini Ultra 1.0 and GPT-4, which are the two most capable LLMs available today. 2. At a million tokens, its context window demolishes Claude 2, the foundation LLM with the next longest context window (Claude 2's is only a fifth of the size at 200k). A million tokens corresponds to 700,000 words (seven lengthy novels) and Gemini Pro 1.5 accurately retrieves needles from this vast haystack 99% of the time! 3. Accepts text, code, images, audio (a million tokens corresponds to 11 hours of audio), and video (1MM tokens = an hour of video). Today's episode contains an example of Gemini Pro 1.5 answering my questions about a 54-minute-long video with astounding accuracy and grace. How did Google pull this off? • Gemini Pro 1.5 is a Mixture-of-Experts (MoE) architecture, routing your input to specialized submodels (e.g., one for math, one for code, etc.), depending on the broad topic of your input. This allows for focused processing and explains both the speed gains and high capability level despite being a mid-size model. • While OpenAI also uses the MoE approach in GPT-4, Google seems to have achieved greater efficiency with the approach. This edge may stem from Google's pioneering work on MoE (Google were the first to publish on MoE, way back in 2017) and their resultant deep in-house expertise on the topic. • Training-data quality is also a likely factor in Google's success. What's next? • Google has 10-million-token context-windows in testing. That order-of-magnitude jump would correspond to future Gemini releases being able to handle ~70 novels, 100 hours of audio or 10 hours of video. • If Gemini Pro 1.5 can achieve GPT-4-like capabilities, the Gemini Ultra 1.5 release I imagine is in the works may allow Google to leapfrog OpenAI and reclaim their crown as the world's undisputed A.I. champions (unless OpenAI gets GPT-5 out first)! Want access? • Gemini Pro 1.5 is available with a 128k context window through Google AI Studio and (for enterprise customers) through Google Cloud's Vertex AI. • There's a waitlist for access to the million-token version (I had access through the early-tester program). Check out today's episode (#762) for more detail on all of the above (including Gemini 1.5 Pro access/waitlist links). The Super Data Science Podcast is available on all major podcasting platforms and a video version is on YouTube. #superdatascience #machinelearning #ai #llms #geminipro #geminiultra