Google's new large language model, Gemini, is introduced and its capabilities are compared to GPT-4.

• Gemini outperforms GPT-4 on nearly all benchmarks including reading comprehension, math, and spatial reasoning.
• Gemini falls short only in completing each other's sentences.
• The presenter highlights Google's hands-on demo where Gemini interacts with a video feed to play games such as one-ball-three-cups.

The presenter critiques the hands-on demo of Gemini, arguing that it is highly edited and does not accurately represent Gemini's capabilities.

• Despite the impressive display, the presenter argues that the demo does not represent real-time interaction with a video stream.
• Gemini's abilities are due to multimodal prompting, combining text and still images from the video.
• Google's blog post is credited for explaining how these demos work, but the presenter believes there's more prompt engineering involved than the video suggests.

A discussion of the controversy around the benchmarks Gemini is compared against, with a focus on the Massive Multitask Language Understanding benchmark.

• Gemini is claimed to be the first model to surpass human experts on the Massive Multitask Language Understanding benchmark, which covers 57 different subjects.
• The presenter criticizes the comparison of GPT-4's 5-shot benchmark with Gemini's Chain of Thought-32 benchmark, stating that it's not a fair comparison.
• In an 'apples to apples' comparison, GPT-4's Chain of Thought benchmark score is higher than Gemini's 5-shot benchmark score.

The presenter discusses the importance of actual user experience over benchmark scores in assessing AI performance and shares thoughts on Gemini's future implications.

• The presenter warns viewers not to trust benchmarks, especially those not from a neutral third party.
• He shares his positive experience using GPT-4 and his skepticism towards Gemini due to its unavailability for public use.
• While acknowledging Google's resources and capabilities, the presenter expresses his intention to reserve judgment on Gemini until it is released for public use.