Gemini 3.0 Pro – Early Tests

The things that I see represented here, may or may not be impressive, but sure as hell have never been the major blockers in achieving progress on complex tasks and software.

I understand you're merely reporting, thank you for that, not criticizing you, but those tests are absolutely irrelevant.

Is this supposed to be a good example?

It looks like something I'd put together, and you don't want me doing design work.

It may well be that they also didn't have a product culture as an organization, but were willing to experiment or let small teams do so.

It's still a lesson, but maybe a different one.

With organizational scale it becomes harder and harder to launch experiments under the brand. Red tape increases, outside scrutiny increases. Retaining the ability to do that is difficult.

Google does experiment a fair bit (including in AI, e.g. NotebookLLM and its podcast feature are I think a standout example of trying to see what sticks) but they also tend to try to hide their experiments in developer portals nowadays, which makes it difficult to get a signal from a general consumer audience.

Google's AI offering is a complete nightmare to use. Three different APIs, at least two different subscriptions, documentation that uses them interchangeably.

For Gemini's API it's often much simpler to actually pay OpenRouter the 5% surchargeto BYOK than deal with it all.

I still can't use my Google AI Pro account with gemini-cli..

This is evident in Android and the pixel lineup, which could be my favorite phone if not for some of the most baffling and frustrating decisions that lead to a very weirdly disjointed app experience (comparing to something like iOS's first party tools).

Like removing location based reminders from google tasks, for some reason? Still no apple shortcuts-like automation built-in, keep can still do location based reminders but it's a notes app so which am I supposed to use? Google tasks or keep? Well, gemini adds reminders to google tasks and not keep if I wanted to use keep primarily.

If they just spent some time polishing and integrating these tools, and add some of their ML magic to it they'd blow Apple out of the park.

All of Google's tech is cool and interesting, from a tech standpoint but it's not well integrated for a full consumer experience.

It would be weird to release that as a serious company. They tried making a deliberately-wacky chatbot but it was not fun.

Letting OpenAI to release it first was a right move.

Fair criticism that it took someone else to make something of the tech that Google initially invented, but Google is furiously experimenting with all their active products since Sundar's "code red" memo.

Without RLHF, LLM-based chat was a psychotic liability.

Nearly all the people that matter use iPhone... Yet Apple really hasn't had much success in the AI world, despite being in a position to win if their product is even only vaguely passable.

The fact that we had Attention Is All You Need was freely available online alone was unbelievably fortunate from hindsight.

It’s amazing how pervasive company cultures can be, and how this comes from the top, and can only be fixed with replacing leadership with an extremely talented CEO that knows the company inside out and can change its course. Nadella from Microsoft comes to mind, although that was more about Microsoft going back to its roots (replace sales oriented leadership with product oriented leadership again).

Google never had product oriented leadership in the same way that Amazon, Apple and Microsoft had.

I don’t think this will ever change at this point.

For those who haven’t read it, Steve Yegge’s rant about Google is worth your time:

1 https://gist.github.com/chitchcock/1281611

Now Grok is publicly boasting PhD level reasoning while Surge AI and Scale AI are focusing on high quality datasets curated by actual PhD humans.

Surge AI is boasting $1B in revenue, and I am wondering how much of that was paid in X.ai stock: https://podcasts.apple.com/us/podcast/the-startup-powering-t...

In my opinion the major advancements of 2025 have been more efficient models. They have made smaller models much, much better (including MoE models) but have failed to meaningfully push the SoTA on huge models; at least when looking at the USA companies.

The other day I asked 2.5 Pro for suggestions. It would provide one, which I rejected with some reasoning. It would provide another, which I also rejected. Asked for more it would then loop between the two, repeating the previous suggestions verbatim. It went on for 3-4 times, even after being told to reflect on it and it being able to recite the rejection reasons.

* Gemini has the highest ceiling out of all of the models, but has consistently struggled with token-level accuracy. In other words, it's conceptual thinking it well beyond other models, but it sometimes makes stupid errors when talking. This makes it hard to reliably use for tool calling or structured output. Gemini is also very hard to steer, so when it's wrong, it's really hard to correct.

* Claude is extremely consistent and reliable. It's very, very good at the details - but will start to forget things if things get too complex. The good news is Claude is very steerable and will remember those details if you remind it.

* GPT-5 seems to be completely random for me. It's so inconsistent that it's extremely hard to use.

I tend to use Claude because I'm the most familiar with it and I'm confident that I can get good results out of it.

It's like the personality of a person. Employee A is better at talking to customers than Employee B, but Employee B is better at writing code than Employee A. Is one better than the other? Is one smarter than the other? Nope. Different training data.

I run claude CLI as a primary and just ask it nicely to consult gemini cli (but not let it do any coding). It works surprisingly well. OpenAI just fell out of my view. Even cancelled ChatGPT subscription. Gemini is leaping forward and _feels like_ ChatGPT-5 is a regression.. I can't put my finger on it tbh.

One advantage Gemini had (or still has, I’m not sure about the other providers) was its large context window combined with the ability to use PDF documents. It probably saved me weeks of work on an integration with a government system uploading hundreds of pages of documentation and immediately start asking questions, generating rules, and troubleshooting payloads that were leading to generic, computer-says-no errors.

No need to go trough RAG shenanigans and all of it within the free token allowance.