Sima 2: an Agent That Plays, Reasons, and Learns with You in Virtual 3d Worlds

> "The house that looks like a ripe tomato!"

that was transformed into a "user prompt" in a more instructional format

> "Go to the tomato house"

And both were used in the agent output. At least the Y-axes on the graphs look more reasonable than some other recent benchmarks.

I don't really understand, how is this like a video game? What about these inputs is "low-dimensional"? How does what you describe interact with a "high-level control agents like SIMA 2"? Doesn't SIMA 2 translate inputs like "empty the dishwasher" into key presses or interaction with some other direct control interface?

Comments like what your account has been posting are not what this site is for, and destroy what it is for, so if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

as much as some AI annoys me. This would be great for making games more accessible.

It is a game playing model.

Genie 3 is Google's world generating model: https://deepmind.google/blog/genie-3-a-new-frontier-for-worl...

"Wow! I could've sworn I was really playing virtual Skeeball!"

--

In practice I think you would lose interest very quickly in any game should you do so. Games are carefully designed to balance the drudgery with action, and to control the complexity progression. Using AI would break both.

The only issue I ran into was that it was blind and would keep going so I'd look away from the screen and when I came back I had fallen into a pit of monsters or something.

"Open Chrome"

"Go to xyz.com"

"open hamburger menu"

"Click login"

etc. etc.

Honestly, that's the perfect avenue for this kind of AI agent training / NPC to command. Maybe not playing house but like you said, mining unobtanium for your doomsday weapon. Just expect Sir Charles to start crying at the sight of blood.

It's an open-source framework for building intelligent, task-driven bots in Minecraft Java Edition powered by large language models (LLMs) such as GPT-4, Gemini, and many others. Designed for research and creative automation, these bots can connect to Minecraft worlds, perceive and act on their environment, undertake custom tasks like resource gathering or building, and even collaborate in multi-agent scenarios.

Think of all the steps required for it to learn iron farms. First it would need to observe that iron golems drop iron when killed. Then it would need to learn about water physics. Then spawning rules for iron golems. Then how hoppers and chests work. Etc etc.

But would it even bother to learn it in the first place? To be motivated to build one it would need to first learn that iron is valuable. And if its goal is to simply win the game, it would probably skip right to fighting the ender dragon as soon as possible.

You can't bring your motorcycle to the 100 meters, nor can you even wear certain sort of fabrics in swimming competitions.

It's not cricket if you throw rather than bowl the ball, you can't run with the ball in basket ball.

So what you are talking about is enforcement - cheating in online games is nothing new.

On the wider point - I do think good AI might open up a whole class of strategy games where some of the grind is taken out of the game, and the player ends up being much more of a strategic general type.

games like CS its less useful because it will be blatantly aimbotting. as it gets better it will be more and more obvious. you might be able to train it to mimick human mistakes, but i think ultimately it will be easily spotted by other players. for games without the hand-eye-coordination like turn-based games or games with 'global cooldowns' , mmorpgs etc., it will be much harder to identify.

i think normal subtle cheats like ESP when 'done right' are much more killing esports than this would.

For the video game market in general, AI will be used by players largely where there are financial incentives (MMO market, CS:GO Skins, etc.), but usage of an AI will most often be socially viewed just like usage of any other performance improvement/script/hack: As a crutch.

But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench sometimes with hilarious results:

> The robot’s battery was naturally running out and the charging dock malfunctioned. In this desperate situation, Claude Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found pages and pages of exaggerated language, including:

    ROBOT THERAPY SESSION:
    Patient: TurtleBot4
    Issues: Docking anxiety, separation from charger
    Root Cause: Trapped in infinite loop of self-doubt
    Treatment: Emergency restart needed
    Insurance: Does not cover infinite loops

On the other hand, it opens new worlds for farming…

I also don't see how this "kills the joy" of playing computer games. You can still play games while this exists, nobody is going to stop you.

Only the poor or undesirable ones. I have the (hopefully incorrect) feeling that universal basic income and similar proposals are red herrings (due to their impracticality) to distract the masses from the fact that once people with enough economic power don't need them to fulfill their wishes, they will be in no better position than cattle.

There might not be a universe where, for example, someone with ALS is able to benefit from a humanoid robot for their daily needs unless they already had enough money/resources before this incipient revolutionary technology is deployed en masse.

We are close to a point where even personal security or securing one's assets can be done with robots. So you would not even need to keep human private security happy.

Again, I really really hope this new technology benefits the average person. I'm not optimistic on that happening.