Marble: A Multimodal World Model

It's worlds better than just doing gaussian splats from images, but given how much the quality is influenced by images the limit to four images with text prompt or eight images without prompt is quite limiting. That's plenty to describe a chair, but almost nothing to describe a home or a space station. I hope they can extend those limits in future updates

Marble renders a static Gaussian Splat asset (like a 3D game engine asset) that you then render in a game engine.

Marble seems useful for lots of use cases - 3D design, online games, etc. You pay the GPU cost to render once, then you can reuse it.

Genie seems revolutionary but expensive af to render and deliver to end users. You never stop paying boatloads of H100 costs (probably several H100s or TPU equivalents per user session) per second.

You could make a VRChat type game with Marble.

You could make a VRChat game with Genie, but only the billionaires could afford to play it.

To be clear, Genie does some remarkably cool things. You can prompt it, "T-Rex tap dancing by" and it'll appear animated in the world. I don't think any other system can do this. But the cost is enormous and it's why we don't have a playable demo.

When the cost of GPU compute comes down, I'm sure we'll all be steaming a Google Stadia like experience of "games" rendered on the fly. Multiplayer, with Hollywood grade visuals. Like playing real time Lord of the Rings or something wild.

Interestingly, there is a model like Google Genie that is open source and available to run on your local Nvidia desktop GPU. It's called DiamondWM [1], and it's a world model trained on FPS gameplay footage. It generates a 10 fps 160x160 image you can play through. Maybe we'll develop better models and faster techniques and the dream of local world models can one day be realized.

[1] https://diamond-wm.github.io/

Along with entertainment, they can be used for simulation training for robots. And allow for imagining potential trajectories

All of these have fairly "exact" representations, and generation techniques are also often fairly "exact" in trying to create worlds that won't break physics engines(big part) or rendering engines, often hand-crafted algorithms but nothing really that really stopped neural networks from being used on a higher level.

One important detail in most generation systems in games is that they are often built to be controllable to work with game-logic (think how Minecraft generates the world to include biomes,villages,etc) or more or less artist controllable.

3d scanning has often relied on point-clouds, but were heavy, full of holes,etc and have been infeasible for direct rendering for long so many methods were developed to make decent polygon meshes.

Nerf's and Gaussian splatting(GS) started appearing a few years back, these are more "approximate" and totally ignore polygon generation instead relying on quantization of the world into NN-matrix-"fields"(NERF) or fuzzy-point-clouds (GS), visually these have been impressive since they managed to capture "real" images well.

This system is built on GS since that probably meshed fairly well with neural network token and diffusion techniques for encoding inputs (images, texts).

They do mention mesh exports (there has been some research into polygon generation from GS).

If the system scales to huge worlds this could change game-dev, and there seems to be some aim with the control methods, but it'd probably require more control and world/asset management since you need predictability with existing things to produce in the long term (same as with code agents).

Other "world model"s are Image + (keyboard input) to Video or Streaming Images, that effectively function like a game engine / video hybrid.

We don't need to agree on one very specific meaning, which is good, because we would fail.

  > I work in AI and, to this day, I don't know what they mean by “world” in “world model”.

But "a physics" can be a model that describes any world, not just the one we live in. For ML models, this should be based on the data they're processing. Ideally we'd want this to be similar to our own, but if it is modeling a "world" where pi=3, then that's still "a physics".

The key points here are that a physics is a counterfactual description of the environment. You have to have language to formalize relationships between objects. In standard physics (and most of science) we use math[0], though we use several languages (different algebras, different groups, different branches, etc) of math to describe different phenomena. But the point is that an equation is designed to be the maximum compression of that description. I don't really care if you use numbers or symbols, what matters is if you have counterfactual, testable, consistent, and concise descriptions of "the world".

Oddly enough, there are a lot of physicists and former physicists that work in ML but it is fairly uncommon for them to be working on "world modeling." I can tell you from my own experience talking to people who research world models that they respond to my concerns as "we just care if it works" as if that is also not my primary concern. Who the fuck isn't concerned with that? Philosophers?

[0] It can be easy to read "The Unreasonable Effectiveness of Mathematics in the Natural Sciences" out of context as we're >60 years past where math has been the lingua Franca of science. But math is a language, we invented it, and it should be surprising that this language is so powerful that we can work out the motion of the stars from a piece of paper. Math is really the closest thing we have to a magical language https://web.archive.org/web/20210212111540/http://www.dartmo...

Presumably de_dust2

From my understanding, admittedly quite a shallow look so far, the model generates gaussian splats then from that could implement the collider.

I guess from the splat and the colliders you could generate actual assets that could be interactable/animated/have physics etc. Unsure, exciting space though! I just don't know how I would properly use this in a game, the examples are all quite on-rails and seem to avoid interacting too much with stuff in the environment.

Edit: After seeing your edited (longer) comment, I have no idea what you’re talking about.

This is immensely useful tech for blocking out consistent scenes, eg. for video generation.

Beyond entertainment, this is going to be huge in the field of robotics.

I wish you could see what I see.

If you go in with the expectation that you give it a single image and it's doing gaussian splatting from a single image and a prompt it's phenomenal. If you deviate too far from the image viewpoint it breaks down, but it looks decent long enough to be very usable. But if you go in with the expectation that it's generating "worlds" it's not very good. This only passes as a world in a 20 second tech demo where the user isn't given camera controls

My best guess is that they are forced (by investors, lack of investors, fear of the AI bubble, or whatever) to release something, and this was something they could polish up to production quality and host with reasonable GPU resources

I’m willing to take that bet. Name any amount you’re willing to lose.

Before you agree: movies take more than 1 year to make and get published, and it takes more than 1 movie to make somebody an a-lister

finally! we should come up for a term for this new tech... maybe computer generated imagery?

Update - it is a paid feature