Project Genie: Experimenting with infinite, interactive worlds

Like, "Your Brain Hallucinates Your Conscious Reality" as exposed by Anil Seth[1]? Found that one while searching for something like "the illusion of the self" a few years ago.

It’s also easy to find this treated in various philosophy/religion through time and space. And anyway as consciousness is eager to project whatever looks like a possible fit, elements of suggesting prior arts can be inferred back as far as traces can be found.

[1] https://www.youtube.com/watch?v=lyu7v7nWzfo

I think this is pretty well established as far as neurologists are concerned and explains a lot of things. Like dreaming for instance.. just something like the model running without sensory input constraining it.

A kurzgesagt on this: Why Your Brain Blinds You For 2 Hours Every Day https://youtu.be/wo_e0EvEZn8 and the sources for that video - https://sites.google.com/view/sources-reality-is-not-real/

Another analogy that kinda fits in with what you're saying is the post-processing on smartphone "photos."

At what point does the processing become so strong that it's less a photograph and more a work of computational impressionism?

Also check out The Case Against Reality by Donald Hoffman

Agreed; everyone complained that LLMs have no world model, so here we go. Next logical step is to backfill the weights with encoded video from the real world at some reasonable frame rate to ground the imagination and then branch the inference on possible interventions (actions) in the near future of the simulation, throw the results into a goal evaluator and then send the winning action-predictions to motors. Getting timing right will probably require a bit more work than literally gluing them together, but probably not much more.

Soft disagree; if you wanted imagination you don't need to make a video model. You probably don't need to decode the latents at all. That seems pretty far from information-theoretic optimality, the kind that you want in a good+fast AI model making decisions.

The whole reason for LLMs inferencing human-processable text, and "world models" inferencing human-interactive video, is precisely so that humans can connect in and debug the thing.

I think the purpose of Genie is to be a video game, but it's a video game for AI researchers developing AIs.

I do agree that the entertainment implications are kind of the research exhaust of the end goal.

Yeah and the goal of Instagram was to share quirky pictures you took with your friends. Now it’s a platform for influencers and brainrot; arguably it has done more damage than drugs to younger generations.

As soon as this thing is hooked up to VR and reaches a tipping point with the general public we all know exactly what is going to happen. The creation of the most profitable, addictive and ultimately dystopian technology Big Tech has ever come up with.

Environment mapping to AI generated alternative outcomes is the holodeck.

I prefer real danger as living in the simulation is derivative.

I think this is the key component of developing subjective experience.

Still cool though…

This is a video model, not a world model. Start learning on this, and cascading errors will inevitably creep into all downstream products.

You cannot invent data.

Whoa, whoa, whoa. That's just one angle. Please don't bin that as the only use case for "world models"!

First of all, there are a variety of different types of world models. Simulation, video, static asset, etc. It's a loaded term, just as the use cases are widespread.

There are world models you can play in your browser inferred entirely by your CPU:

https://madebyoll.in/posts/game_emulation_via_dnn/ (my favorite, from 2022!)

https://madebyoll.in/posts/world_emulation_via_dnn/ (updated, in 3D)

There are static asset generating world models, like WorldLabs' Marble. These are useful for video games, previz, and filmmaking.

https://marble.worldlabs.ai/

I wrote open source software to leverage marble for filmmaking (I'm a filmmaker, and this tech is extremely useful for scene consistency):

https://www.youtube.com/watch?v=wJCJYdGdpHg

https://github.com/storytold/artcraft

There are playable video-oriented models, many of which are open source and will run on your 3080 and above:

https://diamond-wm.github.io/

https://github.com/Robbyant/lingbot-world

There are things termed "world models" that really shouldn't be:

https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0

There are robotics training oriented world models:

https://github.com/leggedrobotics/robotic_world_model

Genie is not strictly robotics-oriented.

That’s part of it but if you could actually pull out 3D models from these worlds, it would massively speed up game development.

Like LLMs, though: Do you really think a simulation will get them to all the corner cases robots/AI needs to know about, or will it be largely the same problem -- they'll be just good enough to fool the engineers and make the business ops drool and they'll be put into production and suddenly we'll see in a year or two stories about robots crushing peoples hands, stepping in drains and falling over or falling off roofs cause of some bizarre miscommunication between training and reality.

So, like, it's very important to understand the lineage of training and not just the "this is it"

I was lucky enough to be an early tester, here's a brief video walking through the process of creating worlds, showing examples--walking on the moon, with Nasa photo as part of the prompt, being in 221B Baker street with Holmes and Watson, wandering through a night market in Taipei as a giant boba milk tea (note how the stalls are different, and sell different foods), and also exploring the setting of my award-nominated tabletop RPG.

https://www.youtube.com/watch?v=FyTHcmWPuJE

It's an experimental research prototype, but it also feels like a hint of the future. Feel free to ask any questions.

I liked that first one and I hope someone creates one of going back to dinosaur age, i want to see that.

Any thoughts about Project Genie?

What about Fei Fei Li's lab? I think they are generating true 3D worlds rather than frames of a video?

Although that probably precludes her from having animations in those worlds...

Still amazed it took ML people so long to realize they needed and explicit representation to cache stuff.

And what if I go somewhere then go back there a week later?

I love AI but I also hope it will paradoxically make people realize the value of real life experiences and human relationships.

After a lifetime career in tech, I want to turn it all off.

Ironically, this brings us one step closer to believing the simulation hypothesis might be true... In which case, maybe there is no real world anyway ;)

Huh, yeah, the sky is blue outside and the sun is shining.

Although, I am feeling a bit lazy so let me see if I can simulate a walk.

I think this would be good for the people living in overcrowded, polluted and dirty cities in the world (let's be honest, they actually exist regardless of how that happened or why).

Maybe they can unplug from 500+ AQI pollution and spend time with their loved ones and friends in a simulated clean world?

Imagine working for 10-12 hours a day, and you come home to a pod (and a building could house thousands of pods, paid for by the company) where you plug in and relax for a few hours. Maybe a few more decades of breakthroughs can lead to simulated sleep as well so they get a full rest.

Wake up, head to the factory to make whatever the developed world needs.

(holy fuck that is a horrible existence but you know some people would LOVE for that to be real)

The more I see of this, the faster I want it to go!

I'm developing filmmaking tools with World Labs' Marble world model:

https://www.youtube.com/watch?v=wJCJYdGdpHg

https://github.com/storytold/artcraft

I think we'll eventually get to the point where these are real time and have consistent representations. I've been excited about world models since I saw the in-the-browser Pokemon demo:

https://madebyoll.in/posts/game_emulation_via_dnn/demo/

At some point, we'll have the creative Holodeck. If you've seen what single improv performers can do with AI, it's ridiculously cool. I can imagine watching entertainers in the future that summon and create entire worlds before us:

https://www.youtube.com/watch?v=MYH3FIFH55s

(If you haven't seen CodeMiko, she's an incredibly talented engineer and streamer. She develops mocap + AI streams.)

Most people don't have access to anything particularly nice in the real world.

It's reality privilege. Most of humanity will yearn for the worlds that AI will cook up for them, customized to their whims.

LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generation behind, in part due to the split efforts and lack of enthusiastic participation by all the Meta AI team. If you look at the chaos and churn at the highest levels across the industry, there's not a lot of room for mission creep by leadership, and LeCun thoroughly demonstrated he wasn't suited for the mission desired by Meta.

I think he's lucky he got out with his reputation relatively intact.

Isn't it more like this: JEPA looks at the video, "a dog walks out of the door, the mailman comes, dog is happy" and the next frame would need to look like "mailman must move to mailbox, dog will run happily towards him", which then an image/video generator would need to render.

Genie looks at the video, "when this group of pixels looks like this and the user presses 'jump', I will render the group different in this way in the next frame."

Genie is an artist drawing a flipbook. To tell you what happens next, it must draw the page. If it doesn't draw it, the story doesn't exist.

JEPA is a novelist writing a summary. To tell you what happens next, it just writes "The car crashes." It doesn't need to describe what the twisted metal looks like to know the crash happened.

You are beyond correct. World models is what saves their Reality Labs investment. I would say if Reality Labs cannot productize World Models, then that entire project needs to be scrapped.

Failures are not publicly reported, in general. Do you we know what they have invested in?

Most people don't like putting on VR headsets, no matter what the content is. It just never broke out of the tech enthusiast niche.

Yup, similar concepts! Just at two opposite extremes of the compute/scaling spectrum.

- That forest trail world is ~5 million parameters, trained on 15 minutes of video, scoped to run on a five-year-old iPhone through a twenty-year old API (WebGL GPGPU, i.e OpenGL fragment shaders). It's the smallest '3D' world model I'm aware of.

- Genie 3 is (most likely) ~100 billion parameters trained on millions of hours of video and running across multiple TPUs. I would be shocked if it's not the largest-scale world model available to the public.

There are lots of neat intermediate-scale world models being developed as well (e.g. LingBot-World https://github.com/robbyant/lingbot-world, Waypoint 1 https://huggingface.co/blog/waypoint-1) so I expect we'll be able to play something of Genie quality locally on gaming GPUs within a year or two.

This was a common argument against LLMs, that the space of possible next tokens is so vast that eventually a long enough sequence will necessarily decay into nonsense, or at least that compounding error will have the same effect.

Problem is, that's not what we've observed to happen as these models get better. In reality there is some metaphysical coarse-grained substrate of physics/semantics/whatever[1] which these models can apparently construct for themselves in pursuit of ~whatever~ goal they're after.

The initially stated position, and your position: "trying to hallucinate an entire world is a dead-end", is a sort of maximally-pessimistic 'the universe is maximally-irreducible' claim.

The truth is much much more complicated.

[1] https://www.arxiv.org/abs/2512.03750

Imo they explain pretty well what they are trying to achieve with SIMA and Genie in the Google Deepmind Podcast[1]. They see it as the way to get to AGI by letting AI agents learn for themselves in simulated worlds. Kind of like how they let AlphaGo train for Go in an enormous amount of simulated games.

[1] https://youtu.be/n5x6yXDj0uo

Take the positive spin. What if you could put in all the inputs and it can simulate real world scenarios you can walk through to benefit mankind e.g disaster scenarios, events, plane crashes, traffic patterns. I mean there's a lot of useful applications for it. I don't like the framing at this time, but I also get where it's going. The engineer in me is drawn to it, but the Muslim in me is very scared to hear anyone talk about creating worlds.... But again I have to separate my view from the reality that this could have very positive real world benefits when you can simulate scenarios. So I could put in a 2 pager or 10 page scenario that gets played out or simulated and allow me to walk through it. Not just predictive stuff but let's say things that have happened so I can map crime scenes or anything. In the end this performance art is because they are a product company being Benchmarked by wall street and they'll need customers for the technology but at the same time they probably already have uses for it internally.

An hybrid approach could maybe work, have a more or less standard game engine for coherence and use this kind of generative AI more or less as a short term rendering and physics sim engine.

As a kid in the early 1980s, I spent a lot of time experimenting with computers by playing basic games and drawing with crude applications. And it was fun. I would have loved to have something like Google's Genie to play with. Even if it never evolved, the product in the demos looks good enough for people to get value from.

It's been very profitable for drug dealers for centuries, who wouldn't want a piece of that market?

  > Why are they not training models to help write games instead?

Genie isn't about making games... Granted, they for some reason they don't put this at the top. Classic Google, not communicating well...

  | It simulates physics and interactions for dynamic worlds, while its breakthrough consistency enables the simulation of any real-world scenario — from robotics and modelling animation and fiction, to exploring locations and historical settings.

The key part is simulation. That's what they are building this for. Ignore everything else.

Same with Nvidia's Earth 2 and Cosmos (and a bit like Isaac). Games or VR environments are not the primary drive, the primary drive is training robots (including non-humanoids, such as Waymo) and just getting the data. It's exactly because of this that perfect physics (or let's be honest, realistic physics[0,1]). Getting 50% of the way there in simulation really does cut down the costs of development, even if we recognize that cost steepens as we approach "there". I really wish they didn't call them "world models" or more specifically didn't shove the word "physics" in there, but hey, is it really marketing if they don't claim a golden goose can not only lay actual gold eggs but also diamonds and that its honks cure cancer?

[0] Looking right does not mean it is right. Maybe it'll match your intuition or undergrad general physics classes with calculus but talk to a real physicist if you doubt me here. Even one with just an undergrad will tell you this physics is unrealistic and any one worth their salt will tell you how unintuitive physics ends up being as you get realistic, even well before approaching quantum. Go talk to the HPC folks and ask them why they need superocmputers... Sorry, physics can't be done from observation alone.

[1] Seriously, I mean look at their demo page. It really is impressive, don't get me wrong, but I can't find a single video that doesn't have major physics problems. That "A high-altitude open world featuring deformable snow terrain." looks like it is simulating Legolas[2], not a real person. The work is impressive, but it isn't anywhere near realistic https://deepmind.google/models/genie/

[2] https://www.youtube.com/watch?v=O4ZYzbKaVyQ

Why is it a dead end, you don’t meaningfully explain that. These models look like you can interact with them and they seem to replicate physics models.

Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by watching boats, and usefully abstract that into a gamified form, is another thing entirely.

The bottleneck for games of any size is always whether they are good. There are plenty of small indies which do not put out good games. I don't see world models improving game design or fun factors.

If I am wrong, then the huge supply of fun games will completely saturate demand and be no easier for indie game devs to stand out.

It's very impressive tech but subject to the same limitations as other generative AI: Inconsistency, inaccurate physics, limited time, lag, massively expensive computation.

You COULD create a sailing sim but after ten minutes you might be walking on water, or in the bath, and it would use more power than a small ferry.

There's no way this tech can run on a PS5 or anything close to it.

> and might be the renaissance for small to one person game studios.

Indie games are already bigger than ever as far as I know.

> If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game studios.

I mean, if making a game eventually boils down to cooking a sufficient prompt (which to be clear, I'm not talking about text, these prompts are probably going to be more like video databases) then I'm not sure if it will be a renaissance for "one person game studios" any more than AI image generation has been a renaissance for "one person artists".

I want to be optimistic but it's hard to deny the massive distribution stranglehold that media publishing landscape has, and that has nothing to do with technology.

Honestly getting a Sunfish is probably cheaper than the a VR headset if you want to "train sailing"

...and then, the pneumatics in your living room.

This reminds me of the comments by programmers roughly two years ago:

"Sure it can write a single function but the code is terrible when it tries to write a whole class..."

It's super cool but I see it as a much more flexible open ended take on the idea of procedurally generated worlds where hard-coded deterministic math and rendering parameters are replaced by prompt-able models.

The deadness you're talking about is there in procedural worlds too, and it stems from the fact that there's not actually much "there." Think of it as a kind of illusion or a magic trick with math. It replicates some of the macro structure of the world but the true information content is low.

Search YouTube for procedural landscape examples. Some of them are actually a lot more visually impressive than this, but without the interactivity. It's a popular topic in the demo scene too where people have made tiny demos (e.g. under 1k in size) that generate impressive scenes.

I expect to see generative AI techniques like this show up in games, though it might take a bit due to their high computational cost compared to traditional procedural generation.

Could you clarify what you mean by checkpointing here? Are world models being trained during inference? Apologies if this is basic - I'm new to world models.

I feel like people are already currently doing this. Essentially storyboarding first.

This guy a month ago for example: https://youtu.be/SGJC4Hnz3m0

I've been experimenting with that from a slightly different angle: teaching Claude how to play and referee a pencil-and-paper RPG that I developed over about 20 years starting in the mid 1970s. Claude can't quite do it yet for reasons related to permanence and learning over time, but it can do surprisingly well up until it runs into those problems, and it's possible to help it past some obstacles.

The game is called "Explorers' Guild", or "xg" for short. It's easier for Claude to act as a player than a director (xg's version of a dungeon master or game master), again mainly because of permance and learning issues, but to the extent that I can help it past those issues it's also fairly good at acting as a director. It does require some pretty specific stuff in the system prompt to, for example, avoid confabulating stuff that doesn't fit the world or the scenario.

But to really build a version of xg on Claude it needs better ways to remember and improve what it has learned about playing the game, and what it has learned about a specific group of players in a specific scenario as it develops over time.

Looks good to me: https://youtu.be/15KtGNgpVnE?t=648&si=urWJGEFWuN5veh43

Ironically the physics are kind of my biggest criticism. They call these "world models", but I think it's more accurate to call them "video game models" because they employ "video game physics" rather than real world physics, among other things

This is most evident in the way things collide.

>an LLM that can generate textures to be fed into a human-coded 3d engine

I'm not certain but I think the LLM is also generating the physics itself. It's generating rules based on its training data, e.g. watch a cat walk enough and you can simulate how the cat moves in the generated "world".

Exactly this, it would essentially be a STTNG Holodeck

The endgame has nothing to do with gaming.

The goal of world models like Genie is to be a way for AI and robots to "imagine" things. Then, they could practice tasks inside of the simulated world or reason about actions by simulating their outcome.

It means you should go the other way. Open world winning against smaller, handcrafted environments and stories was generally a mistake, and so is this.

It seems to be generating images in real time, not 3d scenes. It might still be useful for prototyping.

I would think that building a environment which can be managed by a game engine is the first pass. In a few years when we are able to render more than 60 seconds it could very well replace the game engine entirely by just rendering everything in realtime based on user interactions. The final phase is just prompts which turn directly into interactive games, maybe even multiplayer. When I see the progress we've made on things like DOOM, where it can infer the proper rendering of actions like firing weapons and even updating scores on hits and such it doesn't feel like we're very far off, a few years at most. For a game studio that could mean cutting out almost everything between keyboard and display, but for now just replacing the asset pipeline is huge.

I understand the ultimate end goal to be simulation of life. A near perfect replica of the real world we can use to simulate and test medicine, economy, and social impact.

Screensavers for robots?

Stadia was lightyears ahead, but pro-Microsoft media assassinated it with FUD

While "journalists" were busy bootlicking a laggy 720p Android only xCloud beta, Stadia was already delivering flawless 4K@60FPS in a web browser

They killed the only platform that actually worked just to protect Microsoft

This will be a textbook case study in how a legacy monopoly kills innovation to protect its own mediocrity

Microsoft won't survive the century, they are a dinosaur on borrowed time that has already lost the war in mobile, AI, and robotics

They don't create,, they just buy marrket share to suffocate the competition and ruin every product they touch

Even their cloud dominance is about to end, as they are already losing their grip on the European market to antitrust and sovereign alternatives

clearly

I don't know if that's the simplest explanation, considering how insanely complex the generation is (these world models might literally be the most complex things to ever be created).

But I do think it's a partial existence proof.

The mass-poverty and climate changed ravaged world parts, I could definitely see.

Decades?

I mean, yes, the probability of having that level of tech in decades is quite high.

But the technology is moving very fast right now. It sounds crazy, but I think that there is a 50% chance of having ready player one level technology before 2030.

It's absolutely possible it will take more time to become economical.