The Next Giant Leap For AI Is Called World Models

Imagine being able to create entire worlds with the help of artificial intelligence (AI). This is the promise of world models, a category of generative tools that can simulate full 3D environments, complete with realistic physics and responsive inhabitants. These immersive environments have the potential to revolutionize various industries, from engineering and architecture to robotics and medicine, by enabling the creation of highly realistic simulations that can help us better understand the world.

So, how do world models work? Currently, there are two main methods that AI models use to create virtual worlds. The first approach involves modeling everything dynamically, on the fly, as the user interacts with the world. This method is similar to how generative video models work, where each pixel is predicted based on the model’s understanding of physics and object behavior. However, world models take it a step further by responding to user input, such as moving the camera or interacting with objects, to generate a continuous and immersive experience. The second approach involves transforming a prompt into persistent geometric models, digital assets, and physics metadata, which can then be downloaded and imported into other software tools for manipulation and exploration.

Some of the biggest names in AI are currently developing their own world models. For example, Google’s Genie 3 platform can create worlds that remain persistent for several minutes, while Meta’s (Facebook) Habitat 3 platform is designed to create virtual environments for training embodied AI, such as physical robots. Other companies, like World Labs, are taking a different approach with their Marble world model, which creates persistent, downloadable 3D environments from text, image, or video prompts. Even Elon Musk’s xAI development group is working on a world model that will reportedly be used for both video games and training robots.

The potential use cases for world models are vast and varied. In healthcare, they can be used to create immersive digital twins of clinical environments to simulate patient interactions. In manufacturing, they can be used to create virtual training grounds for industrial robots and autonomous vehicles. Architects can use them to model and interact with buildings, testing how they react to physics, lighting, and airflow before construction begins. Additionally, world models can be used to simulate the human body environment and molecular reactions, which can help in the development of new medicines and treatments.

World models have the potential to be a key stepping stone on the road to artificial general intelligence (AGI), which refers to machines that can apply their knowledge and abilities to any task, regardless of whether they’ve been specifically trained to do it. By giving AI the capacity to understand how the world is built and how it holds together, world models can augment its abilities with language and vision. As such, world models are an exciting and vital area of AI development, and one that anyone interested in the future of AI should be following closely.