Fei-Fei Li’s World Labs unveils its world-generating AI model

World Labs, the AI model developer cofounded by AI pioneer Fei-Fei Li, has released its 3D-space generating model, “Marble.” At the Marble Labs website, creators can now input text prompts, images, or videos of pieces of a real-world environment. Marble uses them to create full 3D environments, which can include interior spaces or expansive exterior ones.

Marble can reconstruct, generate, and simulate 3D worlds—think of it as a type of “world model.” In an interview with Fast Company, Li describes world models as a “significant” evolution of the generative AI era. “The large world model is really a significant step towards unlocking AI’s capability,” a category she calls “spatial.” Spatial intelligence refers to a system’s ability to perceive, model, reason about, and take actions within physical or geometric space—similar to how humans or animals choose their actions based on their understanding of their surroundings. 

World Labs launched in September of 2024, when it began working on the Marble model. Two months ago it released a preview of the model to a group of creatives, who began buliding worlds and giving feedback.

This week, Li posted a sort of manifesto on Substack arguing that spatial intelligence is the next frontier in AI. For humans, she says, spatial intelligence of the physical world around us provides the scaffolding upon which we build our cognition. “Spatial intelligence will transform how we create and interact with real and virtual worlds—revolutionizing storytelling, creativity, robotics, scientific discovery, and beyond,” she writes. World Labs believes that endowing machines (including robots) with such “spatial intelligence” could be transformative for a number of industries in the coming years. 

Using a web interface, users can feed Marble a scene description, images or videos, or coarse 3D layouts and the model will generate a realistic 3D environment. A user might input a set of images from the bedroom where they grew up, then upload the images to Marble, which will then intelligently sew them together to create an immersive digital 3D version of the room. 

The user can then use a set of tools to refine or expand their bedroom recreation, making small touchups like adding a clock. Or, they might make larger changes: adding a desk and chair or rendering the whole room with a different kind of light. More advanced users can create (or import) a rough 3D scene including the major fixtures of an environment, then use text prompts to control the overall style. 

The editing tools “let you iterate with the model and go back and forth and edit what the world looks like in various ways to help you [get] that vision out of your head and making that perfect world,” says World Labs cofounder Justin Johnson. World Labs is also hosting a “hub” where people can share their 3D creations.

Marble can output 3D worlds so that other creators, perhaps using other tools, can build on or enhance them. It can generate worlds as Gaussian splats, meshes, or videos—formats familiar to graphics pros. “That’s really cool because it lets you take those 3D assets and then compose them with all kinds of other traditional workflows,” Johnson says. “You could take your triangle mesh and drop it into a game. You could take your gaussian splat and then use it for a VFX shot and composite and other things.”

In generative AI, a Gaussian splat is the highest quality way of rendering 3D objects and spaces. The model generates millions or billions of tiny “splats”—semi-transparent particles occupying different points within a 3D space. They are small, smooth blobs whose brightness, opacity, color, or density is greatest at their center, with those values falling smoothly off in a bell-curve shape down to zero at their edges. The blobs then interconnect with their neighbors, which increases the smooth, consistent feel. When billions of these splats overlap, they can approximate the smooth surfaces, colors, and lighting of a 3D scene.

While anyone can now experiment with Marble, professionals such as artists, engineers, and VFX designers might find it useful in their work. Li and her cofounders, Ben Mildenhall, Johnson, and Christoph Lassner, say that this “spatial intelligence” could transform a variety of industries, including gaming, film production, and robotics.

Li, who also codirects the Stanford Institute for Human-Centered AI, was recently awarded the Queen Elizabeth Prize for Engineering at a ceremony with King Charles in London. Her cofounders have impressive bona fides, too. Lassner developed Pulsar, a sphere-based renderer that paved the way for 3D Gaussian Splatting. Johnson, who worked with Li as a graduate student at Stanford, created real-time style transfer (in which the visual style of one image is applied to another), which was deployed by Meta, Snap, and Prisma. Ben Mildenhall cocreated the neural radiance field (NeRF) method, which revolutionized 3D scene reconstruction.   

World Labs is offering a tiered subscription plan, starting with a free tier that includes enough credits to generate four worlds. The higher tiers add more credits and more tools, with the top plan priced at $95 per month.

View the full article