‘LegoGPT’ designs Lego models with nothing but a prompt

Researchers at Carnegie Mellon University (CMU) have just answered a question that’s probably occurred to Lego fans for decades: What if I could instantly turn any idea into a Lego set?

In a paper titled “Generating Physically Stable and Buildable LEGO Designs from Text,” published last week, six coauthors lay out an invention they’re calling “LegoGPT.” This generative AI model can take a text-based prompt, like “an acoustic guitar with an hourglass shape,” and determine all of the necessary Lego pieces needed to build that structure and how to assemble them. 

The LegoGPT demo and code is publicly available through the study, meaning that Lego hobbyists are free to try it out at home. Although outputs are currently limited to around 20 categories (including basic items like chairs, guitars, boats, trains, and cars), the researchers are working to expand the model’s capabilities into more complicated categories.

Ultimately, they think a LegoGPT-type tool might serve as the basis for a variety of real-world tasks in architecture and product design.

How LegoGPT predicts its next block

LegoGPT is a fine-tuned version of Meta’s LLaMA-3.2-Instruct-1B language learning model, which you can think of as an open source ChatGPT. To teach the model how to make Lego structures, researchers trained it using a database of 47,000 Lego structures and 28,000 unique 3D shapes, each with their own descriptive captions. Based on that vast swath of designs, LegoGPT is able to predict how to build a hypothetical object using only a text prompt.

To do that, LegoGPT uses something called an autoregressive model, which is common among the most popular generative AI platforms.

“ChatGPT and Llama are autoregressive models because, given the string of words that they’ve already outputted, they want to predict the next word,” explains Ava Pun, one of the study’s coauthors and a PhD student at CMU. “So if you ask, ‘What is the weather,’ and it predicts ‘The weather today is,’ then it will try to predict the next word: sunny, rainy, and so on. With Lego GPT, instead of predicting the next word, it wants to predict the next brick.”

Once LegoGPT has created a 3D model it thinks will work, the LLM needs a way to make sure that the structure will actually be stable. According to Pun, that proved tricky, considering that existing simulators aren’t trained to understand the physics of a Lego brick. So, the CMU team built their own physics algorithm for LegoGPT to check its work.

“We developed a customized physics reasoning algorithm that accounts for all the physical forces that the bricks experience: for example, the downward force due to gravity, friction forces, and contact forces from the other bricks that they’re touching,” Pun says. “The algorithm constructs a force model for the structure and then evaluates the forces over the entire structure. If these physical forces sum to zero, that means the structure will not move around.” 

LegoGPT automatically uses this algorithm to ensure that it’s found a viable solution. If any of the block it’s chosen is causing the model to turn out wobbly, the model will continue iterating until it lands on a new version that passes the test.

A future real-world application

So far, researchers have used LegoGPT to create a range of structures, including vintage cars, steamships, and an electric guitar. Currently, the model only works on a 20x20x20 voxel grid, though Pun says the team is already planning on adding more brick types to the model’s database and expanding the grid resolution.

For Lego fans who want to play around at home, the study’s demo, available through a public portal, can turn simple prompts into a buildable 3D Lego model and a list of necessary parts. Because LegoGPT isn’t made to be Lego-builder-facing, it doesn’t produce step-by-step instructions, meaning the main challenge will be figuring out how to arrange the component parts in the right order. 

Pun says her team used Lego brick assembly to test AI’s 3D-building capabilities because of the blocks’ accessibility. Eventually, though, they believe this concept could be applied to real-world scenarios, like helping architects draft buildings or designing custom furniture from a predefined set of parts.

“Today’s generative AIs can’t offer that—you can generate a cool image or video of a chair, but the model doesn’t know how these things can be made in the real world,” Pun says. “We wanted to address this challenge by integrating physical laws and assembly constraints into generative models and creating objects that function in reality.”

View the full article