Summer is about creating memories. And for robots, that translates into generating lots of data. It’s also a time to catch up on reading. My beach material last June was the research paper on Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (aka UMI). Generalist, now a $2 billion unicorn robotics startup, was inspired by earlier research into this kind of data collection. Essentially, teams at Toyota Research Institute (TRI), Columbia University, and Stanford later popularized a unique data-collection platform that uses puppet-like end effectors operated by people, with GoPro cameras capturing real-world tasks, such as washing dishes and picking up objects. The resulting demonstrations become training data for robot foundation models, enabling collaborative robots to learn and take over these tasks much more quickly.
UMI’s paper demonstration image, illustrating human data capture to full autonomy on co-bots. Source: https://umi-gripper.github.io/
This past June at Automate, Generalist demonstrated live how its models could quickly create policies for different co-bot systems. On one side of the hall, the company demonstrated how it could make Universal Robots (UR) arms fold and build cardboard boxes, while across the McCormick Center, they used Flexiv arms to repair robot vacuums. According to their X post boasting about the display at the show: “What resonated most wasn’t just that the models could do the tasks — it was that they had the intelligence to recover in real time when things went wrong, and this got people thinking differently about automation.”
Then, almost a month later, the AI unicorn posted a blog missive titled “Towards Machines with a Thousand Hands.” To better understand the meaning of Generalist’s newest accomplishment of using different arms and grippers, including spatulas, across a plethora of use cases, I interviewed Samantha Castellanos, their Founding Mechanical Engineer. I wanted to know the process beyond research, especially how it translates into industrial implementations to accelerate adoption and expand usage into new applications.
As a starting point, Castellanos shared with me their overall philosophy: “Our goal, our main goal, is to make the best model in the world. And at the end of the day, I think it’s our model that’s going to be the moat that is going to differentiate us from others.” The mechanical engineer was referring to the already crowded space that has cumulatively raised more than $4 billion, including standouts like Skild AI ($2B), Physical Intelligence ($1B), Field AI ($300M), and RLWRLD ($41M).
She continued to articulate their model’s attributes: “Everything we do is to make the model better. All of the data collection, all the types of tool collection, different types of interacting with the world. Like, I said before, that is all incredibly important. In addition to all the efficiency improvements that we’ve been making, we’re just trying to build that real general intelligence by giving the model everything it needs to know about how the world works, and that I think is what’s going to set us apart.”
Tactically, it’s the Generalist’s approach to data collection that is their most distinctive feature. “From a hardware perspective, our gripper is just a one-degree-of-freedom gripper, and I think that’s very different from what some other folks have started with. And I think there is an elegance to the simplicity of design. A lot of the time, simple things are often the most robust, and that’s definitely one of our core values from a hardware perspective. My focus as a hardware engineer is on simplicity and robustness, because you can’t have a line going down every five minutes for a gripper. You need to be able to switch it out quickly. You need to get the line running again. If it takes two minutes to swap out a finger, that’s great. Your line is only down for two minutes, which is something I’ve really liked about this company.”
It’s amazing how quickly Generalist is going from collection to working robot arm. According to Castellanos, “All of the data that we’ve done for this effort [a thousand hands demonstration] has been data that was collected in our own office in Boston, and some also in our California office. So for a number of hours, there is a spread. It’s all rather small. I mean, I wrote down all of my numbers. We have about 80 hours at the most. The fewest hours were two. Like the peeler, there’s a little brief moment of a peeler in the video, and that was only two hours of data collection.”
The process then went from human collection to robot policies and further refinement by the machines themselves. As she elaborates, “The human data collection between two and 80 hours is what we took for the various tasks, and then we also took robot data. So this would be robot data collection: running the model trained on human-only data on a robot, then collecting more data. Usually, this is in minutes, no more than like 80 minutes, but also as little as four. Like for tape hand, it was a very quick task. It was only about a second. So for robot data collection, I only collected about 50 perfect episodes of tape data collection on the robot, and that was only four minutes. And that got us a pretty substantial model to be able to do the tape dispenser up to 10 times in a row.”
The engineer illustrated the process further, “So I think the biggest surprise for this effort was that initially we didn’t know how well this was going to work. We were assuming that, yes, our model is like a physics model where the end effector doesn’t matter; that we could be gripper-agnostic. So this was our first real effort to prove that. So, the first tool we worked on was the screwdriver hand, and initially it wasn’t working, but it turns out I just didn’t train it long enough on my first attempt. So, I just trained it for more steps and ended up with a screwdriver hand that would work really well.”
While these models performed well in the lab, my big question was what happens when Generalist deploys in industry. Castellanos shared, “We have a lot of customers coming to us. A lot of different kinds of customers coming to us. Ones who want an all-in-one-based solution. We give them everything. They run our model. We give them the hardware. They don’t need to do anything themselves. And then there are other customers who are more robot-savvy and who already have a UR [arm], and this robot system can use your model to run on it. Nothing is off the table regarding how we’re going to integrate this. Also, the model is improving on a weekly and almost daily basis, so much that we’re still figuring out which business model makes the most sense. But everything is on the table.”
One of the biggest distinctions she sees is how quickly the model recalibrates when something goes askew. “So it learns about recovering when things fall. It learns to hand an object from one hand to the other. All that stuff is already baked in. So there is a certain amount of recovery data already in place. Even if it wasn’t taken in the specific task data, like the scrape-and-sweep task, which was at the end of our 1000 hands video, that one is kind of a chaotic model. I find that interesting because it is using recovery behaviors that aren’t in its specific task dataset, like using the brush with the other hand; that’s an ambidextrous behavior that just came out of nowhere, ” Castellanos remarked.
“The recoverability is already in there, and I think that’s what makes our model unique because it learned how to recover from other interactions from our ginormous dataset. As for deployments, I think this is definitely helpful for recovering in real time when issues happen on the line, like when people or the robot drops something, when things don’t go quite perfectly; there’s definitely a chance. And the better our models get, the more likely recovery will already be possible,” declared the engineer.
Looking ahead, as Generalist deploys more of its models in real-world settings, I questioned if humanoids would be next. Castellanos replied, “We haven’t run on a humanoid yet, but we have our own computer that’s running all of our models. I still think it would be feasible for a mobile robot. We would just need to be able to put that computer on the mobile platform. So there is some integration work that does need to be done. And if it’s a humanoid robot that already has a lot of compute on board, I would be excited to run it on the humanoid.”
It should be noted that the UMI lineage is already extending into general-purpose, humanoid-like robots. UMI co-authors Russ Tedrake and Ben Burchfiel, both formerly with TRI, went on to co-found Walden Robotics, which is applying large behavior models to mobile manipulators in real-world industrial environments.
Walden Robotics’ humanoid mobile manipulator demonstrating gripper assembly tasks. Source: Walden Robotics
In closing, the startup engineer commented, “I’m just here for the ride to see what happens, because sometimes it’s very hard to predict how well these models are getting. I would love to see it do more of that on its own without prompting. Being able to get to the point where our models need a couple of minutes of data and can just do the task, and recover from weird edge cases. I’m really excited to see that come to fruition, and I think it might be sooner than we think.”