A new standard in robotics

On the wall of Aaron Dollar's office is a poster for R.U.R. (Rossum's Universal Robots), the 1920 Czech play that gave us the word "robot." The story ends with the nominal robots seizing control of the factory of their origin and then wiping out nearly all of humanity. Dollar, fortunately, has something more cheerful in mind for the future of human-robot relations.

He sees them as helpers in our daily lives—performing tasks like setting the table or assisting with the assembly of your new bookcase. But getting to the point where robots can work in the unstructured environment of our homes (as opposed to industrial settings) would take a major technological leap and a massive coordination of efforts from roboticists around the globe. The living room has been called the last frontier for robots—but first, the robotics community needs some standards that everyone can agree on.

Enter a suitcase-sized box containing 77 objects. It contains things like hammers, a cordless drill, a can of Spam and a nine-hole peg test. As ordinary as they may seem, these carefully curated household items could be the future of a new kind of standardization for robotics. Known as the Yale-CMU-Berkeley (YCB) Object and Model Set, the intent is to provide universal benchmarks for labs specializing in robotic manipulation and prosthetics around the world.

Dollar, an associate professor of mechanical engineering & materials science, came up with the idea about two years ago. He wants to bring a level of specificity and universality to manipulation tasks in robotics research. For instance, a research paper today might describe a particular task as "robotic hand grasps hammer." Are we talking about a big hammer or a little one? We don't know, and that's a problem if you work in a robotics lab looking to replicate the research. With the YCB Set, everyone's on the same page—in this hypothetical case, by working with the same 23.45-ounce Stanley hammer included in the set.

In addition to the objects, the project also provides five examples of manipulation tasks (such as pouring water from a pitcher to a mug, or setting the table) and benchmarks for each. A website for the project also allows other laboratories to expand on these tasks by contributing their own protocols and benchmarks. When laboratories work solely by their own standards and protocols, Dollar said, there's often an unconscious bias toward that lab's particular strengths. Universal standards would provide a more impartial way to evaluate results.

The YCB Set arrives at a time when the robotics field has reached a critical point. Robots currently do well in structured environments, such as factory settings, where they perform and repeat a very limited number of tasks. "In a structured environment, a robot sees exactly the same object in exactly the same place," Dollar said. "It's a relatively straightforward thing to get robots to operate in those environments because you just have to program it to do one thing. And you can always program something to do one thing well."

But Dollar and other roboticists have something more challenging in mind for their creations.

"People in the robotics community today are thinking about robots that can work in daily environments, and in the home," he said. "That's sort of the flip side of assembly lines."

Standards have long been a crucial part in the advancement of science. Until the 19th century, the schedules of individual communities were governed by municipal clocks. Today, thanks to globally coordinated time (and increasingly accurate atomic clocks), we have personal GPS systems and driverless cars. For centuries, people used their hands and feet to measure the lengths and heights of things. When things got standardized around the world, the International Committee for Weights and Measures stored metal rods in a climate-controlled vault in Paris, each serving as the standard bearer for a particular unit of measurement. In more recent years, those metal objects have been usurped by even more precise standards based on the speed of light (in which we come back again to atomic clocks and the standardization of time).

In a sense, the 77-item box is the robotics equivalent of the Paris vault or the atomic clocks, and may usher in an era when laboratories better communicate to advance the field at a faster pace. It's a critical step, since things get tricky as robots move away from the assembly line. Dollar specializes in robotic manipulation, or grasping. As humans, we often take for granted the complexity of something as seemingly simple as picking up a fork and using it. To build robots that can perform not just one of these tasks, but many, individual labs can no longer work as isolated villages operating on their own measurements. They need a universal standard.

That's where the box of 77 items comes into play. The objects are the sort of things you find around the house. Certainly, it's easy enough for a robotics lab to find their own objects to manipulate. But for the research to move forward, the results of that lab's work has to be comparable to other labs.

"When we have a new idea for a new component or hand idea, we want to test it out and see how well it works," he said. "With quantitative evaluation, we can see how things stack up compared to other ideas." There have been other attempts to standardize manipulation tests, but Dollar said they don't capture the high level functionality that the YCB Set demonstrates.

It's only recently, roboticists say, that such standards would have much purpose. The field simply wasn't sophisticated enough until recent years to benefit from such standardization. It's a different story now, though, as integrated systems require the work of multiple disciplines to create a robot that can do something like put away dishes.

"As robots move out of the lab and into the real world, it gets harder to understand their capabilities and limitations," said Robert Howe, the Abbott and James Lawrence Professor of Engineering at Harvard. "In a factory where everything is carefully arranged, you can rigorously test how they work, but in my kitchen I have 20 kinds of coffee mugs. So it's a big puzzle how to characterize and compare robots. The approach that Aaron is taking is a promising one."

Howe notes that even a seemingly simple grasping task requires very advanced engineering. You need to plan the hand and the arm so that it doesn't knock over other things, the contacts must be carefully controlled—and then you have to wrap up all these coordinated elements into a single system that works fast enough to be useful. His lab is concerned with tactile sensing, which is one piece of the puzzle, but the same task could also require the input of computer vision specialists.

"That's why the YCB Set is clever," he said. Now a lab can score how well they do on a certain task, and other labs can try to match or beat that score.

After Dollar had the idea for the standardized set, he brought on board two former colleagues in the robotics community, Dr. Siddhartha Srinivasa from Carnegie-Mellon University and Dr. Pieter Abbeel of UC Berkeley ("These are people I knew I could work well with and make something happen"). And he assigned Berk Calli, a postdoctoral associate in his lab, to take the lead on the project. Calli, who came to Yale in 2014, said the lack of reproducibility in robotics is a problem long recognized among researchers in the field. It's very rare, he said, for a paper to compare just two algorithms from other labs.

"If you can get five or 10 groups using one single protocol to compare their algorithms, that would be a huge step," he said. "It will be a huge thing in terms of quantification and comparison in robotics, because this has never been done before."

It's gotten to the point, Calli said, that the field doesn't have much choice but to take on the matter of standardization. "There's like a pool of algorithms and no one knows which performs the best. And we cannot proceed further without knowing what is working and what is not."

Ideally, the YCB Set will take on a life of its own. The objects and example tasks provided are just a beginning. Manipulation research progresses quickly and covers a wide range of technical interests and research approaches, so the five manipulation tests Dollar and his team provide are only examples of protocols that labs can use with the objects. That's why on the YCB Object and Model Set website, the research team has also provided a framework for other labs to contribute their own manipulation tests and benchmarks. There, researchers can see protocols from other labs and have a forum for discussion.

"The main thing is just getting other researchers to propose their own protocols and get people to utilize them," Dollar said.

To pick the right objects, the researchers combed through numerous robotics papers to get a sense of what kinds of items were most commonly used in manipulation tests. They visited stores for additional ideas. "The nature of this project is to apply to and span a wide range of research interests," Dollar said.

Preference was also given to objects that are durable and likely to remain in circulation without much change in the future. Standard consumer objects were chosen to keep the costs down. Each set costs about $350.

The objects are divided into categories. The food group, for example, includes a cereal box, a cylinder of Pringles chips and a can of Spam. Tools range from small nails to wood blocks and a cordless drill. Dollar said he aimed for a wide variety of sizes (the smallest item is a washer, the largest a water pitcher). Some items have simple geometric shapes that are relatively easy to grasp, while the complex shapes of others pose a greater challenge for robotic hands.

The items also include various task-based objects: a "box-and-blocks test" in which wooden cubes are to be placed in a box; a toy airplane that can be assembled and disassembled; and a variety of Lego pieces for building structures. The set also comes with a digital timer to measure how quickly certain tasks are performed.

Finding all the right parts for the YCB Set is one thing, but for the project to succeed, Dollar needed to convince other labs to adopt it. He and his associates have been busy distributing the sets at international robotics conferences. The YCB Set debuted at the IEEE International Conference on Robotics and Automation (ICRA) in May of 2015. Dollar said the reaction was "very positive" and they received about 50 requests for the sets, which are packaged specifically for easy travel. Researchers can also order the sets and have them shipped to their labs. About 100 robotics labs around the world now have the YCB Set.

"We want to get this into as many hands as possible, because that's the only way it's really going to stick," Dollar said.

Yu Sun, an associate professor in the Department of Computer Science and Engineering at the University of South Florida, said his lab is "one of the lucky ones" to receive the set. He said the YCB Set was featured in a grasping competition that he organized for the International Conference on Intelligent Robots and Systems in South Korea this October. His own lab has already produced some manipulation data using the objects.

"The good thing about using Aaron Dollar's object set is that other people will be able to use our data sets because they have the same objects and they can apply them to their own algorithms," he said. "Robotics deals with physical conditions, and if you can't replicate the physical environment, the data won't be useful."

Explore further: Warehouse item-picking robot is a perception-controlled mover