How robots are mastering the art of dexterity

Humans have proven themselves to be the master of dexterity, but with the advancement of technology robotics could catch up very soon. Robotics are learning dexterity much faster as technology advances

The capabilities of real-world robots fall a long way short of those of Westworld’s murderous hosts, but on this small point, reality and fiction are in agreement: hands, and the manipulation of objects, are particularly challenging aspects of robotics.

“Grasping is the critical grand challenge right now,”

 says Ken Goldberg, an engineer at the University of California, Berkeley.

In the past 50 years, robots have become very good at working in tightly controlled conditions, such as on car-assembly lines.

“You can build a robotic system for one specific task picking up a car part, for example,” 

says Juxi Leitner, a robotics researcher at the Australian Centre for Robotic Vision (ACRV), based at the Queensland University of Technology in Brisbane.

“You know exactly where the part is going to be and where the arm needs to be,” 

he says, because the robot picked up the same thing from the same place
“the last million times”.

But the world is not a predictable assembly line. Although humans might find interacting with the countless objects and environments found beyond the factory gates a trivial task, it is tremendously difficult for robots.

These unstructured environments represent the next frontier for researchers across the field of robotics, but such environments are particularly problematic for robots that grip.


Any robot hoping to interact physically with the outside world faces an inherent uncertainty in how objects will react to touch.

“We can predict the motion of an asteroid a million miles away far better than we can predict the motion of a simple object being pushed across the table,” 

Goldberg says.

Some researchers are using machine learning to empower robots to independently identify and work out how to grab objects.


Others are improving the hardware, with grippers ranging from pincer-like appendages to human-like hands. And roboticists are also gearing up to tackle the challenge of manipulating objects gripped in the hand.

Advances in robots’ ability to handle objects could have enormous societal impact.


 Commercial entities, particularly those involved in the movement of varied goods, are following developments closely.

“There’s a big demand. Industry really wants to address this because of how fast e-commerce is growing, with interest greater now than ever, it’s an opportunity for the research to really be put into practice.”

Learning to learn

The heightened industry interest is exemplified by an annual competition organized by e-commerce giant Amazon for the past three years.


 The Amazon Robotics Challenge asks teams of researchers to design and build a robot that can sort the items for a customer’s order from containers and place them together in boxes.


The items are varied, ranging from bottles and bowls to soft toys and sponges, and are initially jumbled together, which makes it a difficult task in terms of both object identification and mechanical grasping.

In July 2017, Leitner’s ACRV team claimed victory with a robot called Cartman, which resembles a fairground ‘claw’ game. An aluminium frame supports the claw assembly.The robot has two tools for picking up objects, known as end effectors: a gripper with two parallel plates, and a suction cup backed by a vacuum pump.


For each object the robot encounters, the researchers specify which effector it should try first. If that doesn’t work, the robot switches tools.

First, however, the robot must find the item it’s looking for. The team tackled this challenge by using machine learning.


The main input for the robot is an RGB-D camera, a technology that is popular among roboticists and that can assess both colour and depth.


The camera looks down from the effector into the boxes below. From this vantage point, Leitner explains, Cartman labels each pixel according to the object it belongs to a form of deep learning known as semantic segmentation.


 Once a cluster of pixels representing the desired object is found, the camera’s depth-sensing capability helps the robot to work out how to grab the item.

“In simple terms, we attach to the bit that pokes out the most,” 

says Leitner.

Rapid advances in machine learning underpin a lot of recent progress in grasping.

“Software has been the bottleneck for ages, but it’s becoming more advanced thanks to deep learning,” 

says Pieter Abbeel, a deep-learning specialist at the University of California, Berkeley.

 These developments have, he says, opened up “whole new avenues of robotics applications”.

Abbeel is co-founder and chief scientist of covariant.ai, a start-up in Emeryville, California, that uses deep learning to train robots. Rather than program a robot to perform a specific action, humans provide demonstrations that the robot can then adapt to deal with variations of the same problem.

The human trainer views the camera feed of the robot arm through a headset, and uses motion controls to guide the robot arm to pick up objects. The process feeds a neural network with data on the approach taken.

“With just a few hundred demonstrations done in this particular way, you can train a deep neural network to acquire a skill, and I don’t mean acquire a specific motion that it’s going to repeatedly execute, but acquire the ability to adapt the motion to whatever it’s seeing in its camera feed.”

 says Abbeel.

Goldberg uses machine learning to teach robots to grasp, too. But rather than gather data from real-world attempts, his Dex-Net software is trained virtually. “We can simulate millions of grasps very quickly,” he says.


The software lets an industrial robot pick objects from a pile with a success rate of more than 90%, even if it hasn’t seen those objects before. It can also decide for itself whether to use a parallel-jaw gripper or suction tool for a particular object.

Dex-Net’s fourth incarnation will be presented in 2018. According to a metric being developed by Goldberg and roboticists around the world to aid reproducibility, known as mean picks per hour, the Dex-Net system is now among the fastest pickers around. It can achieve over 200 picks an hour — still behind human capacity, estimated at 400–600 picks an hour, but far ahead of the numbers achieved by the teams at the most recent Amazon Robotics Challenge (see ‘A measure of success’).