A robot reading (DARPA via Wikimedia Commons)

A robot reading (DARPA via Wikimedia Commons)

Researchers at Cornell University are developing “Robo Brain”, a knowledgebase/database that could be an invaluable resource for those who build and program robots.

To get them to operate and carry out their assigned tasks – which could range from simple household chores to bomb detection or even performing surgery – robots are programmed with specific instructions.

Right now, that programming can be a very time consuming and tedious process that involves teaching the robot one function or action at a time, each of which could require a number of steps to get the machine to properly process and perform that function.

For example, robots built to assist people with daily chores also need to have a good understanding of various elements of a human’s environment and behavior.

The developers of this system say that Robo Brain will provide robot builders/programmers with both in one package.

Right now, Robo Brain is in the process of building its knowledge/data library by grabbing about 1 billion images, 120,000 YouTube videos, and 100 million how-to documents and appliance manuals that are being downloaded from publicly available Internet resources and websites.

Once material is downloaded from the internet, it’s translated into a form that robots can understand and then saved and stored in a location that robots can easily access when the information is needed.

Data used by Robo Brain system is also augmented by information from a number of related computer simulations as well as real-life robot trials.

Part of the system’s processing is to pick out various objects in the images and videos it downloads.  It then connects those objects to related text so that robots can learn not only how to recognize the objects, but also how the objects are used as well as related human aspects, such as language or behavior.

The researchers explain, as an example, if a robot sees a coffee mug with its built-in cameras or sensors, Robo Brain will provide it not only with the information that object is indeed a coffee mug, but also will provide a number of functional details.   These details could include that that liquids – hot or cold – can be poured in or out of it, that it can be grasped by the handle, and that it must be carried upright when it is full, as opposed to when it is being carried from a dishwasher to a cupboard.

The 'Robo Brain' Team (Cornell University)

The ‘Robo Brain’ Team (Cornell University)

“Our laptops and cell phones have access to all the information we want. If a robot encounters a situation it hasn’t seen before it can query Robo Brain in the cloud,” said one of the system’s developers, Ashutosh Saxena, an assistant professor of computer science at Cornell University in a university press release.

While programmers will still need to use current software platforms such as Robot Operating System or Robot Web Tools to actually program the robots, they’ll be able access needed material and data from Robo Brain via special software known as an application programming interface, or API.

Robo Brain’s developers use a system computer called structured “deep learning,” which is a sophisticated set of machine-learning algorithms, to help teach robots. Saxena tells us in an e-mail, “There are two challenges in learning representations from the different knowledge sources and data. First, there are several layers of abstractions that must be learned, e.g., going from raw pixels in the images to meaningful concepts such as an object.  Or going from raw sensor data to where a robot can grasp an object.  We use deep learning for this.  Second, one needs to capture the structure in the knowledge such as how two objects are related in physical or semantic space.”

Robo Brain’s large-scale database was designed by Aditya Jami, a visiting researcher at Cornell.  He said to think of his creation as something that could look like a table of relationships between Facebook friends, only think of it on the scale of the Milky Way Galaxy.

“One … advantage in this system is its ability to crowdsource the learning via people.  At Robo Brain page, we are showing the current learned concepts by the Robo Brain, and millions of people in the world can give feedback,” said Saxena.  “This help guides the learning of the Robo Brain, helping it correct its mistakes in learning.”

Right now, according to Saxena, while Robo Brain is an open-source effort, it’s only available and open to their university collaborators at Cornell, Stanford, Brown University, and University of California Berkeley.  The developers hope to open Robo Brain to more institutions in a few months, and then after a year of development, it will be offered to everyone.

Ashutosh Saxena Talks About ‘Robo Brain’ at 2014 Robotics: Science and Systems Conference