Professor Paul Newman FREng - Instilling robots with lifelong learning

In the basement of an ageing red-brick Oxford college, a team of engineers is changing the shape of robot autonomy. Professor Paul Newman FREng explained to Michael Kenward how he came to lead the Oxford Mobile Robotics Group and why the time is right for a revolution in autonomous technologies.

Paul Newman’s fascination with mechanical devices started with a toy puppy. While most five-year-olds would spend hours playing with a new present, Newman confesses that, to the distress of his mother, he quickly ‘skinned’ the mechanical puppy so that he could see how it worked. “I can remember then being utterly fascinated about something mechanical acting in the world of its own volition.”

This foray into reverse engineering was the first of many signs indicating that Newman would become an engineer. “I was always trying to make machines,” he adds. When the young Newman set out to make his own devices, Lego was important. He tried to ‘program’ his creations mechanically, using strips of Lego to move things around. This was before the ubiquitous toy went electronic. In any case, there were no computers or computer games at home. “I never got to play on computers. I missed all of that.”

EUREKA MOMENT

Newman’s first encounter with a computer came in 1991, when he went to the University of Oxford to study engineering. That introduction was a revelation, he says: “There is something extraordinary about writing a text file, and then a car drives itself! I was hooked. That was the start of an addiction that I have never wanted to shake off.”

Newman’s interest in mobile devices and how they orientate themselves began after a gap year as a telecommunications engineer in industry. He then moved to Australia to the Centre for Field Robotics at the University of Sydney to study for a PhD. Newman’s research was about getting machines to navigate their way around places they hadn’t encountered before. Although he was now developing software, he had not abandoned machines: he built an autonomous submarine as a test-bed for his research.

After completing his doctorate, Newman moved back to the UK as a navigation engineer with Sonardyne International. The company, with its origins in the UK’s North Sea oil business, needed to control remotely-operated vehicles working underwater. Newman worked on the navigation technology that went on to play a part in the fixing of the leak on the underwater service vehicles on the Deepwater Horizon oil spill in the Gulf of Mexico in 2010.

Professor Paul Newman FREng

Newman returned to academic research in the Department of Ocean Engineering at the Massachusetts Institute of Technology (MIT) where he continued to work on underwater systems. While there, he developed an open software architecture for autonomous robots. With its own annual conference and website, the software, called MOOS – Mission Oriented Operating Suite – it has become an industry standard, with widespread academic and commercial use.

Had it not been for his and his wife’s desire to bring up their children in the UK, Newman might have remained abroad. “America was awesome,” he insists, but apart from the family issues, there was another drawback: no cricket. So, in 2003, he returned to Oxford to take the first step up the academic career ladder as a lecturer in information engineering. During his first summer back, he also joined a local cricket club. “I find some of my happiest moments down there on Sundays in the summer.”

Newman’s move to Oxford meant that he could start to build a research group. His work took him out of the water and onto four wheels. The timing was right: the technology behind autonomous systems had changed greatly since earlier Oxford experiments. Newman’s predecessors had worked wonders with mathematics, but they lacked the computing power needed to make the most of their algorithms, and to process all the data it takes for vehicles to drive themselves. “We have the flip side of the maths now. That’s the data. You don’t have the maths without the data.”

The processing power of today’s computing is one of the elements that has changed robotics research. It is also now much more affordable: the cost of the memory needed to store data has plummeted. In effect, Newman explains, “Memory is ‘free’ right now. Disk space is free.” This means that mobile systems can store data even if there is no obvious immediate need for it. “I always say to people that the cost of deleting something is potentially infinite, but keeping it is finite,” says Newman. “We keep the data around with us, so the robots accumulate experience,” he adds. He illustrates this with an example from a very different domain. “Imagine a surgical robot that can share all of the experiences of all the other surgical robots as soon as it is turned on.”

There has also been a leap forward in the ability to collect that data, with the laser- and vision-based sensing that is at the heart of Newman’s approach to autonomy. There has been an explosion in camera technology even in the time that Newman has been working in the area.

COMMERCIALISING KNOW-HOW

Today, as BP Professorial Fellow in Information Engineering, Newman runs a group of 45, the Mobile Robotics Group (MRG) in the Engineering Department at Oxford University. “We work on what are the fundamental problems for mobile autonomy,” he explains. The MRG’s remit means that Newman can indulge both his software addiction and his early interest in mechanical devices. The group’s work is, he says, about 80% software and 20% hardware. Software is also the focus of a business that Newman has set up with his colleague, Professor Ingmar Posner, to ensure that their academic research does not languish on library shelves and that the UK benefits economically from their work. The company, Oxbotica, is commercialising the software and technical know-how that the researchers develop to enable autonomous vehicles. “I am extremely ambitious about what I want us to do for the UK,” says Newman. “I care a lot about a sovereign capability for UK robotics and autonomous systems.”

The system that they have devised, and protected by a suite of intellectual property, travels under the name Mercury. Oxbotica can get into detailed engineering that does not fit into the scope of an academic group. It hopes to do this by working with other companies that supply systems to the motor industry, as well as car makers themselves. “We want to help companies get autonomous technology into their products,” he explains. “We have no immediate plan to physically build a forklift, a train, or a space vehicle, but we do have the software and algorithms that can make those machines smarter to map, navigate and actively interact with their surroundings.”

Driverless cars are the most eye-catching examples of the MRG’s research in mobile robotics. The garage in its basement contains a Nissan electric car with a boot full of computer hardware. The car sits next to a golf buggy, bristling with sensors and electronic technology that has been getting to know the roads of Milton Keynes.

The newest vehicle in the basement is a LUTZ Pathfinder pod. This is the prototype for a fleet of 40 pods that are destined to demonstrate the practicality of self-driving cars on routes around Milton Keynes city centre as a part of the £20 million UK Autodrive programme. The pods, due on the road in 2016, will have as their ‘brains’ the MRG’s Mercury autonomy system. When the MRG has finished experimenting with the first LUTZ pod, Oxbotica will develop the autonomous control systems and sensor subsystems for the fleet of pods. The company will be involved in transplanting the golf buggy’s knowledge into the vehicles that will link Milton Keynes station to its centre.

MRG MISSION

The MRG collects and stores data from its autonomous vehicles and then uses information engineering to process it and add value. Newman says that the value of this information varies: “Sometimes the information gathered could be just wealth creation because you want to predict some financial outcome. On the other hand, it could create value because you want an early diagnosis of a tumour.”

In the case of mobile robots, information engineering has to deal with three basic questions that, as Newman puts it, “dominate my working life”. Those key questions are “Where am I? What’s around me? And what do I do next?” The acronym for this task is SLAM – simultaneous localisation and mapping. SLAM is the ability to put an autonomous vehicle somewhere that it does not know about and then have the vehicle look around the place and create a map of its environs and, while it builds the map, to use it to navigate its way around.

The ability to store a vehicle’s memory of where it has been changes the game for autonomous vehicles. “We call it ‘lifelong learning’,” says Newman. Vehicles can even share that memory. If one car travels along a route, it can pass that information on to another vehicle. This is just what the MRG is doing with its memory transplant between the golf buggy and the Lutz pods.

Experience-based navigation: by accumulating experiences, MRG’s robotcar is able to handle diverse and often cyclic appearance changes. The system gauges the varying appearance of locations (columns) in a wide range of lighting and weather conditions, scene clutter, camera occlusions, and permanent structural change in the environment

Driverless cars are just one example of autonomous vehicles and of the systems that the MRG works on. Some applications of the group’s technology have to remain under wraps, but they go beyond self-driving cars. Autonomous variants of just about anything that moves could be produced: trains, forklift trucks, boats and mining equipment.

Another important aspect of MRG’s work is endurance. Newman isn’t interested in getting something to work in time for a conference and then leaving it to gather dust in a corner. “We are trying to do thousands of kilometres of navigation and localisation. That is really challenging for a research group.”

Newman and Posner enjoy throwing additional challenges at their group. Newman tells of the time a few years ago when he took the team down to see the Nissan electric car that had just arrived. He told them to forget about their usual research and gave them just three months to convert the car “from forecourt to autonomy”.

The deadline provided lessons for academics more used to projects that last three or four years. “It was absolutely invaluable,” says Newman. “It showed in such a stark and unforgiving light that, when you thought that you could do something, actually you can only do it sometimes. It is interesting to take an academic group through that,” he adds. “Three years on, we are still reaping the benefits of those 14 weeks of intensity.” Such experience also comes in handy when you are trying to convince commercial engineers that you understand what they have to do in their work.

Another tactic that is, apparently, novel for many academic engineering groups is to recruit a small team of engineers to help the researchers turn ideas into demonstrable working systems. This helps convince businesses that the products of research can be real-world technology and not just academic theory. As well as having this team of dedicated engineers working alongside the MRG, the group’s basement also has an additive printing system to ‘print’ professional-looking components.

BUSINESS MODEL

Newman’s desire to ensure that their research is used and appreciated in the outside world explains why the MRG adopts a different funding model from many academic groups. The group avoids individual research contracts with companies. Instead, Newman has assembled a club of like-minded companies and organisations that share an interest in autonomous systems. As Newman describes it, the club is a two-way affair, with the academics bouncing ideas off the industry members who chip in with their own views on the real issues that they face. “Working with a group of companies keeps us focused on the core tech problems. It stops us from going down a rabbit hole of solving a particular problem for one partner, because it gets balanced by the other partners.”

Industrial backing is also an important part of Newman’s approach to financing the MRG’s work. “I have a very complicated funding mechanism,” he explains. “I am pretty good at raising money.” Here, too, he likes to see things in business terms, drawing on his own time in industry. “In a way, this place is like running a company. You have to worry about products, sales, fundraising. You have to worry about recruitment.”

An important part of the MRG’s funding comes from the Engineering and Physical Sciences Research Council (EPSRC). In 2010, Newman received a prestigious Leadership Fellowship worth £1.5 million from the EPSRC “to develop fully autonomous robots and vehicles that do not need infrastructure”. Newman says that without this funding, he would never have been able to focus on building a large world-class research group. The Council followed this up last year with an EPSRC Programme Grant to Newman and his colleagues of nearly £5 million for CAREER TIMELINE AND DISTINCTIONS

Born, 1973. Awarded M Eng, Engineering Science, Balliol College, Oxford University, 1995. Telecommunications Engineer, Wandle and Goltermann, 1995–1996. Awarded Ph.D. in autonomous vehicle navigation, Centre for Field Robotics, University of Sydney, Australia, 1999. Navigation Engineer, Sonardyne International, 1999–2000. Postdoctoral Associate then Research Scientist, Department of Ocean Engineering, Massachusetts Institute of Technology, 2000–2002. Departmental Lecturer in Information Engineering, University of Oxford, 2003–2005. University Lecturer in Information Engineering and Fellow of New College, Oxford, 2005–2008. Reader in Engineering Science and Fellow of New College, 2008–2012. EPSRC Leadership Fellowship, 2010–2015. BP Professor of Information Engineering and Fellow of Keble College, 2012–Present. Fellow of the Royal Academy of Engineering, 2014. Fellow of the Institution of Engineering and Technology (IET), 2015.

BIOGRAPHY

Michael Kenward OBE has been a freelance writer since 1990 and is a member of the Ingenia Editorial board. He is Editor-at-Large of Science|Business.