Robots and Drones Get the Gift of Israeli Machine Sight

By David Shamah in The Times of Israel

Robots that move around and do the heavy lifting, drones that drop off stuff you ordered on-line an hour before, self-driving cars, even self-driven vacuum cleaners that don’t bump into the furniture – it’s all part of the big technology revolution that 3D vision is on the verge of bringing to the world. And according to Igal Iancu, a senior manager on Intel’s RealSense 3D vision tech team, based in Haifa, none of it is going to be possible without a heavy dose of made-in-Israel technology.

“Israel is a hub of innovation in machine vision, chip development, and 3D technology,” Iancu told The Times of Israel in an exclusive interview. “Israel was the natural place for the development of RealSense, which combines hardware and software to bring human-like senses to personal devices, so they can experience the world like we do.

Brian Krzanich, the chief executive of Intel
Brian Krzanich, the chief executive of Intel

The idea of RealSense, said Iancu, is to enable devices to “see” in the same way humans do. “When we look at the world around us, our brains automatically build a 3D model of our surroundings,. They identify objects like people, animals, and pieces of furniture and figure out how big and far away they are. By introducing Intel RealSense 3D cameras, Intel is enabling devices to see like us, so they can understand the people using them and the world around them. This will allow us to interact with our devices in a much more natural way and have an immersive experience.”

RealSense sees the distance between objects, separating objects from the background layers behind them. This gives much better object, facial and gesture recognition than a traditional camera, according to the company. This visual data, creates a touch-free interface that responds to – and understands – hand, arm, and head motions as well as facial expressions.

3D vision technology – in which 3D cameras can be used to interact with computers, TVs, and gaming consoles – has been around for a few years. One of the original pioneers in 3D vision tech was an Israeli firm called PrimeSense, which supplied technology to Microsoft to enable it to build the Kinect 3D interface, used in Sony’s Xbox gaming console (PrimeSense was subsequently bought in 2013 by Apple for $350 million). Many other Israeli firms, such as Omek Interactive (bought by Intel in 2013) are also responsible for important breakthroughs in the field.

But until now, all that 3D interactivity has been in large devices, like gaming consoles – and while RealSense could be used for those purposes as well, with a RealSense-based 3D camera for PCs already on the market, Intel has much more in mind for its 3D tech, said Iancu.

“3D cameras are going to be very important in Internet of Things (IoT) devices, because in order for a device to interact with its environment it has to understand it,” said Iancu. “A robot that moves around needs to understand depth, to detect how far away it is from an object and activate a software routine that tells it to turn and move in a different direction.”

You can do this with a 2D camera – coupled with very heavy processing – as well, but it’s much easier and more efficient with a 3D camera; in devices without huge amounts of memory and computing capability, like IoT devices, there is no way to do 3D processing without a chip that can communicate environmental information to a camera. Devices with Intel RealSense 3D cameras have three lenses: a conventional camera, an infrared camera, and an infrared laser projector. Together, the three lenses allow the device to infer depth by detecting infrared light that has bounced back from objects in front of it.

“Our objective is to provide the 3D tech that can be integrated into hundreds of devices, and provide a software development kit that will allow developers to build applications for those devices to interact with the environment and with users,” said Iancu. “In order for such an SDK to exist, the technical “back office” details that make 3D interaction work already have to be on board, “and we roll all that capability into RealSense.”

Naturally, all that crunching has to be rolled into a chip that is small enough to fit into a small device, while operating with a low power requirement (the more power used, the more battery power a device needs).

The Israeli angle
Fortunately, for Intel, it has just the chip – its new Skylake line of processors and chips, which were also developed in Israel. “For a device to understand what is happening – whether it be a robot or a vacuum cleaner that ‘sees’ an obstacle and goes around it, or a TV that responds by changing the channel when a viewer moves their hand in a certain way – you need a lot of processing power.” RealSense’s system captures the raw data seen by the camera and sends it to a high-speed chip of the Skylake family, which then delivers the information to an application.

Some of those applications, said Iancu, could include Advanced photography editing, measuring objects and environments, augmented video conferences, eliminating passwords, and of course gaming. In one application, said Iancu, “you could use the camera to create an automatic green screen, to mute the background and to segment the actor from it.” The current background could be replaced with any other one, and the viewer would see it as ‘real life,’ in 3D, on their screen.

3D can also be used to allow a device to detect a user, allowing access only to the authorized individual. “We’re working with car companies to build a system where the car will identify the person sitting in the driver’s seat, meaning that it won’t start unless the right person is there.” A version of such face authentication is now available in Windows 10; computers equipped with a RealSense 3D (F200) camera, coupled with the Windows Hello protocol, allows users to log onto a computer just by looking at the camera.

There are vision systems out there in devices, but they are far less sophisticated than RealSense. “The Roomba vacuum cleaner operates automatically, but it does bump into things,” said Iancu. “The current versions use sensors to avoid obstacles, but with a RealSense camera installed, it wouldn’t bump into anything.” Ditto for drones; at a tech show earlier this year, Intel demonstrated a drone outfitted with an array of RealSense cameras that provided 360 degree coverage, with the drone managing to avoid people and other objects that were in its path by switching direction.

Intel is working with dozens of software vendors to develop new applications and software that takes advantage of RealSense technology. During the keynote address at this year’s Consumer Electronics Show, Intel CEO Brian Krzanich demonstrated new applications and capabilities for RealSense from FoodNetwork, Lego, McAfee, and Disney, in addition to previously announced third-party collaborations and plans with 3D Systems, Autodesk, Dreamworks, Iridium, Scholastic, Tencent, and many others.

“We decided to move ahead with this about 6 or 7 years ago, when we were thinking about what would make people want to upgrade their computers,” said Iancu. “It was becoming clear that vision was the next big thing, but we wanted something more than just vision – we wanted a performance killer. When you add ‘biological’ capabilities to a system – like sight, hearing, speech – you can build something that people want to buy. Israel has been an essential part of this, for several reasons, and we will keep developing and perfecting this technology, leading it from Haifa.”