We speak to Prof. Yoshihiro Watanabe of Tokyo Institute of Technology about the development of a new high-speed projector that utilizes Dynamic Projection Mapping to change the appearance of a moving object without the need for markers. With this technology we come closer to redefining the limits of visual technologies, bringing alternative realities into the real world.
Please could you introduce yourself and tell us what inspired your career into visual technologies?
I am an Associate Professor at the Tokyo Institute of Technology and the principal investigator of the Latent Sense Lab, a research group exploring the possibilities to invoke a new sense of reality based on advanced technology centering on visual sensing.
When I was a bachelor's student, I wanted to research creating new robots. However, robot research involves various technologies such as actuators, control theory, and more.
I believe that visual sensing technology is a good choice to start with because it affects how the robot moves. In addition, to differentiate from other robots, I consider it practical to give the robot a performance transcending human capabilities.
As a result, I focused on improving the recognition speed of visual sensing to outcompete the abilities of humans. After a while, I found that such a visual technology was important not only for robotics but also for various unexplored applications.
I believe that high-speed vision can have the power to solve various limitations in a wide range of applications. Now, I mainly focus on exploring the possibilities invoked by combining this technology with a high-speed projector.
© Latent Sense Lab
The digital age is continuing to evolve at a rapid pace, allowing new technologies to take center stage within science. How are new, emerging technologies allowing us to interact differently with digital media?
Emerging technologies will allow us to interact more efficiently and naturally with digital media. As a result, the boundary between the physical and digital world will become ambiguous.
The LCD monitor is not the boundary anymore. Our five senses will capture physical and digital realities directly in the mixture. Our reality will be inevitably updated to survive in a world that fuses physical and digital matters
How did you become involved with Dynamic Projection Mapping (DPM) in particular?
My research strategy is to develop a new visual sensing technology, find its killer application, improve other insufficient technologies, and create an ideal application system. Under this strategy, projection mapping attracts my interest.
Projection mapping is a promising technology to augment the world. Without wearing any devices, we can seamlessly be immersed in a new visual world, fusing the physical object and projected virtual appearance.
However, it can only be applied to a static object like a building. If the object moves, the misalignment between the object and projected image occurs and breaks the augmented reality. I consider it a waste to limit the projection mapping in such static situations.
Dynamic projection mapping is a new stage of projection mapping, showing a visually augmented world even in a dynamic scene.
DPM inherently requires high speed in recognition and projection to avoid perceived misalignments between the physical moving object and projected image due to the large latency. The required speed reaches the level of 500 - 1,000 frames per second (fps). Such performance is 10 - 100 times faster than humans.
My technology, high-speed visual sensing, can be a piece to shape the concept of DPM. Based on this background, I was involved in the DPM research.
Depth-Aware Dynamic Projection Mapping using High-speed RGB and IR Projectors
Depth-Aware Dynamic Projection Mapping using High-speed RGB and IR Projectors. © Watanabe Lab/YouTube.com
Could you describe the new projector?
As I mentioned above, the other essential technology for DPM is high-speed projection. We need a high-speed projector that can project images at the level of 1,000 fps (frames per second). However, conventional projector applications do not require such speed. To break this limitation, we have started the development of high-speed projectors.
We have succeeded in developing a 1,000-fps high-speed monochrome projector in 2015. In 2019, we developed a high-speed and high-brightness color projector.
Although such projectors enabled new demonstrations of DPM, visual sensing possesses limitations; it requires attaching markers to the target to simplify the recognition and improve the speed.
A new RGB+IR high-speed projector overcomes this limitation as it can control and project visible RGB (red, green, and blue) and invisible infrared (IR) images simultaneously at high speed.
Using this projector, we can construct a system that can obtain the shape of the entire target surface directly using invisible wavelengths in a markerless way based on a projector-camera configuration, a long-established computer-vision technology.
At the same time, the system can adaptively manipulate the images to display in visible wavelengths based on its sensing results.
The projector can handle display and sensing simultaneously without interference. How did the research team approach developing this technology?
We chose the Digital Light Processing (DLP) scheme as DLP controls the projected image using a digital micromirror device (DMD), and knew that it is effective for high-speed projection.
A new challenge we faced was to realize high-speed IR projection. An IR projector has almost no demands, and its high-speed projection has never been realized. In addition, we were challenged to mount two DMDs to control RGB and IR images simultaneously
First, the research team diverts the IR light source—developed primarily for illumination in machine vision—to our projection application. Next, the RGB and IR images are aligned coaxially using our original optics engine, designed from scratch and optimized for the light sources used to keep their output power as high as possible.
We also optimize the projection optics to use as few lenses as possible while maintaining high-quality projection performance.
The same focal range for the RGB and IR images was maintained by considering the effects of different wavelengths.
Moreover, the newly developed control circuit and housing system can realize the maximum frame rates of the 24-bit RGB and 8-bit IR projections are 925 fps and 2777 fps, respectively. Two images are transferred from the computer to the projector with low latency via an optical fiber connection
© Latent Sense Lab
Are there any applications that you believe this new projector particularly lends itself well to?
DPM can provide a fantastic experience, especially as entertainment in amusement parks, game arcades as well as live stage productions.
It can also be used to meet advertisement demands, catching the attention of audiences.
DPM can reproduce the appearance of any material virtually, instantly changing it. This aspect is practical for design and prototyping.
The technology can use any surface to provide information presentation, supporting our work from tasks like medical surgery to daily tasks like cooking.
What is ‘invisible sensing’, and how does this relate to the research?
DPM needs to recognize the projected target. However, DPM changes the target appearance sequentially, making it difficult to realize robust visual sensing. Therefore, it is practical to achieve visual sensing, not in visible range but invisible range.
The captured IR image is unaffected by the projection mapping in the invisible range.
Moreover, we introduce visual sensing with the pattern projection in the projector-camera configuration using the new projector. If we project the pattern for sensing in the visible range, the mapping results are interfered with and degraded. Invisible sensing is a specific requirement for DPM to reproduce high-quality augmented reality.
Why was it important to project both visible RGB and invisible infrared images simultaneously?
It is important to achieve the highest speed projection. Our projector mounts two projection devices, which are DMDs, for each visible RGB and invisible IR image, respectively. Therefore, they can be controlled simultaneously.
If we mount only one DMD, RGB and IR images need to be projected in turn. Then, the latency from the target motion to the completion of the projection becomes larger. This increased latency becomes a critical problem in DPM because it causes the misalignment between the target and projected image, and augmented reality is not invoked.
Were any sensor technologies, such as infrared or image sensors, utilized to develop this projector?
By developing the RGB+IR projector, we carefully checked the sensitivity of the image sensors in the IR range. This is because the high-accurate sensing requires the captured IR image to be bright.
We needed to combine the IR camera and RGB+IR projector, develop the high-speed sensing based on the projector-camera system configuration, and develop high-speed target tracking technology to obtain the 3D pose.
High-speed RGB+IR projector. © Latent Sense Lab
The latter two technologies have been developed. Details can be found here. In this research, we introduce the RGB projector and IR projector separately.
We capture the 3D depth image based on the IR projector-camera system and apply high-speed 3D tracking using the captured image. Using these sensing results, the entire dynamic scene is shown to be freely augmented. By introducing the new RGB+IR projector to the current prototype, the system is expected to be compact.
High-speed RGB+IR projector. © Latent Sense Lab
A key aspect of DPM is the ability to match the surface of complex moving targets without noticeable misalignment. Are there any targets this is particularly difficult for?
DPM usually assumes that the target has a white surface. If the surface has color, we need to compensate for the color-changing effect. Although it is difficult, it could technically be solved.
Applying to non-rigid deforming surface in a markerless way is the next challenge to be solved. Although we already have overcome this problem in a marker-based way the markerless and high-speed non-rigid surface tracking remains challenging.
The final goal is to exist the augmented target and standard target together. To meet this goal, we need to realize DPM in a bright environment.
At the heart of the projector are the Digital Light Processing micromirror devices. Could you explain how they work?
Digital Light Processing (DLP) is based on a digital micromirror device (DMD). In DMD, many tiny mirrors are arrayed corresponding to the required resolution.
DMD projectors are superior in terms of speed. Each mirror can be tilted at two angles: at one angle, the light reflected at the mirror is outputted through the projection lens, and at the other angle, light is not outputted. Based on this principle, the luminance of each pixel in the projected image is controlled by flipping the mirror at high speed.
However, there is a limit to the speed increase that can be achieved based on DMD control alone because the mirror flipping speed is mechanically limited.
Our high-speed projector overcomes the limitation of the DMD flipping speed by using high-speed light source modulation control in precise synchronization with the mirror control.
Conventional markers compared to the new markerless system. © Latent Sense Lab
A significant benefit of this new technology is that markers are not required. What applications of DPM do you think will benefit from this?
DPM applications range from entertainment, advertisement, design, live production, fashion to work support. Although limited applications accept attaching markers, most prefer not to have them. For example, if the application is close to our daily life or the target is a living matter, it is better to introduce DPM without any markers.
What do you predict the future of projection technologies to look like in 30 years?
Projection technologies can have possibilities to support our life in various situations.
I hope the projection technologies play a role in connecting the physical and virtual world seamlessly and updating our definition of reality.
I also expect projection technologies to become the intermediate between the information display and illumination device. In the future, such technologies might replace current lighting technologies.
What aspect of this research do you find the most exciting?
Driving two wheels, which are developing new technologies and designing new applications, always excites me as an engineering researcher.
Also, the frame rate in our sensing and projection technologies is so fast that humans cannot perceive. It means that our natural evolution regards such a performance as useless. But now, once the technology obtains the speed, it finds a valuable role to fit with our new age. Such a situation is amusing for me.
In particular, DPM could have the power to shake the implicit understanding of reality, which is a topic that all people think about. I'm delighted to have one of the few occasions to involve people in thinking about how our research could impact the future.
Stay up to date with this research on the laboratory's website.
About Prof. Yoshihiro Watanabe
Yoshihiro Watanabe is an Associate Professor at the Tokyo Institute of Technology. He received a Ph.D. degree in information science and technology from the University of Tokyo in 2007. From 2018, he served in his current position.
He leads the Latent Sense Lab, a research group exploring the possibilities to invoke a new sense of reality based on the advanced technology centering on visual sensing. The key is speed transcending human capabilities. He believes that the next reality is driven by the technological control of the unseen moment. His group invents a new type of high-speed sensing, computational projection, dynamic digital archiving, user interface, and augmented reality.
Disclaimer: The views expressed here are those of the interviewee and do not necessarily represent the views of AZoM.com Limited (T/A) AZoNetwork, the owner and operator of this website. This disclaimer forms part of the Terms and Conditions of use of this website.