Sony talks about the future of “Polarization Image Sensor”

One day we might get those kind of sensors on E-mount cameras. Here is what Sony has to say about this:

Imaging technology is not all about creating visual images of what we can see, as we might think. At Sony Semiconductor Solutions Group (SSS Group), we pursue a challenge of capturing images of the invisible.
The polarization image sensor detects polarized light, which is the orientation of the light wave invisible to our naked eyes. With this sensor, conventionally difficult tasks, such as shape recognition, reflection removal, and fault detection will be made easy. The development team worked strenuously in this endeavor with a big idea. They delved in this almost uncharted territory of polarization, developed the world’s first*1 polarizer integrated into the semiconductor manufacturing processes, and brought it to mass production. It offers a great potential for progress in manufacturing, the intelligent transport systems (ITS), and further afield, such as medicine and scientific research. The team’s stories about how they developed this sensing technology reveal the future possibilities brought about by polarized light.

*1. According to Sony research on May, 2021

M.Osawa

Sony Semiconductor Solutions Corporation
Imaging System Business Division
Main job : establishment of evaluation technology
Responsibility : production engineering

M.Kato

Sony Semiconductor Solutions Corporation
Imaging System Business Division
Main job : business marketing planning and production of promotion contents for polarization image sensor
Responsibility : business planning

I.Yamagishi

Sony LSI Design Incorporated
System Solution Engineering Division
Main job : development of optical environment for testing and evaluation
Responsibility : development of evaluation environment

T.Yamazaki

Sony Semiconductor Solutions Corporation
Research Div.1
Main job : development of image sensors
Responsibility : device development

Yamagishi：Light behaves like a wave. It has amplitude (brightness), wavelength (color), wave vector, and orientation of the oscillation (polarization).
To put it simply, we humans can perceive light only partially, like its brightness, color, and the direction of propagation, but polarization refers to the orientation of the light oscillation, which cannot be directly perceived.
The polarization image sensor captures this orientation of the wave, and can be used to detect changes in the orientation as the light beam hits the surface of an object. This will enable, for example, to discern shapes of objects or remove the reflection on a water or glass surface to capture what is on the other side.

Kato：Mainly, it is being evaluated and introduced in the manufacturing and ITS industries. So far, polarization has been utilized in manufacturing only for some areas, such as distortion inspections of glass or transparent plastic products and visual inspections of metal products, as well as in microscopes. Sony’s image sensor with four-directional polarizers now begins to expand the scope of the sensor application to new areas. In manufacturing, for example, inspections of black-colored objects were done using the naked eye, which involved high chances of failed detection. This can be automated. Also, where previously it was necessary to rotate the polarizing filter and take multiple shots in order to identify defect parts, the sensor enables to do the task in one shot. For the ITS industry, the sensor offers value in terms of reflection removal on vehicle windscreens.
Furthermore, diverse industries are showing interest for various applications, such as in automotive, medical, research, mobile, and security industries.

Kato：It is not the aesthetic quality that is required for industrial sensors, but the quantitative data. It is like yes/no distinction or specific values such as length of the object’s surface, in order to detect contamination or defect, or identify objects.
Sensing technology relies on changes in contrast. Conventional image sensors look for the contrast in brightness and color. With the polarization image sensor, there is an added parameter, that is the contrast in polarized light. This contributes to more accurate detection.
To obtain polarization information using a polarizing filter, it is necessary to take several shots, each time rotating either the object or the polarizing filter. This requires a mechanism to rotate the polarizing filter, which makes it a time-consuming process, so it was not really practical to apply polarization to inspections on production lines. Sony polarization image sensor is equipped with a four-directional polarizer, which enables to realize polarization imaging in one shot. This means that polarization information can be obtained in real time.*2

*2. This depends on the subsequent processing capacity.

Yamazaki：We were to develop the polarization image sensor that the world had never known before. It was all new experience in many ways.

For example, we had to figure out ourselves what kind of components were necessary for the image sensor, what should be measured to obtain the required polarization information, and what level of precision we should aim for. In terms of the polarization data processing, we worked closely with the polarization algorithm development team from Sony’s R&D division, and the device properties gradually emerged through document investigation and repeated polarization simulations and field tests on processing data for reflection removal and object shape recognition.

There was also a problem with respect to the extinction ratio. This is a property unique to polarization image sensors. It is a parameter that indicates how accurate the polarization measurement is. The higher the value the more accurate it is. However, it is inversely proportional to the transmittance (sensitivity). There was a dilemma, that we wanted to achieve higher extinction ratio but at the same time avoid compromising the transmittance. These two incompatible properties posed a major challenge, but finally we achieved the target level of extinction ratio while maintaining the level of transmittance.

Kato：To explain the structure, the diagram below illustrates a section of the polarization image sensor. The stripe represents the polarizer, which is a piece of processed metal with linear slits (in other words, it is like an array of ultra-thin metal strips).

Yamazaki：Initially, the gaps between the fine strips of polarizer were filled in with oxidation film from the viewpoint of the stability and reliability of the processes, but it turned out that this method was an obstacle for solving the incompatibility problem. Theoretically, both the extinction ratio and transmittance can be improved by making the polarizer sufficiently finer than the light wavelength (i.e., thinner wire grid and narrower gaps), but there is a technical limitation in metal processing. Moreover, the wavelength becomes shorter as the light passes through the oxidation film. We realized that this structural configuration demanded even finer polarizer.
We then came up with the idea to empty the gaps in a polarizer, what we called the air-gap structure. This innovative structure made it possible to solve the problem of incompatibility between extinction ratio and transmittance.
One problem was solved, but an array of other challenges awaited. For example, establishing the stability and reliability of the air-gap structure, nano-order processing to realize a multi-layered structure of polarizer, and mounting this infinitesimal structure on an uneven surface of the image sensor.

Osawa：In terms of mass-production, it is necessary to ensure that this structure meets a certain quality level. If it is a 5MP image sensor, it requires 5 million of this microfabricated pixels uniformly manufactured. It is a very challenging task to maintain both high quality and yield, but this is what customers recognize in Sony products. So, we wanted to make sure that the polarization image sensor had an excellent quality, just like our other image sensors.

Yamagishi :I was involved in a team for establishing evaluation methods. It is generally believed that light from a light source is unpolarized unless it is filtered through polarizers. I was therefore thinking that we could create the test environment to evaluate polarization rather easily. In my mind, it was just a matter of mounting and rotating a polarizing filter on the same light source that was used for evaluating ordinary image sensors.

However, as we experimented in this setting, the pixel output was not exactly as expected no matter how many times we repeated it. Then we investigated the light source itself and found out that it was not entirely unpolarized, but slightly polarized from the beginning.
Light source equipment available in the market is standardized in terms of light intensity, homogeneity, etc., but there are no specifications to determine the polarization properties, which is not guaranteed. The actual state of polarization cannot be known without verifying it, and we found that even the light sources of the same model number could be variable (inconsistent) in their states of polarization. Furthermore, the light source once verified and adopted can alter its state of polarization over time. We found this phenomenon while pursuing evaluations, necessitating us to come up with a countermeasure quickly.
Likewise, polarizing filters first needed to be tested and verified after purchase. If they failed to perform to an expected level, we needed to procure more filters and repeat the initial process, making our struggles far more than initially anticipated.

Osawa：While design and manufacturing processes are important, there is an equally important factor which is the pre-shipment inspection of mass-produced wafers to secure quality assurance. This is where I worked closely with Yamagishi-san. We drew on the know-how developed in the prototyping by the Research Division and, with the help of engineers from Sony Semiconductor Manufacturing Corporation’s Kumamoto Technology Center, we compiled test data as the product was being developed, which allowed us to formulate the quality requirements for Sony polarization image sensor.
This being said, it was not as straightforward as it sounds. It was often the case that we did not get the test results as we expected. Every time this happened, we analyzed the evaluation environment, device properties, etc. to identify the cause of the failure.
Sony boasts rich data and know-how about ordinary image sensors, but there was almost none for polarization properties. Conventional methods often ended in failures. It was a tremendous struggle to create something new from scratch.
Luckily, we had specialists in device, equipment, optics, etc., who enabled quick and correct improvement. Their contributions were invaluable in bringing to market a high-quality polarization image sensor.

Kato：In manufacturing, it was customary to inspect products that are black or transparent visually or by taking samples due to their low-contrast attribute and difficulty in capturing with ordinary cameras.

The polarization image sensor makes it possible to automate these types of inspections, and also lower the chance of misidentification. Furthermore, it solves the issue of time-consuming, point-by-point identification of defects which involved rotating a polarizing filter. The polarization image sensor enables the inspection in one shot, realizing clear image rendering within the angle of view. This will help to reduce the time and manpower required for the inspection.
Other solutions using polarization information include reflection removal, 3D recognition, material identification, distortion detection of transparent materials, and more.
The polarization image sensor has a very wide scope of application, which cannot simply be described in a few words. Our dedicated website offers more information, including video materials. Also, please refer to the White Paper.
It would be nice if the readers, if interested, try it to see for themselves, as many polarization cameras by machine vision camera manufacturers are available in market.

Kato：Controlling light is an important factor in obtaining polarization information. We expect that the sensor will be introduced initially in factories, where they are able to control light sources.
For the manufacturing industry, the sensor should hopefully be leveraged more in inspections. Potentially, the polarization image sensor can make significant contributions to such inspections that previously could not be automated or were impractical with ordinary image sensors.
Another promising solution it can offer is the reflection removal on vehicle windows for ITS. Conventional polarizing filters are ineffective for removing reflection on certain vehicle models. I hear that there were some ITS projects that were not realized due to this problem. I hope that our polarization image sensor will make a breakthrough for such cases.

※For more information about reflection removal for ITS use, please click here.

Clear imaging without reflections – improved face recognition

Yamagishi：It is always advantageous for AI to have more data to base its judgments on. In this sense, the polarization image sensor will add another axis of information to what is already available, which is good. Where reflection interferes with the image, the technology enables to filter out unwanted data, which is a powerful advantage. With the polarization image sensor removing the unwanted light information, AI can learn which part of the data is unnecessary, so that the accuracy will also improve. Therefore, coupling with AI is ideal in both data enrichment and also identification tasks.
Similarly, combining the polarization image sensor with other types of sensors may contribute toward further improved accuracy in, for example, multispectral sensing to detect the freshness of meat, in which AI can have the wavelength information and additional polarization information to make a judgment.

Kato：I agree that polarization image sensor and AI are a very good match. With the increased volume of data obtained by the sensor, the sensing accuracy must improve in principle.
I once experimented the recognition of people, vehicles, and transparent bottles, comparing the recognition rates using the deep learning. In the environment for this trial, the polarization image sensor turned out to be superior than other types of sensors in every case.
In order to make a practical combination of polarization image sensor and AI, understanding of polarization is necessary to an extent, in addition to the knowledge about image data processing. This is rather a technically demanding requirement, but I think it means more opportunities for us in this area.

Yamazaki：There are the advantages unique to polarization image sensors, but there is a disadvantage in the current state of the device as well, which is that mounting a polarizer compromises the sensor’s sensitivity and image resolution. I would like to pursue the development of a polarization image sensor that overcomes this issue of compromised sensitivity and resolution, broadening the application of this sensor.

Yamagishi：Structurally speaking, it is possible to mount a polarizer on image sensors of various specifications. As the demand for polarization technology grows in the future, the need for polarization image sensors will diversify. I would therefore try and develop expertise in the technology for testing environments and light sources that ensure high quality no matter what kinds of properties they require.

Osawa：I have more opportunities to communicate with our clients these days, and they sometimes tell me about the challenges they face. With Sony’s strong line of unique sensors, including the polarization image sensor, I would like to be able to propose optimal solutions leveraging the best of these products.
If I come across a conundrum in trying to find a solution, I would take it as an opportunity to lead to the development of new technology. In doing so, my efforts would be a contribution to a better world.

Kato：I believe that sensing technology, as it continues to develop, will liberate people from manual tasks by enabling machines to do the tasks. It will help people to find more time to enrich their lives.
However, complex technology like this polarization technology would be wasted if people find it difficult to implement it. So, I think it is also important to develop an environment that allows people to take advantage of such technology.
Its value can only be realized by the technology being used. I would therefore work hard on adapting new technology to practical applications.

via Sony.net