Imaging systems are highly effective visual inspection solutions for display qualification and measurement. Images provide the contextual evaluation of a display to notice defects by comparing visual changes in color, luminance, and more characteristics throughout the full area of the display.
The method of converting light to digital input to generate an image is not exactly one-to-one, as inconsistencies in electronic signals happen when the values of light are converted into electronic data.
Different types of imaging sensor (CMOS and CCD) perform this conversion method in various ways, each with its specific advantages and weaknesses.
According to the imaging system’s sensor (along with other system specifications), unavoidable inconsistencies that are created during the conversion process of light data into an image may be more or less obvious, negatively impacting or benefiting the performance of the imaging system.
It is critical to understand the influence of imaging system specifications and sensor features in order to select a system that enhances the repeatability and accuracy of measurement data.
This is even more crucial when assessing the very restricted data-sampling area of one distinct display pixel, an important indicator of quality in today’s emissive, high-resolution displays.
Figure 1. Measurement images of an emissive OLED display taken by a high-resolution imaging photometer. The luminance level of each display pixel is measured to evaluate discrepancies in uniformity from pixel to pixel. Data can be used to adjust pixel output of non-uniform displays (left) to correct display uniformity at the pixel level (right). (Luminance values are shown in false-color scale to illustrate variations in brightness.)
Display Trend: More Pixels
In 2017, at the Society for Information Display (SID) Display Week, the demand from the keynote speaker, Clay Bavor (VP, Virtual and Augmented Reality at Google), was clear: “We need more pixels. Way, way more pixels.”1
Display innovation is a constant pursuit of increased pixel density and a higher resolution in displays that are observed ever closer to the eye.
To create visuals that are similar to real life, with greater color depth and contrast, the sharpness of visual elements must be improved, and screen-door effects must be eliminated in immersive virtual reality environments (among other objectives). This necessitates increasing the amount of pixels in a given display area, and enhancing the pixel fill factor.
As a virtual medium for communicating reality, a display must combine the virtual experience with reality with no interruptions, everything that is seen in a display should be presented with the same (or greater) detail.
This accuracy makes sure that displays have value as a piece of equipment for visualization. In wearables and smart devices, displays have become smaller as a means to increase integration flexibility and mobility.
Perceived at restricted distances, these small-format displays must contain more pixels into smaller spaces to attain the seamless visual features that consumers want. This means that, not only do displays comprise of “way, way more pixels,” but pixels are becoming increasingly smaller.
Figure 2. An example of the screen-door effect, which occurs when a display is magnified or viewed up close, such that the space between pixels is visible.
Figure 3. Increasing the number of display pixels per area requires that pixels become smaller. This illustration shows the impact on pixel size as display resolution increases within a 2.54-cenitmeter (approximately 1-inch) square area.
The Importance of Pixel-Level Measurement
A display pixel’s performance communicates the visual quality of a display. Manufacturers may investigate displays for multiple defects related to pixels in order to ensure quality. At the most simple level of pixel measurement, imaging systems notice dead or ‘stuck on’ pixels.
By analyzing pixel-level luminance values throughout display test images, these defects can be spotted easily.
With the market focusing on emissive displays established on OLED, microLED, and LED technology, more complicated measurement parameters have been released for the detection of pixel and subpixel non-uniformity.
As light is generated by every pixel in these emissive displays, with no broad-scale uniformity supported by a backlight, the luminance of each pixel can be very difficult, particularly across varying levels of brightness or ‘bright states’ of the display.
Figure 4. Measurement image of an uncalibrated OLED display (top) and a close-up of its display pixels (bottomright) where inconsistencies in output at the pixel and subpixel levels have resulted in observable differences in luminance and color.
Figure 5. A measurement image (top) and analysis image (bottom) of a display with pixel defects—including dead pixels, stuck-on pixels, and other particle-like defects.
Along with testing every distinct pixel, display measurement may also need to be carried out at the subpixel degree of the display.
The output luminance of every subpixel (normally creating red, green, and blue) signifies the total color of every display pixel. For example, equally combining RGB subpixel values generates a pixel that is white in color.
If subpixels generate different red, green, and blue values, when the subpixel sets are revealed, the color mixing will produce a large variety of white values. This inconsistency can generate obvious areas of non-uniformity (also known as mura) when seen by a consumer.
To correctly evaluate the visual quality of the present day’s increasingly pixel-dense and high-resolution displays, measurement systems must attain the correct pixel and subpixel level measurement that enhances performance at every light-emitting element.
Photometric measurement systems that are two dimensional (for example, imaging photometers or colorimeters) are especially useful for the analysis of display defects.
Utilizing high-resolution image sensors, these systems can be employed to investigate displays at the pixel and subpixel level and quantify discrepancies in luminance between each element.
In emissive displays such as OLED, microLED, and LED, the photometric imaging system enables manufacturers to determine the corrections for each pixel to create overall uniformity within the display.
As pixels become more densely populated and smaller throughout a display, the limitations of evaluating display quality become more complex.
Sufficient display qualification requires that measurement systems acquire enough apparent details of every pixel to determine their distinct features and photometric values, and provide reliable measurement data from pixel to pixel.
A high imaging resolution is required (the resolution of the imaging system’s sensor) to attain more measurement pixels for each display pixel. It also means decreasing the undesirable image noise recorded in each measurement pixel to make sure that the evaluation at this scale is repeatable.
Sufficient display qualification means that measurement systems must record enough apparent detail of every pixel to determine their distinct features and photometric values, and provide reliable measurement data from pixel to pixel.
Figure 6. High imaging-system resolution ensures that each display pixel is sufficiently isolated for measurement. This measurement image shows an imaging resolution of 10x10 sensor pixels to capture a single display pixel.
Figure 7. High imaging system signal-to-noise ratio (SNR) ensures that measurement accuracy is repeatable across display pixels.
Imaging System Specifications
Imaging systems are perfect for analyzing displays because similar to the human eye, imagers capture all obvious details simultaneously in order to provide contextual analysis throughout the complete spatial area of the display.
Imagers categorize display features such as mura (or light or dark masses in the display), non-uniformity within the display, and different visual features such as color, contrast, and brightness.
Figure 8. A photometric imaging system captures the entire display area in a single two-dimensional image for analysis (actual measurement images shown in “false color” to represent luminance values).
When a digital camera takes an image, photons of light are mapped to the camera sensor’s pixels. The more pixels that a sensor contains (a higher resolution), the more photons can be mapped to particular spatial positions, and the more detail that can be viewed in the captured image.
Through the phenomena of converting light to image data, an unavoidable amount of electron ‘noise’ is also recorded in every pixel on the sensor of the camera. This noise can decrease the precision of the details in the image taken.
Figure 9. An illustration of imaging sensor pixels, where each sensor pixel captures photons from a specific spatial position on the light-emitting display.
Imaging performance can have a critical influence on the imaging system’s ability to gather and translate photometric data from a display with consistency and precision.
The imaging system chosen for display measurement should offer the ideal specifications for the required measurement.
As the pixel density of displays expands, a display test process needs increasingly more precise imaging system performance, which is mostly a result of sensor resolution, optical quality, and electron noise to verify the ability of a system to determine correct light values at the pixel and subpixel level.
Figure 10. A display captured with a low-resolution imaging system. The measurement image (left) captures each display pixel across 3x3 sensor pixels. The pixel luminance is shown in the cross-section (right), where the contrast between pixels is very low, with potential cross-talk of measurement data from one display pixel to the next.
Figure 11. A display captured with a high-resolution imaging system. The measurement image (left) captures each display pixel across 6x6 sensor pixels. The pixel luminance is shown in the cross-section (right), where the contrast between pixels is much higher, reducing cross-talk of measurement data between pixels.
The resolution of an imaging system is crucial to capture detail in display measurement. It becomes very challenging to isolate small points of interest without adequate sensor resolution, for example display sub pixels and pixels, to acquire discrete measurement data for every light-emitting part of the display.
The data in Figure 10 presents an image-based measurement of pixels on the display of a smartphone. This imaging system has recorded each display pixel throughout 3x3 sensor pixels.
In the measurement image to the left, the amount of detail seen for each display pixel is very poor.
The cross-section to the right presents the imaging data with the percentage of maximum luminance throughout the display area (in millimeters). The contrast is very low between each pixel showing a lack of accuracy in defining each pixel by its illuminated area (increased cross-talk with neighboring pixels).
As the resolution is limited, the imaging system cannot gather enough detail to discern the accurate contrast between the dark and light display areas (the areas between each pixel).
The luminance value for every display pixel would be even less precise in much noisier images. A higher-resolution imaging system can collect more precise details about each pixel, which enhances the repeatability of data even in the presence of image noise.
In Figure 11, the same smartphone pixels from Figure 10 are investigated utilizing a system that attains 6x6 sensor pixels for a single display pixel.
There is much more detail to be seen in the left measurement image of Figure 11 compared to the image on the left in Figure 10.
In the cross-section shown in Figure 11, there is much greater contrast between pixels, reducing crosstalk, and considerably enhancing the precision of luminance values for every display pixel.
Signal is the amount of effectively interpreted light input, and the noise is the unavoidable yet undesired activity of the electrons. Signal-to-noise ratio, or SNR, offers a data point to compare imaging system performance.
A greater SNR benefits the repeatability of the imaging system (the system’s ability to constantly gather precise data) from pixel to pixel and from measurement to measurement in a display.
Data inconsistency is caused by a lower SNR as instances of noise are translated as important variations in the measurement, rather than as random fluctuations as a result of electron activity.
High SNR creates an image with precise light measurement data at more accurate spatial locations on the display, which is crucial when employing imaging systems for pixel-level evaluation and measurement.
In a small measurement area, such as the area of a single display pixel, there is a restricted amount of image sensor pixels that can be used to develop an understanding of the display pixel’s correct light values (color, brightness, and more).
If an imager’s sensor gathers a high amount of noise for each measurement pixel, the limited window of understanding the display pixel can become even more difficult, and may create changes in the measurement data from pixel to pixel (low repeatability).
The Rule of Six Sigma in Imaging SNR
An imaging system with high repeatability needs to have a low failure rate when it comes to separating important signals from undesired noise. It is a general rule that imaging systems should apply the theories of six-sigma (6σ) to set a tolerance for SNR performance.
To detect defects and reduce false positives in a repeatable manner, the defect contrast attained for every display pixel should be six standard deviations (6σ) beyond the image noise level of the sensor.
When analyzing displays comprising millions of pixels, maximizing SNR to this standard tolerance can reduce the measurement ‘failure’ or inaccuracy rate per pixel.
A very small defect in a display, such as a pixel defect that changes in contrast only slightly from neighboring pixels, offers relatively low signal compared to the background. A 6σ difference would help the system to detect this defective pixel in a reliable manner, and effectively all of the time.
As defect contrast falls below six standard deviations, the defect becomes more easily mixed up with the noise of the sensor, and the rate of failure is more prevalent.
Figure 12. Illustrations of signal-to-noise ratio (SNR), where blue is the meaningful signal and red is the undesirable noise. Improving this ratio (as in the bottom image) increases the likelihood of discerning the signal.
Figure 13. Data extrapolated from actual display measurement images to compare luminance deviation from background noise (SNR). Where SNR of 6σ is achieved (left), the signal is clearly discernible, even when sampling millions of data points. Where SNR of ~4σ is achieved (right), the signal may become confused with image noise due to statistical variation among millions of data points.
Figure 14. Illustration of the effect of pixel size on SNR given a fixed amount of electron noise (stars) amid the photons received (circles) for each pixel.
The Argument for Larger Pixels
Pixels on a sensor can be various sizes. A small pixel has a smaller capacity for photons (its “well capacity”), whereas a larger pixel has a greater well capacity. As it can contain more photons, a sensor with bigger pixels is more reactive to changes in the values of light values and offers more accurate, repeatable measurement data.
As outlined above, all cameras record images with a consistent, and inherent amount of electron noise, at multiple electrons per sensor pixel. Larger sensor pixels that acquire more photons maximize the ratio of true input (photons that produce the image), to false input (electron noise).
Once saturated (when the well capacity of a sensor’s pixel is reached), a larger sensor pixel will offer a larger ratio of good signal in comparison to undesired electron noise.
The illustration in Figure 14 demonstrates the influence of a specific amount of electron noise as recorded in a small sensor pixel (which gathers fewer photons per noise, resulting in a lower SNR), compared with a larger sensor pixel (which gathers more photons per noise, increasing the SNR).
Sensor Resolution vs. Sensor Size
The resolution of an image is determined by the number of pixels within the sensor’s physical area. A sensor can keep the same physical dimensions while maximizing resolution, for example, an 8-megapixel sensor can be the same physical size as a 29-megapixel sensor.
The variation is the pixel size. To optimize the number of pixels on a sensor of a particular physical size, the sensor pixels must become smaller. Utilizing smaller sensor pixels means that there is a more restricted well capacity for the photons in every pixel, resulting in lower SNR.
While an exceptionally high-resolution sensor would suggest that the images are of a higher quality, if the pixels are decreased in size, the ratio of image noise to good signal within each pixel is increased.
A high-resolution imaging system with a greater amount of inconsistent pixels is a result of this. The image taken by this kind of imaging system would include greater detail, but the details may not present information that is repeatable.
This can make an important difference when analyzing several very small regions of interest such as pixels across a display.
The logical answer to attaining a higher resolution would be to simply increase sensor’s physical size within the imaging system in order to get a larger number of larger sensor pixels.
Increasing sensor resolution along with maintaining pixel size requires a related increase in the sensor’s physical size. A large sensor in turn means that larger camera components are required.
This is an issue due to the restrictions surrounding standard hardware sizes in imaging systems. For a sensor to fit the imaging area recorded by a standard 35 millimeter lens, the sensor pixel size must also be restricted.
Increasing the pixel size without decreasing the number of pixels makes the sensor size beyond the imaging area of a standard 35 millimeter lens.
This means that parts of the sensor area will go unused, and even though the sensor has more pixels, the images captured by the larger sensor will not actually be in full resolution.
Figure 15. Illustration of how resolution is increased within a given physical area of a sensor by reducing the sensor pixel size.
Figure 16. Illustration of how increasing the physical size of a sensor can increase sensor resolution without reducing pixel size.
Figure 17. Increasing sensor size without increasing the size of the lens (top image) results in unused sensor area. Achieving full resolution of a large sensor requires that the size of the system’s hardware (lens, associated optics, camera casing) also increase.
Tailoring hardware size beyond the basic imaging components can result in issues regarding development cost and difficulty of the measurement system.
The aim for optimizing imaging performance is to strike the correct balance of sensor features to optimize the photo-sensing areas of the sensor within the common size limitations of the imaging systems available today.
This requires knowledge of the features of the sensor types available, and a comparison of each sensor’s ability to uphold photosensitivity (large well capacity) with smaller pixels (high resolution).
Figure 18. This measurement image of OLED subpixels gives an example of a low resolution/low-noise image (left) compared to a high-resolution/high-noise image (right). Neither is ideal for measurement—the ideal imaging system strikes a good balance.
Figure 19. A measurement image of OLED subpixels captured by an imaging system that provides an optimal balance of resolution and noise.
CCD vs. CMOS Imaging
There are two main types of imaging sensor, Charge-Coupled Devices, or CCDs, and Complementary Metal-Oxide Semiconductors, also known as CMOS sensors. The pixels of CMOS and CCD sensors both have photo-sensing elements.
The key variation between these sensors is in the structure of every sensor pixel, and the elements that attain the conversion of light to digital images. CCD pixels are analog and adjust their charge from one pixel to the next before reaching an output amplifier at the pixel array’s edge.
CMOS sensors have an amplifier in every pixel. The outcome is that CMOS pixels do not have as much photo-sensing area to collect photons, and many photons meeting the CMOS sensor may not get to the photo-sensing area within every sensor pixel.
Figure 20. An illustration comparing the size of the photo-sensing area per pixel of a CCD sensor versus a CMOS sensor.
As described, the photon-sensing area’s size restricts the well capacity of each pixel. A smaller well capacity can enhance the ratio of image noise per pixel (reducing the SNR), making pixel-level defects more difficult to detect.
CCDs are created to expand the photo-sensing areas of every pixel, and can contain more pixels per each area of the sensor while maintaining its well capacity (effective fill factor may be better for CMOS when a microlens array is applied).
This means that CCDs normally have greater SNR and a higher repeatability than CMOS sensors of the same resolution.
All sensors are good at detecting very apparent defects (such as a dead pixel in a display that is bright), but CCDs are excellent at identifying very low contrast defects for example pixels that are not uniform, even in displays analyzed throughout bright states (dark to bright).
CCD sensors are mainly used in applications that need highly precise, scientific imaging with excellent light sensitivity for this reason.
CMOS sensors are prone to being more sensitive to noise, but there are significant advantages of CMOS technology. CMOS sensors enable a quicker read-out of data when compared to CCDs.
They also function with a low consumption of power, using as much as one hundred times less power than CCDs.
As they can be made on essentially any traditional silicon production line, CMOS sensors are also more inexpensive to make, which drives the cost down for CMOS-based imaging systems.
CMOS sensors have historically provided lower sensitivity and resolution, but they are still selected in applications where defects are more easy to notice, and imaging speed is desired over other elements for optimizing automated visual inspection application (for example, high-speed machine vision applications for quality control on a busy production line).
CCDs are created to optimize the photo-sensing areas of every pixel, and can contain more pixels per sensor area while maintaining well capacity.
Photon Transfer Curve
The most basic and defining comparison of present day CMOS and CCD-based imaging systems is an investigation of Photon Transfer Curves2 or PTC.
The measurement displayed in Figure 21 shows how the SNR of each kind of imaging system adjusts as the sensor pixels become saturated with photons (along with when the capacity is reached).
The SNR should increase as each sensor receives more photons in its pixels, simply as more photons are captured compared to the residual noise generated. The first apparent suggestion from the data in Figure 21 is that the saturation limit is highly different for CCD and CMOS sensors.
This is due to the more restricted photosensing area per pixel in CMOS sensors. CMOS pixels cannot store as many photons as CCD pixels due to their smaller photo-sensing areas and a CMOS pixel’s full well capacity is attained quicker.
CCDs can store much more photons per pixel, enhancing SNR at full well capacity. From the data in Figure 21, the CCD pixel is shown to attain almost perfect SNR at total saturation.
A further point of observation from the data in Figure 21 is the variation in accuracy between CCD and CMOS sensors at lower luminance levels (that is, at the lower part of the X axis, where less photons are received).
CMOS sensors show lower SNR when less photons are received, for example, when the display is evaluated in a dark condition. CCD sensors have nearer to perfect SNR at these low levels of luminance, meaning that defects in dark displays are more simply and reliably noticed by the CCD sensor.
CMOS pixels cannot store as many photons as CCD pixels due to their smaller areas for photo-sensing and a CMOS pixel’s full well capacity is attained sooner. CCDs can store much more photons per pixel, enhancing SNR at full well capacity.
Figure 21. Graphical representation of actual test data showing the single-pixel SNR of two systems with the same image size—CMOS and CCD—compared to a theoretical “perfect” system on the orange line, which has a nearly pure shot-noise limit.
Measurement Across Luminance Levels
Assessing display quality commonly requires display measurement at different levels of luminance or ‘bright states.’
Specific pixels in a display can change significantly in their output performance throughout luminance levels, as they are controlled by various levels of input to generate a desired amount of light.
Variations are particularly frequent in emissive displays such as OLEDs, microLEDs, and LEDs where every pixel is independently controlled to create its own specific luminance output.
Figure 22. A gray-level test image displayed on a monitor is imaged by a CCD-based imaging system (left) and a CMOS-based imaging system (right) of equivalent sensor resolution. The CMOS image exhibits higher image noise in darker areas of the display.
The images in Figure 22 present the apparent variation between two CCD and CMOS sensors of the same resolution, which are utilized to image a display throughout various bright states.
The two imaging systems acquire the same display, presenting a test image with various gray values (dark to bright). When evaluating the darker gray values, the imaging systems receive less photons to their sensors.
A CCD sensor projects less image noise than the CMOS sensor at the darker areas of the display. This verifies the data displayed in the PTC graph shown in Figure 21.
The CCD sensor does not require high saturation to attain image accuracy, partly because of the large photo-sensing areas of its sensor pixels in comparison to CMOS.
The CCD sensor can attain higher SNR than the CMOS sensor while continuing to receive less photons from the darker display areas, enabling accuracy across all display bright states.
CCD-based imaging systems provide the most precise measurement data for very small, low-contrast defects, for example non-uniform subpixels or pixels in a display.
There are considerable advantages to CMOS technology for quick, inexpensive visual inspection; but the precision of current CMOS technology continues to be inadequate for repeatable pixel-level display measurement.
As CMOS accuracy reaches the performance levels of CCD SNR, particularly for analyzing small and densely-populated points of interest such as the present-day’s increasingly small, emissive display pixels, CMOS technology could become the ideal type of sensor for its advantages in power consumption and speed.
At present, more development is required before CMOS reaches the status of CCD performance for repeatability at a higher resolution.
References and Further Reading
1. [Charbax]. (2017, June 8). Google Keynote at SID Display Week, Clay Bavor, VP of Google VR/AR [Video File]. Retrieved from https://www.youtube.com/watch?v=IlADpD1fvuA
2. Gardner, D. (n.d.). Characterizing Digital Cameras with the Photon Transfer Curve. Retrieved from http://www.couriertronics.com/docs/notes/cameras_application_notes/Photon_Transfer_Curve_Charactrization_Method.pdf
3. Radiant Vision Systems. (2018, April 19). Resolution and Dynamic Range: How These Critical CCD Specifications Impact Imaging System Performance. Retrieved from: https://www.radiantvisionsystems.com/learn/white-papers/resolutionand-dynamic-range-how-these-critical-ccd-specifications-impact-imaging-systemperformance
This information has been sourced, reviewed and adapted from materials provided by Radiant Vision Systems.
For more information on this source, please visit Radiant Vision Systems.