Resolution Is Not Accuracy: Reading the Four Indicators According to Conditions
Resolution, repeatability, linearity, and accuracy are different indicators with different measurement conditions. We explain why placing the numbers side by side does not allow for a valid comparison and how to interpret them correctly.
Inquiries regarding displacement sensor specifications and selection
Three Things That Happen When You Misread a Spec Sheet
Rather than the numbers themselves, the conditions under which those numbers were measured are the deciding factor in selection.
Misinterpreting Resolution as Accuracy and Making the Wrong Selection
Resolution is the minimum amount of change that can be identified. Having fine increments in display or output is one thing, but actually being able to read changes in those increments is another, and how close that is to the true value is yet another. If you select based only on the resolution value, actual measurements may not fall within the expected control range due to the effects of linearity and temperature drift. The starting point for selection is to first determine which indicator affects the desired control range before looking at the catalog.
Comparing Numbers With Different Measurement Conditions Side-by-Side
Repeatability is the variation when measured repeatedly under the same conditions. However, the numbers change depending on the number of averages set, whether it is expressed in σ or as a full ± width, and at what measurement distance the value is. Comparing catalogs with different condition descriptions based only on numbers can lead to a conclusion opposite to actual performance. It is a prerequisite to align the conditions when reading.
On-Site Conditions Deviating From the Assumptions of the Specifications
Catalog values are measured using standard targets, installation distances, and temperature conditions. In actual on-site environments, conditions such as rough target surfaces, mirror-like surfaces, tilted targets, and ambient temperature fluctuations are added. In particular, temperature drift gradually appears as a deviation in measured values during long-term continuous measurement. Instead of considering the specification values as guaranteed values on-site, it is necessary to consider how to bridge the gap in conditions.
Four Steps to Reading a Spec Sheet
After understanding the meaning of the indicators, align the conditions before comparing.
- 1 Determine the value to be controlled first – STEP①. First decide whether you want to suppress variation or if you need the correctness of the absolute value. If the former, repeatability is the key indicator; if the latter, linearity and accuracy are the key indicators.
- 2 Check the reference value of ±% of F.S. – STEP②. Linearity is expressed as “±◯% of F.S.” Since F.S. (full scale) refers to the full width of the measurement range, even with the same percentage, the larger the F.S., the larger the actual size of the error.
- 3 Read the measurement conditions for repeatability – STEP③. Check the number of averages, whether it is σ notation or full width, and at what measurement distance the value is. Numbers without specified conditions cannot be compared with other manufacturers’ values as they are.
- 4 Bridge the gap with on-site conditions – STEP④. Check the surface condition of the target, tilting, and ambient temperature fluctuations. If the conditions differ from the assumptions of the specifications, bridge the gap by selecting the head or installation method.
Organizing the Differences Between the Four Indicators
Understand separately what the indicator represents, what it affects, and under what conditions it changes.
| Perspective | Resolution | Repeatability | Linearity | Accuracy |
|---|---|---|---|---|
What it represents |
Resolution
Minimum amount of change that can be identified
|
Repeatability
Variation when repeated under the same conditions
|
Linearity
Deviation from the reference line
|
Accuracy
Overall closeness to the true value
|
When it matters |
Resolution
When you want to detect minute changes
|
Repeatability
When you want to manage variation in pass/fail results
|
Linearity
When you want to manage dimensions with absolute values
|
Accuracy
When comparing values with other measuring instruments
|
Examples of notation |
Resolution
Values in µm or nm units
|
Repeatability
± value or σ notation
|
Linearity
±% of F.S.
|
Accuracy
Often specified individually
|
Precautions when reading |
Resolution
Fine does not mean accurate / Do not confuse with display increments
|
Repeatability
Check the number of averages and whether it is σ or full width
|
Linearity
Check F.S. (= full width of measuring range)
|
Accuracy
Check what is included in the overall value
|
Changes based on measurement distance |
Resolution
Changes depending on the head and distance
|
Repeatability
Changes depending on the distance
|
Linearity
Check the specified range
|
Accuracy
Changes depending on conditions
|
Effect of the number of averages |
Resolution
Increasing the number of averages reduces the readable change
|
Repeatability
The value changes depending on the number of averages
|
Linearity
Almost no effect
|
Accuracy
Changes depending on conditions
|
Effect of the target’s surface condition |
Resolution
Deteriorates when the received light intensity decreases
|
Repeatability
Deteriorates when the received light intensity decreases
|
Linearity
Changes depending on the target
|
Accuracy
Changes depending on the target
|
Effect of temperature changes |
Resolution
Not readily apparent over short periods
|
Repeatability
Not readily apparent over short periods
|
Linearity
Specified at a constant temperature. The temperature component is a separate item.
|
Accuracy
Take temperature characteristics into consideration
|
Align the Conditions Before Comparing the Values
Resolution, repeatability, linearity, and accuracy will result in different values even for the same product if the measurement conditions differ. Because the assumed number of averages, measurement distance, and target differ for each manufacturer, simply comparing the values in the catalog does not establish relative performance. The basic approach is to first decide on the value you want to control, then align the conditions, and finally bridge the gap with the actual site.
Products That Let You Select Specifications Based on Conditions
Since the sensor head can be selected according to the surface condition of the target and the installation distance, the gap between the specifications and the actual site conditions can be bridged at the design stage.
More Than 20 Types of Heads to Choose From According to the Target and Distance — LK-G5000 Series
Ultra High-Speed/High-Accuracy Laser Displacement Sensor
This is an ultra high-speed/high-accuracy laser displacement sensor that uses a reliable and proven triangulation method. More than 20 types of sensor heads are available to choose from according to the surface condition of the target and the installation distance, allowing selection to match the measurement target, such as the micro-measurement type (spot type), rough surface measurement type (wide type), and transparent/specular reflection measurement type (mirror type). Since the values in the specifications change depending on the measurement conditions, selecting the head suitable for the target is the best way to achieve the catalog performance in the field.
The ultra-high-speed sampling of up to 392 kHz not only reliably captures the displacement of moving and rotating targets, but is also effective in reducing variation through averaging. The ABLE II automatic light intensity control function ensures stable measurement values even when the reflection conditions of the target change. Equipped with averaging, filtering, auto-zero, and calibration functions, management can be performed with clearly specified measurement conditions. The mutual interference prevention function supports close installation of multiple heads, and measurement values can be saved using the data storage function, allowing operation while retaining conditions and results.
Obtain Detailed Materials for the LK-G5000 Series
You can download a free catalog summarizing the types, specifications, and measurement conditions of sensor heads.
Inquiries regarding displacement sensor specifications and selection
Examples of Indicators That Are Effective Depending on Measurement Conditions
Even with the same specification sheet, if the target and conditions change, the effective indicators and suitable heads will change.
Vibration Measurement of Ultrasonic Welding Machines
When measuring an object vibrating at high speed, if the sampling cycle is insufficient, the waveform itself will be missed. This is a situation where the conditions for sampling and the number of averages must be checked before the resolution.
Measuring Specular/transparent Objects
Specular and transparent objects reflect light differently from diffuse reflection, and normal heads cannot ensure sufficient received light intensity. Selecting a head that meets the target conditions, which are the premise of the specification values, comes first.
Vehicle Height Measurement Outdoors and Over Long Distances
Over long distances and outdoors, the effects of ambient light and temperature changes appear as fluctuations in measurement values. This is a situation where it is necessary to look at long-term stability rather than short-term variation.
Frequently Asked Questions Regarding the Differences Between Accuracy, Repeatability, and Resolution
Q What is the difference between resolution and accuracy?
A
Resolution represents the smallest amount of change that can be identified, while accuracy represents how close it is to the true value. Even if the resolution is fine, if there are effects from linearity or temperature drift, it will deviate from the true value. Resolution is not a guarantee of accuracy. Note that the "minimum display unit" of the display is the display increment and is a separate item from resolution. Even if the increment is fine, it does not necessarily mean that changes can be read in that unit.
Q What does repeatability represent?
A
This is the variation in measured values when measurements are repeated under the same conditions. However, because the numbers change depending on the number of averages, whether it is expressed in σ or as a full ± width, and at which measurement distance the value is, it must be read in conjunction with the conditions. For our LK-G5000 Series, the values listed are measured using our standard target at the reference distance with 16384 averages.
Q What is F.S. in “±% of F.S.”?
A
F.S. (full scale) refers to the full width of the measurement range of the head. For example, for a head with a measurement range of 20±3 mm, the F.S. is 6 mm. Even with the same ±0.02% of F.S., the larger the F.S. of the head, the larger the actual size of the error. Instead of looking only at the percentage, you should also check the millimeter value specified for the head's F.S.
Q Can products from different manufacturers be compared using only numerical values?
A
No, you cannot. This is because the assumed number of averages, measurement distance, and target conditions differ depending on the manufacturer. It is best to compare them by aligning the items where the conditions are clearly specified, or to evaluate them by creating the same conditions on the actual equipment.
Q Which indicator includes temperature drift?
A
It is not included in any of the four indicators. In many catalogs, it is specified as a separate item called “temperature characteristic”, such as ○% of F.S./°C. This is because it does not appear in short-term variations like repeatability, but rather in the form of measured values slowly changing during long-term continuous measurement. When managing over a long period of time, it is necessary to take this into account by multiplying this temperature characteristic value by the temperature fluctuation range at the site.
Q Which indicator should I look at for my application?
A
If you want to suppress variations in pass/fail judgment, repeatability is key; if you want to manage dimensions with absolute values, linearity and accuracy are key; and if you want to detect minute changes, resolution will be the focus. Deciding on the value you want to manage first before reading the catalog is the shortcut.
Q Are you having trouble reading the specifications?
A
If you tell us your target and the value you want to manage, we will explain the head that meets your conditions and how to think about the specifications. Please contact us from the catalog or inquiry form.
