Machine Vision Judging Conformity, but No MSA Done? —— Five Steps for MSA of Automated Measurement Systems

A certain automotive parts company installed an online visual inspection system last year: after stamping parts are off the line, they are automatically photographed, compared, and judged, with nonconforming products removed by a robotic arm, and data fed into the SPC kanban in real time. The planner's favorite line is, "This equipment can inspect 120 pieces per minute, more accurately than four inspectors combined." During the annual customer audit, the auditor asked just one question: "Please show the measurement system analysis records for this equipment." What was found on-site was a GR&R report from three years ago, during the manual inspection era—where the "evaluator" column was filled with inspector names, and the new equipment had no human judgment, only an algorithm running. The nonconformity issued by the auditor was straightforward: the company had not conducted a measurement system analysis for the automated inspection equipment used for product release.

A more common scenario is: during equipment acceptance, a single repeatability test was conducted, where the same standard part was measured ten times, and the data was almost identical. The conclusion was, "The equipment has high precision, MSA is qualified." This conclusion does not hold. It only proves that the equipment is very stable in the same position and at the same moment, but it does not prove that the measured data represents the true state of the part, nor does it prove that it can truly identify nonconforming products.

1. What Should Be Filled in the "Evaluator" Column for Automated Equipment?

In manual measurement systems, variation is divided into two parts: repeatability (same person, same part, multiple measurements) and reproducibility (between different people). Automated equipment has no "people," so many simply force the form, filling the "evaluator" column with shift, equipment number, or three operators "responsible for loading and unloading," to create a seemingly complete report.

In reality, the sources of variation for an automated measurement system are different and need to be re-identified:

• Fixture Positioning and Clamping: The same part placed in the fixture three times with a positional difference of 0.02mm results in a measurement difference of 0.02mm.
• Part Orientation: Visual inspection is extremely sensitive to direction, angle, and occlusion. A part flipped 180° may yield completely different results.
• Environment: Light decay, lens contamination, temperature drift, and vibration, which are partially mitigated by the human eye in manual measurements, are entirely reflected in the data for automated equipment.
• Algorithm and Thresholds: Tightening the grayscale threshold changes the judgment results, and algorithm changes often lack version records.
• Calibration Status: Expired calibration, worn calibration blocks, and the equipment itself does not issue alerts.

Therefore, MSA for automated equipment is not about "copying a GR&R," but rather designing experiments based on the new variation structure.

2. Equipment Does Not Equal Measurement System: Define the Boundaries First

The first step in MSA is not calculation, but defining the object being measured. The boundaries of an automated measurement system should be listed in a checklist: the equipment itself (camera, lens, light source, sensor), fixture and positioning mechanism, part orientation and loading/unloading method, software algorithm and threshold parameters, calibration and daily inspection schedule, environmental conditions, and personnel (calibration personnel, personnel for rechecking and release). After the list is complete, ask one more question: which product characteristics are determined by the data output from this equipment?

If the boundaries and purposes are unclear, all subsequent analysis conclusions will lack a clear归属—when the indicators exceed limits, it's unclear whether to change the lens, the positioning, or the threshold.

3. Numerical Type: Break Down GR&R into Three Variations

Automated measurement equipment can also undergo GR&R, but the "evaluator" factor is replaced by three types of variation:

Repeatability (Equipment Variation): The same standard part is measured continuously 25 to 30 times without moving, to examine the standard deviation and range, reflecting the stability of the equipment itself.

Position Variation: The same part is re-measured in three different orientations or positions, to check if the results are consistent. This is a unique and often overlooked aspect of automated equipment.

Part-to-Part Variation: Select 10 samples covering the tolerance range (including those close to the upper and lower limits), and each sample is measured 2 to 3 times.

The criteria for %GRR remain the same: less than 10% is acceptable, 10% to 30% can be conditionally used (considering the importance of the characteristic and product risk), and greater than 30% cannot be used for key characteristic determination and release.

Here's a trap: the repeatability of automated equipment is often very small, making the %GRR look very good. However, if the samples are only selected from conforming products and the tolerance range is not fully covered, the part-to-part variation is suppressed, and this percentage loses its meaning. The samples must cover the actual process variation range, which is more critical to the conclusion than the equipment itself.

4. Attribute Type: Miss Rate and Kappa Are Key

For visual inspection and automatic pass/fail judgment, the output is OK/NG, not numerical. Conducting GR&R for such equipment is meaningless; instead, the focus should be on verifying the detection capability using a known defect sample set. The core indicators are three:

• Miss Rate: The rate at which nonconforming products are judged as conforming. This is the most critical indicator, as missed nonconformities will flow directly to the customer. The sample set must include critical defects—those close to the judgment threshold, minor but detectable scratches, and parts with dimensions just at the edge.
• False Rejection Rate: The rate at which conforming products are judged as nonconforming. While it does not harm the customer, it leads to manufacturing scrap, rework, and production stoppages, and target values must be set.
• Consistency: Compare with standard judgments (or experienced inspectors) using Kappa. A Kappa value below 0.75 indicates insufficient judgment consistency, requiring adjustment of algorithm thresholds or additional manual rechecks.

The sample set is not a one-time prop. It should be stratified by defect type and severity and regularly re-verified. After changing the light source, lens, threshold, upgrading software, or moving or overhauling the equipment, the test must be redone—these changes often have a more direct impact on the visual system than the wear of the equipment itself.

5. Five Practical Steps

Step 1: Define Boundaries and Establish Records. List the components, purposes, and involved product characteristics of the measurement system according to the checklist in Section 2, clearly identifying the equipment number, key parameter versions, and calibration status.

Step 2: Determine Characteristics and Criteria. First, distinguish which characteristics are judged numerically and which are judged by attributes, then set the criteria: for numerical types, define the %GRR threshold and sample coverage requirements; for attribute types, set the miss rate, false rejection rate targets, and Kappa lower limit. The criteria should be written into the control plan, not just stored on the quality engineer's computer.

Step 3: Sampling and Implementation. For numerical types, prepare 1 standard part plus 10 samples covering the tolerance range; for attribute types, establish a golden sample set by defect type, including conforming and nonconforming samples, and intentionally include critical defects. Measurements must be taken under normal production conditions and according to daily operating procedures, without temporary parameter adjustments or changes in orientation.

Step 4: Judgment and Improvement. If the indicators exceed limits, do not rush to buy new equipment. Instead, break down the variations: if repeatability is high, check positioning, clamping, and light stability; if position variation is high, modify the fixture or fix the orientation; if part-to-part variation is too small, it indicates that the samples do not represent the process, and new samples should be taken and retested. Identifying the main source of variation before making changes is much cheaper than replacing the equipment.

Step 5: Regularization. Once automated equipment is online, it often runs unattended for years, making it easy to become a black box after two years. Write three tasks into the equipment management procedures: daily inspection (measure with a standard part once daily and record, draw trend charts if necessary), periodic verification (re-verify with the golden sample set), and retesting triggered by changes (software, light source, lens, calibration, relocation), and retain all original records and versions.

6. Three Common Pitfalls

First, forcing manual GR&R forms onto automated equipment and hard-fitting "evaluators." The report may look complete, but the actual sources of variation are not the real sources of variation for the equipment.

Second, conducting MSA only once during acceptance. The equipment is in its best state on the day of acceptance, and subsequent issues such as lens contamination, light source decay, and algorithm upgrades have not been verified, meaning the equipment only provides a number, not evidence.

Third, focusing solely on the %GRR number and ignoring the miss rate. A numerical device with a good number does not necessarily mean reliable release; an attribute device with excellent repeatability, if its detection capability has not been verified, will still allow batches of abnormal parts to flow out.

The credibility of an automated measurement system does not automatically increase just because "no human is involved." It requires you to re-ask the variation sources in the language of the equipment: does this equipment measure the true state of the part, and does it correctly identify the items you truly do not want?

Equipment does not become automatically credible just because it is fully automated—MSA is the question that clarifies its variation structure and detection capability.

Knowledge code: 6.2.1

Version: v20260918

Author: QTank QTank is dedicated to providing systematic professional knowledge, methodologies, and practical tools for quality management practitioners, helping companies continuously improve their quality capabilities.