Quote:
Originally Posted by HHP
(Post 3779992)
Does averaging work when you know you have an absolute to deal with such as the temperature at which the chemical explodes. You are OK on one side of the variation and vaporized on the other.
In the case of the guitar tuner, you would seem to be working to match an absolute like A440.
|
Constructing a scenario in which measurement failure has critical consequences doesn't change how measurement works. It just requires that it work better if a disaster is to be averted. Bias and consistency are fundamental aspects of measurement. If one considers a single outcome rather than a set of outcomes, all bets are off. Probabilities aren't defined in the single instance (kind of like division by zero isn't meaningful). Single instances have binary outcomes -- 1 or 0. How likely is it that I will die of cancer? Either 1 or 0. I will or I won't. On the other hand, for
people like me, probability estimates are continuous between 1 and 0 with increasing precision as the sample size and appropriateness of the sampling frame increase. Similarly, a single shot is either a hit or a miss. The shooter may have aimed badly but a random error or bias may just exactly compensate so that a hit results. Or the same error sources may result in a miss when the aiming is good. The single instance isn't informative with respect to the measurement (although it can certainly be consequential with respect to the outcome). If one shot hits or misses a target, you can't say anything about the accuracy of the sights or the quality of the gun. You don't know what accounted for the outcome (although some scenarios may be more likely than others because the area of non-target is larger than the area of target). But with repeated observations, you can determine what factors (gun, shooter, their interaction, wind, etc.) are driving the outcome to various extents.
Precision and accuracy, as we are discussing them are not a trade-off. They are independent. A wide scatter or a narrow scatter can both average to either a "true score" or to an error. However, if the focus is on the achievement of an accurate prediction, they are dependent, with precision setting a limit on the likelihood of "success". In the context of reliability/validity, the validity coefficient can't exceed the square root of the reliability coefficient. This has to do with the theoretical relationship of observed scores to true scores.
WARNING -- UNLESS YOU"RE REALLY INTO THIS DISCUSSION, YOU MIGHT WANT TO SKIP THIS NEXT PARAGRAPH:
Reliability, in techno-speak, is the proportion of variance in a set of observations that arises from the state of the entity being measured (e.g., the proportion of variance in observed temperature values that arises from actual differences in temperature and not other factors, such as quirks of the measurement tool). If the proportion of variance between an indicator and a true score is .64, the correlation between the true score and the indicator is the square root of .64, or .80. If validity is established by means of camparing the indicator being evaluated to another indicator of perfect reliability (the best case), the latter would have a correlation with the true score of 1.0. Thus, the maximum correlation possible between the measurement and the infallible indicator is the product of the two indicators' correlations with the true score, or 1.0 x .80 = .80, the square root of the reliability. So, the greater the reliability, the higher the possible validity. Note that, in this context, validity is defined in terms of correlation -- the ability to predict the true score. It doesn't imply that the measurement obtained is the same as the true score (think of predicting Fahrenheit temperature from a Celsius thermometer -- excellent prediction even though the numbers are different). The discrepancy between the estimate and the true score is not measurement error but miscalibration. The predictive power isn't compromised by miscalibration but without correcting the miscalibration, the estimate will be off the mark (in completely predictable, nonrandom, ways).