Experienced radiologists who evaluated chest x-rays as "normal" or "abnormal" contradict themselves 20% of the time when they see the same picture on separate occasions. A study of 101 independent audits revealed a similar degree of inconsistency. A review of 41 separate studies of the reliability of judgments made by auditors, pathologists, psychologists, organizational (?), and other professionals suggest that this level of inconsistency is typical even after a case is reevaluated within a few minutes. Unreliable judgments cannot be valid predictors of anything.
Quotes are taken from the research journals of Kyle Vanderburg.