For the estimation of many parameters, borrowing statistical strength across related observations via hierarchical modeling has been one of the most powerful statistical developments over the past 60 years. The phenomenon is especially evident in classical frameworks where dominating ensemble information shrinks noisy parameter estimates to an overall mean. This is exactly what happened with Medicare's Hospital Compare random effects model, which asserted that 99.5% of hospital mortality rates for acute myocardial infarction (AMI) were ''no different than the U.S. national rate''. In this talk we shall see that their conclusions stemmed from seemingly innocuous, though controversial, assumptions about ensemble means and variances that were at odds with the data. As an alternative, we propose hierarchical random effects models with flexible prior structure that emancipate the means and variances and yield dramatically different conclusions. The superior calibration of our models is demonstrated with comparisons based on predictive Bayes factors and predictive matched samples. Finally, direct rather than indirect standardization is seen to be superior for public reporting.
This paper begins by presenting a simple model of the way in which experts estimate probabilities. The model is then used to construct a likelihood-based aggregation formula for combining multiple probability forecasts. The resulting aggregator has a simple analytical form that depends on a single, easily-interpretable parameter. This makes it computationally simple, attractive for further development, and robust against overfitting. Based on a large-scale dataset in which over 1300 experts tried to predict 69 geopolitical events, our aggregator is found to be superior to several widely-used aggregation algorithms.