For discussion at 26 Sept 2011 ATLAS/CMS Statistics Meeting

-------------


            Status of Limit Procedures at the LHC

Bob Cousins, Glen Cowan, Kyle Cranmer, Luc Demortier, Tommaso Dorigo,
Eilam Gross, Louis Lyons, Bill Murray, Alex Read and Ofer Vitells.

For setting limits, the following general points are agreed by both
the ATLAS and CMS Statistics Groups:

The groups recognize the utility and convenience of using a single
baseline method for reporting limits.  To this end, the CLs procedure
has been agreed as a common method, together with Bayesian limits for
those analyses where this has been existing practice.

Beyond the baseline methods, many representatives of the Statistics
Groups believe there is an important benefit in reporting results
based on more than one approach, e.g.,

Frequentist
Bayesian
Likelihood-based

The relative merits of the different alternatives remains a topic of
some debate within the ATLAS and CMS Statistics Groups.  We therefore
propose to continue discussing these alternatives, particularly the
issue of one-sided versus unified frequentist intervals.

There is general agreement that Bayesian limits should be used and
therefore this is emerging as the method with the greatest level of
common support between the two collaborations.  This is already the
method used for some analyses.  For the Bayesian approach to be
recommended more widely, a number of important issues should be
resolved, most crucially the choice of prior probabilities.

It has been argued on both the ATLAS and CMS sides that the constant
prior for a Poisson mean (e.g., cross section) is already widely
established as a benchmark in the community and has certain desirable
properties, such as providing upper limits equal to the frequentist
limits in certain special cases.  Others (also on both the ATLAS and
CMS sides) prefer so-called reference priors.  Priors for other types
of parameters of interest (e.g., SUSY parameters) must also be
considered.  We propose to continue discussing this issue and produce
a common recommendation on a time scale of the next several months.

It is also recommended to investigate likelihood-based methods (i.e.,
minuit/minos), particularly for multi-parameter problems.

These recommendations should not preclude analyses by either
collaboration that employ alternative methods for cross checks.

It is planned to provide more detailed recommendations for
implementing the various methods as soon as possible.