How we didn’t find a new particle… Part 1 : Explaining the Analysis

11 min readOct 15, 2020

This three part series will explain how a very specific type of supersymmetry was not found (yet! [or yet again!] ) by the ATLAS experiment on LHC at CERN.

In 2019 the ATLAS collaboration published a paper¹ on searches for pair-produced sbottom quarks (supersymmetric partner of the bottom quark). Soon after the original publication of the analysis, ATLAS also published a supplementary note² — detailing how the mathematical Likelihood of the analysis is built and how it can be reproduced.

The ATLAS detector is an experiment designed to record and analyse the collisions from the Large Hadron Collider (LHC) at CERN. Famous for the discovery of the Higgs boson.

From 2013 to 2020 I worked on ATLAS as a PhD student and then as a Research Assistant, and I was on the analysis team that designed and performed this analysis

Note: ATLAS has ~3000 physicists, plus 1000s of support staff plus. Analysis Teams stand on the top of a pyramid of work performed by others in the collaboration.

TL;DR — we didn’t find any “statistically significant” evidence of any new particles in this analysis.

For those who want to continue to learn why “no significant excess” was observed, this article series will:

Show you how to obtain the published analysis Likelihood (Part 1)
Give a brief overview of what particles we were looking for, and how we looked for them. (Part 1)
With some hands on examples, show you how we handled uncertainties and calculated the statistical significance of our predictions and observations using Maximised Likelihood Estimation. (Part 2)
How we interpreted the results and what this says about Supersymmetry (Part 3)

Obtaining the published likelihood

Before we get started describing the analysis, we can download this likelihood, it will help to explain various concepts.

Starting from the public HEP-data record:

HEPData

The result of a search for the pair production of the lightest supersymmetric partner of the bottom quark…

www.hepdata.net

Find the ‘resources’ button, then ‘Common Resources’ and navigate to and download the gz file for ‘Archive of full likelihoods in the HistFactory JSON format’

If we extract and inspect the contents, we see a useful README.md file, and three subfolders name RegionA , RegionB and RegionC .

The three regions correspond to the three separate likelihood fits the analysis performed.

These articles will focus on `RegionC`, which I think is the most interesting.

Each Region subfolder contains two json files:

BkgOnly.json — contains the full likelihood assuming no new physics (background)
patchset.json — contains a series of patches to the likelihood in the scenario of the presence of new physics (signal), with a variety of different parameters

.
├── README.md
├── RegionA
│   ├── BkgOnly.json
│   └── patchset.json
├── RegionB
│   ├── BkgOnly.json
│   └── patchset.json
└── RegionC
    ├── BkgOnly.json
    ├── patchset.json

Inspecting a BkgOnly.json file you will see they have the following structure:

dict_keys(['channels', 'measurements', 'observations', 'version'])

Channels: predictions of the data with uncertainties, in orthogonal areas of phase-space.
Measurements: what is being measured in the various channels — the luminosity? The strength of the signal?
Observations: the actual observations of data made in the various channels.
Version (irrelevant): simply the published version, will be 1 unless the analysis has been revised due to a mistake.

This might be a bit jargon-heavy, hopefully by the end you will understand these concepts.

Get pyhf…

To help explore the likelihood json and later perform the statistical analysis preserved within it, we will use pyhf

pyhf has some fantastic documentation and examples. As described on pyhf’s scikit page, in short, it is a…

…pure-python implementation of that statistical model for multi-bin histogram-based analysis and its interval estimation is based on the asymptotic formulas of “Asymptotic formulae for likelihood-based tests of new physics” [arXiv:1007.1727]`

pyhf can be easily installed via pip:

pip install pyhf

it can be installed with a variety of backends, e.g. tensorflow, if you want to experiment…

In a simple example using pyhf, we can load one of the BkgOnly.json files, and using the workspace class, begin to inspect the likelihood:

For RegionC this will output the following:

channels: ['CRtt_cuts', 'CRz_cuts', 'SR_metsigST', 'VRtt_cuts', 'VRzAlt_cuts', 'VRz_cuts']observations: {'CRtt_cuts': [143.0], 'SR_metsigST': [28.0, 12.0, 4.0, 3.0], 'VRzAlt_cuts': [242.0], 'CRz_cuts': [73.0], 'VRtt_cuts': [118.0], 'VRz_cuts': [141.0]}

In the case of RegionC , we see five channels (orthogonal areas of phase-space):

Two Control Regions (CRs)
— single bins of data where 143 and 73 events were observed
Three Validation Region (VRs)
— single bins of data observing 242 , 141 and118 events
One Signal Region (SR)
— four bins of data: [28, 12, 4, 3]

These values correspond to the values of the data points that can be seen in Figures 5&6 of the original paper¹.

We can see these values of [28, 12, 4, 3] correspond to what is called SRC22 — SRC28 in the paper.

In SRC22 there is an excess of something close to one sigma.

But how do we quantify any excess and know it’s not significant? What does this say about supersymmetry? What are the CRs/VRs and SRs? What are these colourful histograms?

We have to step back to explain the physics before we can answer these questions...

What were we looking for?

Since this is a medium article, not a theoretical nor a phenomenological physics paper, in short - we looked in our data for evidence of this Feynman-diagram-like signature…

Proton-proton collisions (p-p) could be high enough energy to produce heavy supersymmetric bottom (sbottom) quarks — the red b’s with a tilde sign above.

yes, really —for supersymmetric quarks an ‘s’ is just stuck in front of the name — because they are “scalar”.

Sbottoms almost immediately decay — in this scenario — to b-quarks and the “neutralino-2” (another theorised supersymmetric particle). These neutralinos could decay into a lighter, stable neutralino (possibly the source of Dark Matter) and a Higgs boson.

What we looked for is data events with this signature.

So, what does this signature look like in a detector?

Higgs bosons decay into pairs of b-quarks (actually, b-quark-anti-b-quark pairs) the majority of the time.

In proton collisions produce b-quarks, they quickly hadronise and shower energy deposits in the ATLAS detector material surrounding the collision point. This is recorded and reconstructed into objects called b-jets.

The stable neutralinos (Dark Matter) are undetectable. There are known particles that are also not directly detectable (by ATLAS), such as neutrinos.

However, due to conservation of momentum in the transverse plane, we are able to know if there was a significant amount of missing-transverse-energy.

The momentum in the x-y (transverse) plane before a proton-proton collision is zero. All the incoming momentum is in the z-direction (the direction of the beam containing protons travelling at >99% the speed of light towards each other). After the collision the sum of the momentum in the x-y plane must still be zero, therefore an imbalance caused by invisible particles is easy to see !

Below, an event display of the transverse plane shows another physics process that can create b-jets and missing transverse energy — a candidate HZ event. This illustrates what is physically going on in the detector:

Proton-proton collisions produce a Higgs Boson and a Z Boson
The Higgs decays to an b-quark and an anti-b-quark
→ the b-quarks hadronise creating a shower of particles picked up as electronic signals by the detector (yellow and teal-green energy calorimeter deposits)
The Z boson decays to a neutrino and an anti-neutrino
→ these are not detectable
→ but a dashed line is drawn from the momentum imbalance in this plane, which is the signature of where the neutrinos went

Transverse view of a simulated pp -> HZ collision (left). Schematic diagram of the ATLAS detector (right)

the above example is for two b-jets and missing transverse energy.

In this analysis the signature is — six b-jets and large missing transverse energy.

This is a pretty unique signature. Yet, from the billions and billions of recorded events, totalling 100s of PBs of data and simulations which were recorded between 2015–2018 — there are still millions of events to be analysed after selecting this signature. There are multiple known Standard Model processes that can create this signature.

When searching for new physics, we want to see an excess of data above our predictions from simulating these processes (coloured histograms).

The main background in this analysis is the production of a Z-boson that decays into neutrinos (undetectable) in association with the production of b-jets from QCD interactions (yellow histogram, labelled Z+jets).

To discover new particles, we don’t want the data points to follow the background (red line, SM Total), we want it to follow the background + signal (dashed pink histogram). Sadly, the data follows the SM Total in the SRC — though there are some deviations, for which, in the next part we will assess how statistically significant these are.

SR example — you expect to observe 40 data events if there is no supersymmetry and expect to observe 70 data events if there is supersymmetry.

Signal Regions (SRs) are designed by applying cuts on kinematics variables, such as the significance of the missing transverse energy, to define a region of phase-space where the expected signal sticks out above the background.

The significance of the missing transverse energy is approximated to be the magnitude of the missing transverse energy divide by the scalar sum of all visible transverse energy deposits. The “object” based definition is similar but takes into account resolutions of the energy deposits, a description of this can be found here.

Focusing on the RegionC — If we have a signal region with four bins in this variable Object Based missing transverse energy significance, we have cut away billions and billions and billions of data events, approximately 1 billion colliding per second when running the LHC between 2015–2018 … and ended up with the [28, 12, 4, 3] found in the json and printed out by pyhf . This totals 47 events in the RegionC SR.

This is the area of phase-space we were expecting to see new physics, which is an unbelievably small when considering how many collisions were made by the LHC — billions per second for 3 years and we used 47 in total — a needle in a haystack of all hay there ever was. Of course… the other events are used by other analyses, looking for other types of new physics and making measurements!

Using pyhf and pandas we can inspect the workspace and build a DataFrame for the SRC to show the total data, the total raw prediction and the expected number of events from various known SM processes:

Which produces the following table:

We can also take a look at some of the signal samples using the patch file. The names of all signals can be listed with:

$ jq -r ".patches[].metadata.name" RegionC/patchset.json 
...
sbottom_900_850_60
sbottom_900_895_60
sbottom_950_230_100

The three numbers correspond to the mass (given in units of GeV) of the sbottom, the mass of the neutralino-2 and the mass of the neutralino. These are unknown parameters in the model, hence they are scanned over by building a signal grid.

Patching of example signal onto the BkgOnly.json creates a new json file containing the background + signal:

$ jsonpatch RegionC/BkgOnly.json <(pyhf patchset extract RegionC/patchset.json --name "sbottom_1200_1150_60") > RegionC/sbottom_1200_1150_60.json

Using the same code on this new file instead, we get the table:

We see that in the SRC supersymmetry existed with a sbottom of mass 1200 GeV, a neutralino-2 of 1150 GeV and a neutralino of 60 GeV — assuming particular decays of the sbottom and neutralino-2 — then we’d expect to see ~27 events in the SR.

In this particular signal+background hypothesis we would have 47 data observed with 68 events expected, which is clearly less consistent with the background-only hypothesis of 41 expected events.

Before analysing these numbers, the main backgrounds, e.g. Z+jets have to be normalised to the data in Control Regions (CRs) in a region of phase-space where there is no expected signal.

This is because the default cross-section (how many events from this process we expect given the amount of collisions) has been measured in a much looser area of phase-space and we cannot fully trust this in this tighter area of phase-space, it needs to be corrected.

This is extremely important, if a background was being under-predicted (or over-predicted), due to an inaccurate normalisation, this might appear as a large excess in the signal region (or as a lack of excess).

Using channel_list = ['CRz_cuts'] in the previous code, we can also check the CRZ totals:

This suggests that the Z+jets is slightly over-estimated, as there is no signal here, the SM Total should be 73.

For the process Z+jets, the Z-boson can also decay into electrons and muons, particles which are very easily detected by ATLAS. The expected signal does not decay into these particles. Therefore this CR is defined by taking electron and muon events (2L) and mimicking the kinematics selections of the SR pretending that the electron-pair (or muon-pair) was not detected.

Chopping up of the Missing Energy Significance and Number of Leptons phase space, in order to create orthogonal regions for Control, Validation and Signal.

Validation regions can also be defined in phase space, with no electrons or muons (0L), inverting a kinematics selection, to validate that the normalisation is approximately correct.

If you are interested, please read the original paper¹ for more detail

Stay tuned for Part Two and Part Three of the series where we will used pyhf to fit the likelihood contained within the workspace, in order to answer:

How did this analysis quantify the significance of any difference in data and predictions under the hypotheses of background-only and signal?
If no big excess was found, does this mean supersymmetry doesn’t exist? (spoiler: absolutely not)

[1] ATLAS. Search for bottom-squark pair production with the ATLAS detector in final states containing Higgs bosons, b-jets and missing transverse momentum. J. High Energ. Phys. 2019, 60 (2019). https://link.springer.com/article/10.1007/JHEP12(2019)060

[2] ATLAS. Reproducing searches for new physics with the ATLAS
experiment through publication of full statistical
likelihoods. ATL-PHYS-PUB-2019–029, 2019. https://cds.cern.ch/record/2684863