Advanced Python Tutorial

Welcome to the advanced Python tutorials of the starterkit. This lecture covers multiple topics and the notebooks available may fill more than the scheduled lesson. However, they also serve as a knowledge base that one can always come back to lock up things.

Contents:

• 1: Basics
  • Basics
  • Markdown
  • Jupyter
  • Importing modules
• Advanced Python Concepts
  • Packing and unpacking of values
  • Context manager
    • Using yield
  • Using a class
  • Decorators and factories
    • Decorator
  • Exceptions
    • Custom Exception
  • Catching exceptions
    • pitfall “guaranteed execution”
  • Exceptions as control-flow
• Advanced Classes
  • Dunder
  • len
  • str
  • Callable
  • Indexing (iterating)
  • self
• Danger zone
• 2: First look at data
  • Two plotting libraries?
  • Recap: Importing modules
    • 5. The toy dataset
  • Loading data
    • 6. Plotting a simple histogram
  • Adding variables
    • Using rectangular cuts
  • Comparing distributions
• 3: Multivariate Analysis
  • Using a classifier
    • TODO Add a diagram of a decision tree for the above plot
• 4: Extension on Classification
  • Alternative implimentations
  • Feature engineering
  • \(k\)-folding
  • Turn this into a scipt using argparse
• 5: Boosting to Uniformity
  • Loading data
    • Distributions in the Dalitz features for signal and background
    • Preparation of train/test datasets
    • Setting up classifiers, training
    • Let’s look at the results of training
    • ROC curves after training
• Model tuning setup
• Cross-validation
• \(k\)-folding & early stopping
• Hyperameter optimisation
• 6: Histograms
  • Axes
    • Regular
    • Variable
  • Axis Name
    • Compatibility with mplhep
    • Plotting with hist
  • Multiple dimensions
  • Access Bins
  • Getting Density
  • Projecting axes
  • Accessing everything relevant
    • Multi dimensional
  • Arithmetics
  • Weights
• 7: Demonstration of distribution reweighting
  • Downloading data
  • prepare train and test samples
    • Original distributions
  • train part of original distribution
  • test part for target distribution
  • Bins-based reweighting in n dimensions
  • Gradient Boosted Reweighter
    • Comparing some simple expressions:
  • GB-discrimination
    • Great!
    • What did just happen?
  • How to tune
  • Folding reweighter
  • GB discrimination for reweighting rule
• 8: Likelihood inference
  • Scope of this tutorial
  • Getting started
  • Difference of the two spaces
  • Plotting
  • Loss
    • Fixing parameters
• 9: sPlot
  • Simple sPlot example
  • Observed distributions
  • Applying sWeights
    • Compare
  • More complex case
  • Splot
  • Alternative: Known probabilities
    • Building sPlot over mass
    • Of course we don’t have labels which events are signal and which are background beforehand
    • We have no information about real labels
  • Fitting doesn’t give us information about real labels
  • Appying sPlot
  • Using sWeights to reconstruct initial distribution
  • An important requirement of sPlot
  • Derivation of sWeights (optional)
    • Under assumption of linearity:
    • Minimization of variation
    • Uncorrelatedness
  • Conclusion
• 10: Scikit-HEP
  • formulate - converting expressions
  • Particle
  • hepunits
  • Vector
    • Vector properties

Pure Python, advanced

• Notebook 10 starts out with a repetition of the basics of Python.

• In notebook 11, advanced concepts like exceptions, context manager and the factory-pattern together with decorators is introduced.

• 12 is a tutorial about classes, especially on the focus of dunder (__meth__) methods and covers from simpler __len__ and add up to the advanced __getattr__.

Data loading and plotting

Notebook 20 introduced data loading with uproot, Pandas DataFrames as the default container for columnar data that can apply cuts and the plotting libraries.

More on pandas can be found in their excellent documentation or by searching the web

Multivariate Analysis and Machine Learning

These notebooks provide an introduction to more sophisticated cuts using machine learning techniques.

• 30 starts out with the basics of using a BDT and the scikit-learn standard library for this kind of problems and algorithms.

• 31 demonstrates another BDT, XGBoost, which is the state-of-the-art algorithm of a BDT

• 32 contains a tutorial about a special kind of BDTs which actively de-correlate the BDT response from variables. This is often required in physics analysis in order not to bias the inference.

• 33 is a currently outdated intro to Neural Networks

More tutorials for general machine learning algorithms can be found in the scikit-learn tutorial section

Reweighting

Reweighting a distribution can be a useful technique to apply corrections.

• 45 demonstrates two methods of non-parametric reweighting of two distributions in order to correct for MC and data differences. Histogram-based as well as the more powerful, yet harder to control GradientBoostingReweighter are introduced

Statistical Inference

The last step in most analysis is the inference through a likelihood based method. This involves to fit a model to data or toy datasets repeatedly in order to infer the physical parameters that we are interested in but also it’s uncertainty.

• 50 introduces the likelihood model fitting with the zfit library. While the main introduction is pointed towards an actual zfit tutorial, it provides a guide to implement the fit to the data obtained in previous tutorials. hepstats is also introduced as it works with zfit models and is used to estimate the significance of the discovery.

More tutorials on zfit as well as hepstats are available.

sPlot technique

The sPlot - or sWeights - technique is introduced; a technique to statistically subtract the background events in a variable. The technique as well as the library to obtain the weights is demonstrated.

Scikit-HEP

Many libraries that are seen in this tutorial are part of Scikit-HEP, the HEP Python ecosystem. Not all packages have been used though and in tutorial 70, a few smaller, yet useful packages are presented to give an idea of what is available.