This lesson is being piloted (Beta version)

Tools for scaling up

Overview

Teaching: 10 min
Exercises: 0 min
Questions
  • How do I turn working code fragments into batch jobs?

  • Where can I look for more help?

Objectives
  • Learn where to go next.

Scaling up

The tools described in these lessons are intended to be used within a script that is scaled up for large datasets.

You could use any of them in an ordinary GRID job (or other batch processor).

However, the Coffea project (documentation) is building a distributed ecosystem that integrates Pythonic analysis with data analysis farms. This is too large of a subject to cover here, but check out the software and join the Coffea user meetings if you’re interested.

Scikit-HEP Resources

and finally

Key Points

  • See Coffea for more about scaling up your analysis.

  • Pythonic high-energy physics is a broad and growing ecosystem.