Benchmarks (protopipe.benchmarks)#

The protopipe.benchmarks module is composed of 3 parts:

  • notebooks, a folder containing benchmarking Jupyter notebooks divided by analysis stage,

  • operations, a sub-module containing functions to perform many different operations on data related to benchmarking

  • plot, a sub-module containing plotting functions

  • utils, a sub-module containing utility functions for the notebooks

  • book_template, a folder containing the Jupyter Book template for a CTA analysis


Much of what is contained in the sub-modules is the product of a long refactoring process of old material from the notebooks. Many things can be improved or imported by ctaplot/cta-benchmarks and ctapipe as the refactoring of the pipeline takes progress. Also, not all notebooks are exatcly the same in terms of global options, a notebook template will be added.

All benchmarks can be launched by means of the protopipe-BENCHMARK script (Benchmarking). This is the recommended method, as it integrates with the rest of the analysis interface.

API reference#

protopipe.benchmarks Package#


add_BTEL1010_weigths_to_data(data[, subarray])

Update data table with weights from requirement B-TEL-1010-Intensity-Resolution.

add_stats(data, ax[, x, y, fontsize])

Add a textbox containing statistical information.

average_bias_of_charge_resolution(...[, ...])

Calculate the average bias of charge resolution.

calculate_RMS_around_1(values, weights)

Root Mean Square around 1 as proposed from comparison with CTA-MARS.

compute_bias(x_bin_edges, reco, true[, ...])

Compute bias as a binned statistic.

compute_psf(data, ebins, radius)

compute_resolution(x_bin_edges, reco, true)

Compute a resolution as a binned statistic.

compute_weight_BTEL1010(true_energy[, ...])

Compute the weight from requirement B-TEL-1010-Intensity-Resolution.


Returns a function f(x,y) that evaluates the lookup at a point.

get_evt_subarray_model_output(data[, ...])

Returns DataStore with keepcols + score/target columns of model at the level-subarray-event.

get_fig_size([ratio, scale])

Get size of figure given a ratio and a scale.

get_single_pixels_spectrum(x, bins, ...)

Log-Log data for single pixels spectrum.

load_tel_id([file_name, tel_id])

Load R0 and R1 waveforms for 1 telescope.

plot_DL1a_reco_true_correlation(ax, reco, true)

Plot the correlation between reconstructed and true number of photoelectrons.

plot_background_rate(input_file, ax, label, ...)

plot_bias(ax, x, bias, **opt)

Plot a bias.

plot_binned_mean(x, y[, bins, yerr, ax])

Plot mean of y in each bin in x with standard deviation as errorbars.

plot_binned_median(x, y[, bins, yerr, ax])

Plot median of y in each bin in x with the central 1-sigma interval as errorbars.

plot_distributions(suptitle, feature_list, ...)

Plot feature distributions for several data set.

plot_evt_roc_curve_variation(ax, data_test, ...)


plot_hist(ax, data, nbin, limit[, norm, ...])

Utility function to plot histogram

plot_profile(ax, data, xcol, ycol, n_xbin, ...)

Plot a profiled histogram.

plot_psf(ax, x, y, err, **kwargs)

plot_resolution(ax, x, resolution[, label, fmt])

Plot a resolution.

plot_roc_curve(ax, model_output, y, **kwargs)

Plot ROC curve for a given set of model outputs and labels

plot_sensitivity_from_pyirf(ax, data[, ...])

Produce a sensitivity plot.

prepare_requirements(input_directory, site, ...)

Prepare requirements data as a dictionary.


Raise an exception as a statement.


Convert True/False strings to booleans.




Class producing diagnostic plot for the BDT method

ClassifierDiagnostic(model, ...[, ...])

Class to plot several diagnostic plot for classification.

ModelDiagnostic(model, feature_name_list, ...)

Base class for model diagnostics.


Class to dynamically compute one-dimensional binned statistics.

RegressorDiagnostic(model, ...)

Class to plot several diagnostic plots for regression.

Class Inheritance Diagram#

Inheritance diagram of protopipe.benchmarks.plot.BoostedDecisionTreeDiagnostic, protopipe.benchmarks.plot.ClassifierDiagnostic, protopipe.benchmarks.plot.ModelDiagnostic, protopipe.benchmarks.operations.OnlineBinnedStats, protopipe.benchmarks.plot.RegressorDiagnostic