dynadojo.challenges.Challenge#

class dynadojo.challenges.Challenge#

Bases: object

Challenge class used to benchmark algorithms and systems.

Methods

__init__(N, L, E, t, ...[, system_kwargs, ...])

Initialize the class.

evaluate(algo_cls[, algo_kwargs, ...])

Evaluates an algorithm class (NOT an instance) on a dynamical system over a set of experimental parameters.

system_run(rep_id, latent_dim, embed_dim, ...)

For a given system latent dimension and embedding dimension, instantiates system and for a specific N, evaluates the algorithm on the system.
__init__(N, L, E, t, max_control_cost_per_dim, control_horizons, system_cls, reps, test_examples, test_timesteps, system_kwargs=None, verbose=True)#

Initialize the class.

Parameters:

  • N (list of int) – Train sizes, (# of trajectories).

  • L (list of int) – Latent dimensions.

  • E (int or list or None, optional) – Embedded dimensions. If list, then evaluate iterates across embedded dimensions. (e >= l) If int, then evaluate uses a fixed embedded dimension. (E >= max(L)) If None, then evaluate sets the embedded dimension equal to the latent dimension. (e = l)

  • t (int) – Timesteps (length of a trajectory).

  • max_control_cost_per_dim (int) – Max control cost per control trajectory.

  • control_horizons (int) – Number of times to generate training data with control.

  • system_cls (type[AbstractSystem]) – Class constructor (NOT instance) for a concrete system.

  • reps (int) – Number of times to repeat each experiment.

  • test_examples (int) – Test size.

  • test_timesteps (int) – Test timesteps.

  • system_kwargs (dict or None, optional) – Additional keyword arguments for the system constructor.

  • verbose (bool, optional) – Whether to print verbose output. Defaults to True.

evaluate(algo_cls, algo_kwargs=None, fit_kwargs=None, act_kwargs=None, ood=False, noisy=False, id=None, num_parallel_cpu=-1, seed=None, reps_filter=None, L_filter=None, rep_l_filter=None)#

Evaluates an algorithm class (NOT an instance) on a dynamical system over a set of experimental parameters.

Parameters:

  • algo_cls (type[AbstractAlgorithm]) – Algorithm class to be evaluated.

  • algo_kwargs (dict or None, optional) – Keyword arguments to be passed to algo_cls.

  • fit_kwargs (dict or None, optional) – Keyword arguments to be passed to algo_cls.fit.

  • act_kwargs (dict or None, optional) – Keyword arguments to be passed to algo_cls.act.

  • ood (bool, optional) – If True, also test on out-distribution initial conditions for the test set. (For FixedError, search is performed on ood_error if ood=True.) Defaults to False. If False, generate out-of-distribution initial conditions for the test set.

  • noisy (bool, optional) – If True, add noise to the train set. Defaults to False. If False, no noise is added.

  • id (int or None, optional) – Algorithm ID associated with evaluation results in the returned DataFrame.

  • num_parallel_cpu (int, optional) – Number of CPUs to use in parallel. Defaults to -1, which uses all available CPUs.

  • seed (int or None, optional) – Seed to seed the random number generator for seeding systems and algorithms. Defaults to None. Is overridden by seeds in system_kwargs or algo_kwargs.

  • reps_filter (list of int or None, optional) – If provided, will only evaluate system_runs with the given rep_ids. Defaults to None, which evaluates all repetitions.

  • L_filter (list of int or None, optional) – If provided, will only evaluate system_runs with the given latent dimensions. Defaults to None, which evaluates all latent dimensions.

  • rep_l_filter (list of tuple(int, int) or None, optional) – If provided, will only evaluate system_runs with the given (rep_id, latent_dim) pairs. Defaults to None, which evaluates all (rep_id, latent_dim) pairs.

Returns:

A DataFrame where each row is the result of an algorithm trained and evaluated on a single system.

Return type:

pandas.DataFrame

system_run(rep_id, latent_dim, embed_dim, algo_cls, algo_kwargs=None, fit_kwargs=None, act_kwargs=None, noisy=False, test_ood=False, system_seed=None, algo_seed=None, **kwargs)#

For a given system latent dimension and embedding dimension, instantiates system and for a specific N, evaluates the algorithm on the system. Across runs, the algorithm is re-initialized with the same seed.

Note

The seed in algo_kwargs and system_seed in system_kwargs takes precedence over the seed passed to this function.

Parameters:

  • algo_cls (type[AbstractAlgorithm]) –

  • algo_kwargs (dict | None) –

  • fit_kwargs (dict | None) –

  • act_kwargs (dict | None) –

  • noisy (bool) –

  • test_ood (bool) –