dynadojo.challenges.FixedError#

class dynadojo.challenges.FixedError#

Bases: Challenge

Challenge where the target error is fixed and the latent dimensionality is varied and the number of training samples to achieve the error is measured. Performs a binary search over the number of training samples to find the minimum number of samples needed to achieve the target error rate.

Methods

__init__(L, t, max_control_cost_per_dim, ...)

param L:

List of latent dimensions to test.

evaluate(algo_cls[, algo_kwargs, ...])

Evaluates a model class (NOT an instance) on a dynamical system over a set of experimental parameters.

plot(data[, target_error, show, show_stats])

system_run(rep_id, latent_dim, embed_dim, ...)

For a given system latent dimension and embedding dimension, instantiates system and evaluates a single trial of finding a number of training examples that achieves the target error.
__init__(L, t, max_control_cost_per_dim, control_horizons, system_cls, reps, test_examples, test_timesteps, target_error, E=None, system_kwargs=None, n_precision=0, n_starts=None, n_window=0, n_max=10000, n_min=1, n_window_density=1.0, verbose=True)#
Parameters:

  • L (list) – List of latent dimensions to test.

  • t (int) – Number of timesteps of each training trajectory.

  • max_control_cost_per_dim (int) – Maximum control cost per dimension.

  • control_horizons (int) – Number of control horizons to test.

  • system_cls (type) – System class to use.

  • reps (int) – Number of repetitions to run for each latent dimension.

  • test_examples (int) – Number of test examples to use.

  • test_timesteps (int) – Number of timesteps of each test trajectory.

  • target_error (float) – Target error to test.

  • E (list, optional) – List of embedding dimensions to test. If None, defaults to L.

  • system_kwargs (dict) – Keyword arguments to pass to the system class.

  • n_precision (float) – Uncertainty interval around number of training samples to achieve target error. If 0, we find the exact minimum number of samples needed to achieve the target error rate.

  • n_window (int) – Number of n to smooth over on left and right when calculating the error rate during search. If 0, no averaging is done.

  • n_starts (list, optional) – List of starting points for the binary search over the number of training samples for each latent dim in L. len must equal len(L). If None, defaults to 1.

  • n_max (int) – Maximum number of training samples to use.

  • n_min (int) – Minimum number of training samples to use.

  • n_window_density (float) – Density of n to test during smoothing. 1 = test every n. 0.5 = test every other n. 0.25 = test every fourth n. etc.

  • verbose (bool) –

evaluate(algo_cls, algo_kwargs=None, fit_kwargs=None, act_kwargs=None, ood=False, noisy=False, id=None, num_parallel_cpu=-1, seed=None, reps_filter=None, L_filter=None, rep_l_filter=None)#

Evaluates a model class (NOT an instance) on a dynamical system over a set of experimental parameters.

Parameters:

  • algo_cls (type) – Model class to be evaluated.

  • algo_kwargs (dict) – Keyword arguments to be passed to algo_cls.

  • fit_kwargs (dict) – Keyword arguments to be passed to algo_cls.fit.

  • act_kwargs (dict) – Keyword arguments to be passed to algo_cls.act.

  • ood (bool, optional) – If True, also test on out-distribution initial conditions for the test set. (For FixedError, search is performed on ood_error if ood=True.) Defaults to False. If False, generate out-of-distribution initial conditions for the test set.

  • noisy (bool, optional) – If True, add noise to the train set. Defaults to False. If False, no noise is added.

  • id (str) – Model ID associated with evaluation results in the returned DataFrame.

  • num_parallel_cpu (int, optional) – Number of CPUs to use in parallel. Defaults to -1, which uses all available CPUs.

  • seed (int, optional) – Seed to initialize the random number generator for seeding systems and models. Defaults to None. Is overridden by seeds in system_kwargs or model_kwargs.

  • reps_filter (list, optional) – If provided, will only evaluate system_runs with the given rep_ids. Defaults to None, which evaluates all repetitions.

  • L_filter (list, optional) – If provided, will only evaluate system_runs with the given latent dimensions. Defaults to None, which evaluates all latent dimensions.

  • rep_l_filter (list, optional) – If provided, will only evaluate system_runs with the given (rep_id, latent_dim) pairs. Defaults to None, which evaluates all (rep_id, latent_dim) pairs.

Returns:

A DataFrame where each row is a result from an algorithm trained and evaluated on a single system.

Return type:

pandas.DataFrame

system_run(rep_id, latent_dim, embed_dim, algo_cls, algo_kwargs=None, fit_kwargs=None, act_kwargs=None, noisy=False, test_ood=False, system_seed=None, model_seed=None)#

For a given system latent dimension and embedding dimension, instantiates system and evaluates a single trial of finding a number of training examples that achieves the target error.

Note

The model seed in model_kwargs and system_seed in system_kwargs takes precedence over the seed passed to this function.

Parameters:

  • algo_cls (type[AbstractAlgorithm]) –

  • algo_kwargs (dict | None) –

  • fit_kwargs (dict | None) –

  • act_kwargs (dict | None) –

  • noisy (bool) –

  • test_ood (bool) –