Learning

The final component of Emmental’s pipeline is to learn the user defined deep learning model based user defined data.

Core Learning Objects

These are Emmental’s core objects used for learning.

Emmental learner.

class emmental.learner.EmmentalLearner(name=None)[source]

Bases: object

A class for emmental multi-task learning.

Parameters

name (Optional[str]) – Name of the learner, defaults to None.

learn(model, dataloaders)[source]

Learning procedure of emmental MTL.

Parameters
  • model (EmmentalModel) – The emmental model that needs to learn.

  • dataloaders (List[EmmentalDataLoader]) – A list of dataloaders used to learn the model.

Return type

None

Schedulers

These are several schedulers supported in Emmental learner.

Emmental scheduler module.

class emmental.schedulers.MixedScheduler(fillup=False)[source]

Bases: emmental.schedulers.scheduler.Scheduler

Generate batch generator from all dataloaders in mixture for MTL training.

Parameters

fillup (bool) – Whether fillup to make all dataloader the same size.

get_batches(dataloaders, model=None)[source]

Generate batch generator from all dataloaders in mixture for one epoch.

Parameters
Return type

Iterator[Union[Batch, List[Batch]]]

Returns

A generator of all batches.

get_num_batches(dataloaders)[source]

Get total number of batches per epoch.

Parameters

dataloaders (List[EmmentalDataLoader]) – List of dataloaders.

Return type

int

Returns

Total number of batches per epoch.

class emmental.schedulers.RoundRobinScheduler(fillup=False)[source]

Bases: emmental.schedulers.scheduler.Scheduler

Generate batch generator from all dataloaders in round robin order.

Parameters

fillup (bool) – Whether fillup to make all dataloader the same size.

get_batches(dataloaders, model=None)[source]

Generate batch generator from all dataloaders for one epoch.

Parameters
Return type

Iterator[Union[Batch, List[Batch]]]

Returns

A generator of all batches.

get_num_batches(dataloaders)[source]

Get total number of batches per epoch.

Parameters

dataloaders (List[EmmentalDataLoader]) – List of dataloaders.

Return type

int

Returns

Total number of batches per epoch.

class emmental.schedulers.SequentialScheduler(fillup=False)[source]

Bases: emmental.schedulers.scheduler.Scheduler

Generate batch generator from all dataloaders in sequential order.

Parameters

fillup (bool) – Whether fillup to make all dataloader the same size.

get_batches(dataloaders, model=None)[source]

Generate batch generator from all dataloaders for one epoch.

Parameters
Return type

Iterator[Union[Batch, List[Batch]]]

Returns

A generator of all batches.

get_num_batches(dataloaders)[source]

Get total number of batches per epoch.

Parameters

dataloaders (List[EmmentalDataLoader]) – List of dataloaders.

Return type

int

Returns

Total number of batches per epoch.

Configuration Settings

Visit the Configuring Emmental page to see how to provide configuration parameters to Emmental via .emmental-config.yaml.

The learning parameters of the model are described below:

# Learning configuration
learner_config:
    optimizer_path: # path to optimizer state
    scheduler_path: # path to lr scheduler state
    fp16: False # whether to use 16-bit precision
    fp16_opt_level: O1 # Apex AMP optimization level (e.g., ['O0', 'O1', 'O2', 'O3'])
    local_rank: -1 # local_rank for distributed training on gpus
    epochs_learned: 0 # learning epochs learned
    n_epochs: 1 # total number of learning epochs
    steps_learned: 0 # learning steps learned
    n_steps: # total number of learning steps
    skip_learned_data: False # skip learned batches if steps_learned or epochs_learned nonzero
    train_split: # the split for training, accepts str or list of strs
        - train
    valid_split: # the split for validation, accepts str or list of strs
        - valid
    test_split: # the split for testing, accepts str or list of strs
        - test
    online_eval: 0 # whether to perform online evaluation
    optimizer_config:
        optimizer: adam # [sgd, adam, adamax, bert_adam]
        parameters: # parameters to optimize
        lr: 0.001 # Learing rate
        l2: 0.0 # l2 regularization
        grad_clip: # gradient clipping
        gradient_accumulation_steps: 1 # gradient accumulation steps
        asgd_config:
            lambd: 0.0001
            alpha: 0.75
            t0: 1000000.0
        adadelta_config:
            rho: 0.9
            eps: 0.000001
        adagrad_config:
            lr_decay: 0
            initial_accumulator_value: 0
            eps: 0.0000000001
        adam_config:
            betas: !!python/tuple [0.9, 0.999]
            eps: 0.00000001
            amsgrad: False
        adamw_config:
            betas: !!python/tuple [0.9, 0.999]
            eps: 0.00000001
            amsgrad: False
        adamax_config:
            betas: !!python/tuple [0.9, 0.999]
            eps: 0.00000001
        lbfgs_config:
            max_iter: 20
            max_eval:
            tolerance_grad: 0.0000001
            tolerance_change: 0.000000001
            history_size: 100
            line_search_fn:
        rms_prop_config:
            alpha: 0.99
            eps: 0.00000001
            momentum: 0
            centered: False
        r_prop_config:
            etas: !!python/tuple [0.5, 1.2]
            step_sizes: !!python/tuple [0.000001, 50]
        sgd_config:
            momentum: 0
            dampening: 0
            nesterov: False
        sparse_adam_config:
            betas: !!python/tuple [0.9, 0.999]
            eps: 0.00000001
        bert_adam_config:
            betas: !!python/tuple [0.9, 0.999]
            eps: 0.00000001
    lr_scheduler_config:
        lr_scheduler: # [linear, exponential, reduce_on_plateau, cosine_annealing]
        lr_scheduler_step_unit: batch # [batch, epoch]
        lr_scheduler_step_freq: 1
        warmup_steps: # warm up steps
        warmup_unit: batch # [epoch, batch]
        warmup_percentage: # warm up percentage
        min_lr: 0.0 # minimum learning rate
        reset_state: False # reset the state of the optimizer
        exponential_config:
            gamma: 0.9
        plateau_config:
            metric: model/train/all/loss
            mode: min
            factor: 0.1
            patience: 10
            threshold: 0.0001
            threshold_mode: rel
            cooldown: 0
            eps: 0.00000001
        step_config:
            step_size: 1
            gamma: 0.1
            last_epoch: -1
        multi_step_config:
            milestones:
                - 1000
            gamma: 0.1
            last_epoch: -1
        cyclic_config:
            base_lr: 0.001
            max_lr: 0.1
            step_size_up: 2000
            step_size_down:
            mode: triangular
            gamma: 1.0
            scale_fn:
            scale_mode: cycle
            cycle_momentum: True
            base_momentum: 0.8
            max_momentum: 0.9
            last_epoch: -1
        one_cycle_config:
            max_lr: 0.1
            pct_start: 0.3
            anneal_strategy: cos
            cycle_momentum: True
            base_momentum: 0.85
            max_momentum: 0.95
            div_factor: 25.0
            final_div_factor: 10000.0
            last_epoch: -1
        cosine_annealing_config:
            last_epoch: -1
    task_scheduler_config:
        task_scheduler: round_robin # [sequential, round_robin, mixed]
        sequential_scheduler_config:
            fillup: False
        round_robin_scheduler_config:
            fillup: False
        mixed_scheduler_config:
            fillup: False
    global_evaluation_metric_dict: # global evaluation metric dict