block_zoo package

Submodules

block_zoo.BaseLayer module

class block_zoo.BaseLayer.BaseConf(**kwargs)[source]

Bases: abc.ABC

Basic configuration

Parameters:
  • input_dim (int) – the dimension of input.
  • hidden_dim (int) – the dimension of hidden state.
  • dropout (float) – dropout rate.
  • (others).. –
add_attr_exist_assertion_for_dev(attr)[source]

check if there are some attributes being forgot by developers

Parameters:attr (str) – the attribution name
Returns:None
add_attr_exist_assertion_for_user(attr)[source]

check if there are some attributes being forgot by users

Parameters:attr (str) – the attribution name
Returns:None
add_attr_range_assertion(attr, range, bounds_legal=(True, True))[source]

check if attribute falls into the legal range

Parameters:
  • attr (str) – the attribution name
  • range (tuple) – (num/float(‘-inf’)/None, num/float(‘inf’)/None), None means -inf or inf.
  • bounds_legal (tuple) – (bool, bool), if the left/right bound is legal
Returns:

None
add_attr_type_assertion(attr, specified_type)[source]

check if the types of attributes are legal

Parameters:
  • attr (str) – the attribution name
  • specified_type (None/str/list) – one specified_type of a list of specified_type(including None)
Returns:

None
add_attr_value_assertion(attr, legal_values)[source]

check if attr equals to one of the legal values

Parameters:
  • attr (str) – the attribution name
  • legal_values (list) – include the legal value
Returns:

None
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None
class block_zoo.BaseLayer.BaseLayer(layer_conf)[source]

Bases: torch.nn.modules.module.Module

The base class of layers

Parameters:layer_conf (BaseConf) – configuration of a layer
forward(*args)[source]
Parameters:*args (list) – a list of args in which arg should be a pair of (representation, length)
Returns:None
is_cuda()[source]

To judge if the layer is on CUDA if there are parameters in this layer, judge according to the parameters; else: judge according to the self.layer_conf.use_gpu

Returns:whether to use gpu
Return type:bool

block_zoo.BiGRU module

class block_zoo.BiGRU.BiGRU(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Bidirectional GRU

Parameters:layer_conf (BiGRUConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.BiGRU.BiGRUConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiGRU

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.BiGRULast module

class block_zoo.BiGRULast.BiGRULast(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Get the last hidden state of Bi GRU

Parameters:layer_conf (BiGRULastConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.BiGRULast.BiGRULastConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of the layer BiGRULast

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.BiLSTM module

class block_zoo.BiLSTM.BiLSTM(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Bidrectional LSTM

Parameters:layer_conf (BiLSTMConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.BiLSTM.BiLSTMConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiLSTM

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
  • num_layers (int) – number of BiLSTM layers
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.BiLSTMAtt module

class block_zoo.BiLSTMAtt.BiLSTMAtt(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

BiLSTM with self attention

Parameters:layer_conf (BiLSTMAttConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.BiLSTMAtt.BiLSTMAttConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiLSTMAtt layer

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
  • num_layers (int) – number of BiLSTM layers
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.BiLSTMLast module

class block_zoo.BiLSTMLast.BiLSTMLast(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

get last hidden states of Bidrectional LSTM

Parameters:layer_conf (BiLSTMConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.BiLSTMLast.BiLSTMLastConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiLSTMLast

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
  • num_layers (int) – number of BiLSTM layers
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.BiQRNN module

class block_zoo.BiQRNN.BiQRNN(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Bidrectional QRNN

Parameters:layer_conf (BiQRNNConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, hidden_dim]
Return type:

Tensor
class block_zoo.BiQRNN.BiQRNNConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiQRNN

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • window – the size of the convolutional window. Supports 1 and 2. Default: 1
  • zoneout – Whether to apply zoneout (failing to update elements in the hidden state). Default: 0
  • dropout (float) – dropout rate bewteen BiQRNN layers
  • num_layers (int) – number of BiQRNN layers
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
class block_zoo.BiQRNN.ForgetMult[source]

Bases: torch.nn.modules.module.Module

ForgetMult computes a simple recurrent equation: h_t = f_t * x_t + (1 - f_t) * h_{t-1}

This equation is equivalent to dynamic weighted averaging.

Inputs: X, hidden
  • X (seq_len, batch, input_size): tensor containing the features of the input sequence.
  • F (seq_len, batch, input_size): tensor containing the forget gate values, assumed in range [0, 1].
  • hidden_init (batch, input_size): tensor containing the initial hidden state for the recurrence (h_{t-1}).
forward(f, x, hidden_init=None)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class block_zoo.BiQRNN.QRNN(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, **kwargs)[source]

Bases: torch.nn.modules.module.Module

Applies a multiple layer Quasi-Recurrent Neural Network (QRNN) to an input sequence.

Parameters:
  • input_size – The number of expected features in the input x.
  • hidden_size – The number of features in the hidden state h. If not specified, the input size is used.
  • num_layers – The number of QRNN layers to produce.
  • dropout – Whether to use dropout between QRNN layers. Default: 0.
  • bidirectional – If True, becomes a bidirectional QRNN. Default: False.
  • save_prev_x – Whether to store previous inputs for use in future convolutional windows (i.e. for a continuing sequence such as in language modeling). If true, you must call reset to remove cached previous values of x. Default: False.
  • window – Defines the size of the convolutional window (how many previous tokens to look when computing the QRNN values). Supports 1 and 2. Default: 1.
  • zoneout – Whether to apply zoneout (i.e. failing to update elements in the hidden state) to the hidden state updates. Default: 0.
  • output_gate – If True, performs QRNN-fo (applying an output gate to the output). If False, performs QRNN-f. Default: True.
Inputs: X, hidden
  • X (seq_len, batch, input_size): tensor containing the features of the input sequence.
  • hidden (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for the QRNN.
Outputs: output, h_n
  • output (seq_len, batch, hidden_size * num_directions): tensor containing the output of the QRNN for each timestep.
  • h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
forward(input, hidden=None)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

reset()[source]

If your convolutional window is greater than 1, you must reset at the beginning of each new sequence

tensor_reverse(tensor)[source]
class block_zoo.BiQRNN.QRNNLayer(input_size, hidden_size=None, save_prev_x=False, zoneout=0, window=1, output_gate=True)[source]

Bases: torch.nn.modules.module.Module

Applies a single layer Quasi-Recurrent Neural Network (QRNN) to an input sequence.

Parameters:
  • input_size – The number of expected features in the input x.
  • hidden_size – The number of features in the hidden state h. If not specified, the input size is used.
  • save_prev_x – Whether to store previous inputs for use in future convolutional windows (i.e. for a continuing sequence such as in language modeling). If true, you must call reset to remove cached previous values of x. Default: False.
  • window – Defines the size of the convolutional window (how many previous tokens to look when computing the QRNN values). Supports 1 and 2. Default: 1.
  • zoneout – Whether to apply zoneout (i.e. failing to update elements in the hidden state) to the hidden state updates. Default: 0.
  • output_gate – If True, performs QRNN-fo (applying an output gate to the output). If False, performs QRNN-f. Default: True.
Inputs: X, hidden
  • X (seq_len, batch, input_size): tensor containing the features of the input sequence.
  • hidden (batch, hidden_size): tensor containing the initial hidden state for the QRNN.
Outputs: output, h_n
  • output (seq_len, batch, hidden_size): tensor containing the output of the QRNN for each timestep.
  • h_n (1, batch, hidden_size): tensor containing the hidden state for t=seq_len
forward(X, hidden=None)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

reset()[source]

block_zoo.Conv module

class block_zoo.Conv.Conv(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Convolution along just 1 direction

Parameters:layer_conf (ConvConf) – configuration of a layer
forward(string, string_len=None)[source]

process inputs

Parameters:
  • string (Tensor) – tensor with shape: [batch_size, seq_len, feature_dim]
  • string_len (Tensor) – [batch_size]
Returns:

shape: [batch_size, (seq_len - conv_window_size) // stride + 1, output_channel_num]
Return type:

Tensor
class block_zoo.Conv.ConvConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of Conv

Parameters:
  • stride (int) – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1
  • padding (int) – implicit zero paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0
  • window_size (int) – actually, the window size is (window_size, feature_dim), because for NLP tasks, 1d convolution is more commonly used.
  • input_channel_num (int) – for NLP tasks, input_channel_num would always be 1
  • output_channel_num (int) – number of feature maps
  • batch_norm (bool) – If True, apply batch normalization before activation
  • activation (string) – activation functions, e.g. ReLU
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.ConvPooling module

class block_zoo.ConvPooling.ConvPooling(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Convolution along just 1 direction

Parameters:layer_conf (ConvConf) – configuration of a layer
forward(string, string_len=None)[source]

process inputs

Parameters:
  • string (Tensor) – tensor with shape: [batch_size, seq_len, feature_dim]
  • string_len (Tensor) – [batch_size]
Returns:

shape: [batch_size, (seq_len - conv_window_size) // stride + 1, output_channel_num]
Return type:

Tensor
class block_zoo.ConvPooling.ConvPoolingConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of Conv + Pooling architecture

Parameters:
  • stride (int) – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1
  • padding (int) – implicit zero paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0
  • window_sizes (list) – for each window_size, the actual window size is (window_size, feature_dim), because for NLP tasks, 1d convolution is more commonly used.
  • input_channel_num (int) – for NLP tasks, input_channel_num would always be 1
  • output_channel_num (int) – number of feature maps
  • batch_norm (bool) – If True, apply batch normalization before activation
  • activation (string) – activation functions, e.g. ReLU
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.Dropout module

class block_zoo.Dropout.Dropout(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Parameters:layer_conf (DropoutConf) – configuration of a layer
forward(string, string_len=None)[source]

process inputs

Parameters:
  • string (Tensor) – any shape.
  • string_len (Tensor) – [batch_size], default is None.
Returns:

has the same shape as string.
Return type:

Tensor
class block_zoo.Dropout.DropoutConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration for Dropout

Parameters:dropout (float) – dropout rate, probability of an element to be zeroed

Returns:

declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.Embedding module

class block_zoo.Embedding.Embedding(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Embedding layer

Parameters:layer_conf (EmbeddingConf) – configuration of a layer
forward(inputs, use_gpu=False)[source]

process inputs

Parameters:
  • inputs (dict) –

    a dictionary to describe each transformer_model inputs. e.g.:

    char_emb’: [[char ids of word1], [char ids of word2], […], …], shape: [batch_size, seq_len, word character num]

    ’word’: word ids (Variable), shape:[batch_size, seq_len],

    ’postag’: postag ids (Variable), shape: [batch_size, seq_len],

    …

  • use_gpu (bool) – put embedding matrix on GPU (True) or not (False)
Returns:

the embedding representation with shape [batch_size, seq_len, emb_dim]
Return type:

Variable
class block_zoo.Embedding.EmbeddingConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration for Embedding

Parameters:conf – a dictionary. The key is embedding type, such as word embedding, char embedding, Part-of-Speech embedding and so on.

Example:

"conf": {
  "word": {
    "cols": ["question_text", "answer_text"],
    "dim": 300,
    "fix_weight": true
  },
  "postag": {
    "cols": ["question_postag","answer_postag"],
    "dim": 20
  },
  "char": {
    "cols": ["question_char", "answer_char"],
    "type": "CNNCharEmbedding",
    "dropout": 0.2,
    "dim": 30,
    "embedding_matrix_dim": 8,
    "stride":1,
    "window_size": 5,
    "activation": null
  }
}
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.EncoderDecoder module

class block_zoo.EncoderDecoder.EncoderDecoder(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

The encoder decoder framework

Parameters:layer_conf (EncoderDecoderConf) – configuration of a layer
forward(string, string_len)[source]

process inputs with encoder & decoder

Parameters:
  • string (Variable) – [batch_size, seq_len, dim]
  • string_len (ndarray) – [batch_size]
Returns:

decode scores with shape [batch_size, seq_len, decoder_vocab_size]
Return type:

Variable
class block_zoo.EncoderDecoder.EncoderDecoderConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of Encoder-Decoder

Parameters:
  • encoder (str) – encoder name
  • encoder_conf (dict) – configurations of encoder
  • decoder (str) – decoder name
  • decoder_conf (dict) – configurations of decoder
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Dimension inference of encoder and decoder is conducted here, but not in the Model.

Returns:

verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.Flatten module

class block_zoo.Flatten.Flatten(layer_conf)[source]

Bases: torch.nn.modules.module.Module

Flatten layer to flatten the input from [bsatch_size, seq_len, dim] to [batch_size, seq_len*dim]

Parameters:layer_conf (FlattenConf) – configuration of a layer
forward(string, string_len)[source]

process input

Parameters:*args – (Tensor): string,string_len e.g. string (Tensor): [batch_size, seq_len, dim], string_len (Tensor): [batch_size]
Returns:[batch_size, seq_len*dim], [batch_size]
Return type:Tensor
class block_zoo.Flatten.FlattenConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of Flatten layer

declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.HighwayLinear module

class block_zoo.HighwayLinear.HighwayLinear(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

A Highway layer does a gated combination of a linear transformation and a non-linear transformation of its input. \(y = g * x + (1 - g) * f(A(x))\), where \(A\) is a linear transformation, \(f\) is an element-wise non-linearity, and \(g\) is an element-wise gate, computed as \(sigmoid(B(x))\). This module will apply a fixed number of highway layers to its input, returning the final result.

Parameters:layer_conf (HighwayLinearConf) – configuration of a layer
forward(string, string_len)[source]

process inputs

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, 2 * hidden_dim]
Return type:

Tensor
class block_zoo.HighwayLinear.HighwayLinearConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of BiLSTM

Parameters:
  • hidden_dim (int) – dimension of hidden state
  • dropout (float) – dropout rate
  • num_layers (int) – number of BiLSTM layers
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.Linear module

class block_zoo.Linear.Linear(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Linear layer

Parameters:layer_conf (LinearConf) – configuration of a layer
forward(string, string_len=None)[source]

process inputs

Parameters:
  • string (Tensor) – any shape.
  • string_len (Tensor) – [batch_size], default is None.
Returns:

has the same shape as string.
Return type:

Tensor
class block_zoo.Linear.LinearConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration for Linear layer

Parameters:
  • hidden_dim (int or list) – if is int, it means there is one linear layer and the hidden_dim is the dimension of that layer. if is list of int, it means there is multiple linear layer and hidden_dim are the dimensions of these layers.
  • activation (str) – Name of activation function. All the non-linear activations in http://pytorch.org/docs/0.3.1/nn.html#non-linear-activations are supported, such as ‘Tanh’, ‘ReLU’, ‘PReLU’, ‘ReLU6’ and ‘LeakyReLU’. Default is None.
  • last_hidden_activation (bool) – [Optional], whether to add nonlinearity to the last linear layer’s output. Default is True.
  • last_hidden_softmax (bool) – [Optional], whether to add softmax to the last linear layer’s output. Default is False.
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None
verify_before_inference()[source]

Some conditions must be fulfilled, otherwise there would be errors when calling inference()

The difference between verify_before_inference() and verify() is that:
verify_before_inference() is called before inference() while verify() is called after inference().
Returns:None

block_zoo.Pooling module

class block_zoo.Pooling.Pooling(layer_conf)[source]

Bases: block_zoo.BaseLayer.BaseLayer

Pooling layer

Parameters:layer_conf (PoolingConf) – configuration of a layer
forward(string, string_len=None)[source]

process inputs

Parameters:
  • string (Tensor) – any shape.
  • string_len (Tensor) – [batch_size], default is None.
Returns:

Pooling result of string
Return type:

Tensor
class block_zoo.Pooling.PoolingConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Parameters:
  • pool_type (str) – ‘max’ or ‘mean’, default is ‘max’.
  • pool_axis (int) – which axis to conduct pooling, default is 1.
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

block_zoo.Transformer module

class block_zoo.Transformer.Transformer(layer_conf)[source]

Bases: torch.nn.modules.module.Module

Transformer layer

Parameters:layer_conf (TransformerConf) – configuration of a layer
forward(string, string_len)[source]

process input

Parameters:
  • string (Tensor) – [batch_size, seq_len, dim]
  • string_len (Tensor) – [batch_size]
Returns:

[batch_size, seq_len, output_dim], [batch_size]
Return type:

Tensor
class block_zoo.Transformer.TransformerConf(**kwargs)[source]

Bases: block_zoo.BaseLayer.BaseConf

Configuration of Transformer

Parameters:
  • attention (str) – attention name
  • attention_conf (dict) – configurations of attention
  • layernorm1 (str) – layernorm1 name
  • layernorm1_conf (dict) – configurations of layernorm1
  • mlp (str) – mlp name
  • mlp_conf (dict) – configuration of mlp
  • layernorm2 (str) – layernorm2 name
  • layernorm2_conf (dict) – configurations of layernorm2
  • n_layer (int) –
declare()[source]

Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

num_of_input is N(N>0) means this layer accepts N inputs;

num_of_input is -1 means this layer accepts any number of inputs;

The rank here is not the same as matrix rank:

For a scalar, its rank is 0;

For a vector, its rank is 1;

For a matrix, its rank is 2;

For a cube of numbers, its rank is 3.

… For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

if num_of_input > 0:

len(input_ranks) should be equal to num_of_input

elif num_of_input == -1:

input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

Returns:None
default()[source]

Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

Returns:None
inference()[source]

Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

Returns:None
verify()[source]

Define some necessary varification for your layer when we define the model.

If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

Returns:None

Module contents