Skip to content

Commit 21d6bb2

Browse files
b-flob-flo
committed
force use identity activation in branchformer conv mod + fix main_params doc
1 parent 8f97c7d commit 21d6bb2

3 files changed

Lines changed: 10 additions & 13 deletions

File tree

‎doc/espnet2_tutorial.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -456,13 +456,13 @@ It is similar to the custom encoder in ESPnet1, meaning we don't need to set the
456456
The first and second configurations are optional. If needed, the following parameters can be modified in each configuration:
457457

458458
main_conf:
459-
pos_wise_act_type: Position-wise activation type. (str, default = "swish")
460-
conv_mod_act_type: Convolutional module activation type. (str, default = "swish")
459+
pos_wise_act_type: Conformer position-wise feed-forward activation type. (str, default = "swish")
460+
conv_mod_act_type: Conformer convolution module activation type. (str, default = "swish")
461461
pos_enc_dropout_rate: Dropout rate for the positional encoding layer, if used. (float, default = 0.0)
462462
pos_enc_max_len: Positional encoding maximum length. (int, default = 5000)
463463
simplified_att_score: Whether to use simplified attention score computation. (bool, default = False)
464-
norm_type: Normalization module type for X-former. (str, default = "layer_norm")
465-
conv_mod_norm_type: Normalization module type for Branchformer convolutional module. (str, default = "layer_norm")
464+
norm_type: X-former normalization module type. (str, default = "layer_norm")
465+
conv_mod_norm_type: Branchformer convolution module normalization type. (str, default = "layer_norm")
466466
after_norm_eps: Epsilon value for the final normalization module. (float, default = 1e-05 or 0.25 for BasicNorm)
467467
after_norm_partial: Partial value for the final normalization module, if norm_type = 'rms_norm'. (float, default = -1.0)
468468
# For more information on the parameters below, please refer to espnet2/asr_transducer/activation.py
@@ -521,7 +521,7 @@ The only mandatory configuration is `body_conf`, defining the encoder body archi
521521
conv_mod_norm_momentum (optional): Momentum value for Batchnorm1d in the convolutional module. (float, default = 0.1)
522522
dropout_rate (optional): Dropout rate for some intermediate layers. (float, default = 0.0)
523523
att_dropout_rate (optional): Dropout rate for the attention module. (float, default = 0.0)
524-
pos_wise_dropout_rate (optional): Dropout rate for the position-wise module. (float, default = 0.0)
524+
pos_wise_dropout_rate (optional): Dropout rate for the position-wise feed-forward module. (float, default = 0.0)
525525

526526
In addition, each block has a parameter `num_blocks` to build **N** times the defined block (int, default = 1). This is useful if you want to use a group of blocks sharing the same parameters without writing each configuration.
527527

‎espnet2/asr_transducer/encoder/building.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,13 @@ def build_main_parameters(
4343
"""Build encoder main parameters.
4444
4545
Args:
46-
pos_wise_act_type: Position-wise activation type.
47-
conv_mod_act_type: Convolutional module activation type.
46+
pos_wise_act_type: Conformer position-wise feed-forward activation type.
47+
conv_mod_act_type: Conformer convolution module activation type.
4848
pos_enc_dropout_rate: Positional encoding dropout rate.
4949
pos_enc_max_len: Positional encoding maximum length.
5050
simplified_att_score: Whether to use simplified attention score computation.
51-
norm_type: Normalization module type for X-former.
52-
conv_mod_norm_type: Normalization module type for convolution modules.
51+
norm_type: X-former normalization module type.
52+
conv_mod_norm_type: Conformer convolution module normalization type.
5353
after_norm_eps: Epsilon value for the final normalization.
5454
after_norm_partial: Value for the final normalization with RMSNorm.
5555
dynamic_chunk_training: Whether to use dynamic chunk training.
@@ -165,7 +165,6 @@ def build_branchformer_block(
165165
conv_mod_args = (
166166
linear_size,
167167
configuration["conv_mod_kernel_size"],
168-
main_params["conv_mod_act"],
169168
conv_mod_norm_class,
170169
conv_mod_norm_args,
171170
dropout_rate,

‎espnet2/asr_transducer/encoder/modules/convolution.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,6 @@ class ConvolutionalSpatialGatingUnit(torch.nn.Module):
112112
Args:
113113
size: Initial size to determine the number of channels.
114114
kernel_size: Size of the convolving kernel.
115-
activation: Type of activation function.
116115
norm_class: Normalization module class.
117116
norm_args: Normalization module arguments.
118117
dropout_rate: Dropout rate.
@@ -124,7 +123,6 @@ def __init__(
124123
self,
125124
size: int,
126125
kernel_size: int,
127-
activation: torch.nn.Module = torch.nn.ReLU(),
128126
norm_class: torch.nn.Module = torch.nn.LayerNorm,
129127
norm_args: Dict = {},
130128
dropout_rate: float = 0.0,
@@ -154,7 +152,7 @@ def __init__(
154152
)
155153

156154
self.norm = norm_class(channels, **norm_args)
157-
self.activation = activation
155+
self.activation = torch.nn.Identity()
158156

159157
self.dropout = torch.nn.Dropout(dropout_rate)
160158

0 commit comments

Comments
 (0)