Describe the bug
I'm trying to fine-tune the 'lang-id-voxlingua107-ecapa' embedding model, for 21 indian languages but as soon as the training hits i get the cufft error. I initially thought this could be problem because of small duration audios, so i filtered out them. This error still exists:
Is there any lower limit for audio duration we need the audios to be in??
Expected behaviour
This is the error:
Traceback (most recent call last):
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 310, in
lid_brain.fit(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1575, in fit
self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1400, in _fit_train
loss = self.fit_batch(batch)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1199, in fit_batch
outputs = self.compute_forward(batch, sb.Stage.TRAIN)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 78, in compute_forward
feats, lens = self.prepare_features(batch.sig, stage)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 49, in prepare_features
wavs, lens = self.hparams.wav_augment(wavs, lens)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 392, in forward
output, output_lengths = self.augment(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 252, in augment
out = augment_fun(next_input[idx, ...])
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/time_domain.py", line 409, in forward
rev_waveform = reverberate(waveforms, rir_waveform, rescale_amp="avg")
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 372, in reverberate
waveforms = convolve1d(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 287, in convolve1d
result = fft.rfft(waveform) * fft.rfft(kernel)
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
To Reproduce
No response
Environment Details
No response
Relevant Log Output
Additional Context
No response
Describe the bug
I'm trying to fine-tune the 'lang-id-voxlingua107-ecapa' embedding model, for 21 indian languages but as soon as the training hits i get the cufft error. I initially thought this could be problem because of small duration audios, so i filtered out them. This error still exists:
Is there any lower limit for audio duration we need the audios to be in??
Expected behaviour
This is the error:
Traceback (most recent call last):
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 310, in
lid_brain.fit(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1575, in fit
self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1400, in _fit_train
loss = self.fit_batch(batch)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/core.py", line 1199, in fit_batch
outputs = self.compute_forward(batch, sb.Stage.TRAIN)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 78, in compute_forward
feats, lens = self.prepare_features(batch.sig, stage)
File "/speech/uu/Shabd/speechbrain/recipes/CommonLanguage/lang_id/train.py", line 49, in prepare_features
wavs, lens = self.hparams.wav_augment(wavs, lens)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 392, in forward
output, output_lengths = self.augment(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/augmenter.py", line 252, in augment
out = augment_fun(next_input[idx, ...])
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/augment/time_domain.py", line 409, in forward
rev_waveform = reverberate(waveforms, rir_waveform, rescale_amp="avg")
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 372, in reverberate
waveforms = convolve1d(
File "/speech/uu/miniconda3/envs/gputorch/lib/python3.9/site-packages/speechbrain/processing/signal_processing.py", line 287, in convolve1d
result = fft.rfft(waveform) * fft.rfft(kernel)
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
To Reproduce
No response
Environment Details
No response
Relevant Log Output
Additional Context
No response