robmsmt

Benchmarking ASRs - Results from the first 'Out of the Box' Test on an open source dataset

2021-09-04T00:00:00+00:00

First ‘Out of the box’ results for 5 ASRs: VOSK vs Sphinx vs Coqui vs Google vs Amazon

1. Overview

In this post the testing procedure is discussed on the initial small dataset that is included with SpeechLoop. Speech systems are updated continuously so it’s sometimes hard to keep track of what the latest WER of each system is. This tool allows for easy benchmarking for a given dataset, to answer that question.

Although the test itself has a few limitations, it’s very small and the simplest which may be useful for those who want to quickly get up and running and evaluating different speech systems.

2. The Data

There are 78 WAVs of a single female person speaking very short nouns (mostly single words).

    FileCount: 78
    TotalLength: 00:01:15
    ReadType: read
    AudioQualityType: high
    AudioNoiseType: none
    MicQuality: high
    AudioDist: near-field
    Persons: solo
    Language: EN
    Accent: US
    Gender: female
    Filetype: wav
    AudioFile: Signed 16 bit Little Endian, Rate 16000 Hz, Mono

Although this isn’t a great comprehensive test on it’s own - it is another single data point that can help paint the picture of how good the ASR is in real world settings. The details on the dataset can be found here.

🎉 The dataset is open sourced under creative commons along with the SpeechLoop tool under Apache2 for anyone to download and repeat. I would appreciate anyone who validates and double checks and finds flaws in any part of the data/test.

Sometimes single word commands are harder for ASRs when using WER as the metric. This is for two reasons, firstly, because the utterances are usually very short there’s less information to evaluate (relies on the acoustic matching more so than pure unigram probabilities). Secondly, a single character when wrong can cause the word to be incorrect which contributes very large error to the overall score. This can make this a useful test to run!

3. Setup

Since this is the first benchmark, every step is repeated in detail so that others can follow.

Since we are planning to use GoogleASR and AWSTranscribe we need to setup the credentials. Suggested way is in the ~ /.bashrc:

#AWS
export AWS_ACCESS_KEY_ID=<accesskeyhere>
export AWS_SECRET_ACCESS_KEY=<secret_access_key_here>
#GOOGLE
export GOOGLE_APPLICATION_CREDENTIALS=/home/you/path/to/your/creds.json

Then re-source the bashrc with: source ~/.bashrc

Next, setup the repo as per the instructions:

git clone https://github.com/robmsmt/SpeechLoop && cd SpeechLoop
python3 -m venv venv/py3
source venv/py3/bin/activate
pip install -r requirements-dev.txt

Run:

cd speechloop
python main.py --input_csv='data/simple_test/simple_test.csv' --wanted_asr=all

4. ASRs

When we run the script we provide --wanted_asr=all this means (as of writing):

sp = Sphinx - pocketsphinx 0.1.15
vs = Vosk - based off of model en-small-us-0.15
cq = Coqui - based off of model v0.9.3
gg = GoogleASR - 21-09-06
aw = AwsTranscribe - 21-09-06

The last two being cloud ASRs so all we can use to distinguish is today’s date.

5. Results

After some minor tidying up of the dataframe, see the Jupyter notebook for details… a dataframe is generated which we can export to JSON, as seen below.

Something interesting to note, all ASRs mistranscribed the WAVs “xebec” and “oars”.

There were many words that ALL ASRs got correct e.g. “class”, “camera”, “nail”, “skateboard”, “windows”, and “hammer”.

6. Conclusion

In summary, after transcription corrections for each ASR output, the following are the results.

Overall the results are somewhat expected. CloudASRs tend to dominate, with VOSK being very good modern alternative. Coqui had surprisingly bad results but it is expected to do better on longer sentences or with a customized language model.

For the next tests, we will attempt to tune each for the above dataset (where possible). As well as trying on everyone’s favourite opensource dataset…!

Introducing SpeechLoop

2021-08-14T00:00:00+00:00

Here is a tool developed to evaluate multiple ASR (automatic speech recognition) engines and benchmark them to help answer the question, what is the best ASR for my dataset?

The Problem(s)

Every time an ASR is released, commercial or otherwise, it often comes with a claim that they are the best.

It’s a very difficult claim to substantiate. How do they know they’re the best? Maybe they’ve publicly benchmarked? Sometimes they release a vague chart with cherry picked results. Other times there’s a more detailed study. But still it’s hard to believe that the person doing the benchmarking is likely to report findings that show their ASR in a bad light.

Progress in the ML space moves very quickly. Overnight a new model is trained. What about the papers last year with the new dizzying end-2-end-2-end with mega-transformers? and with guest appearances from all your favourite childhood tv-shows. Were these compared? It can be difficult to keep track, plus you don’t want to keep hitting your competitors’ ASR all the time - this takes time away from R&D.

One can judge from experiment, or one can blindly accept authority.
Robert A. Heinlein

This is one of the reasons why SpeechLoop was developed.

Design Decisions

The following were some design decisions made while developing SpeechLoop:

Easy to use with sensible defaults - following the “for humans” philosophy that other python packages have popularized.
Heavy use of docker where required - Docker completely solves the issue of having to install big and heavily conflicting libraries, some of which are only needed for the install part of the process. If you have a reasonable amount of disk space and any x86_64 cpu you can go quite far with just sending bytes to an already built container.
Limited number of 3rd party dependencies - this can be a challenge when installing a library, the more packages there are, the more that can go wrong.
Pandas Dataframes - for easy analysis and simple WER calculation… the dataframe felt like the best fit where each row corresponds to an audio file and each ASR transcript would be a column.

Features

There are many features to list, but a video of simple usage will be even more useful.

Here we demonstrate using the CLI to record microphone and pipe the data into the selected 3 ASRs.

Above we run the CLI to evaluate against the included test dataset. It outputs a CSV. Below is a copy of the output text - following blogposts will focus on ASR results themselves.

------------------------------
Average value of vs_corrected_wer is 0.2308

------------------------------
Average value of sp_corrected_wer is 0.5128

------------------------------
Average value of cq_corrected_wer is 0.5513

Total inference time taken for vs is 0:01:15.081824
Total inference time taken for sp is 0:00:42.342496
Total inference time taken for cq is 0:01:27.646377
------------------------------
------------------------------
Finished, with total runtime: 0:03:42

Often it’s good to load the CSV in excel-like tools to quickly skim the results. For anything more advanced a jupyter notebook might be the way to go.

Future

Different data formats (e.g. Kaldi has SCP, it would be nice to read those)
Cost calculator for APIs
More ASRs
Automated benchmarks of common datasets - for example librispeech or one of the newly released huge datasets

Thanks

Thanks to all ASRs - this tool just builds on top of others work. Without them it wouldn’t make sense.
My employer MerlynMind who have actively encouraged this work - note that we are currently hiring for many roles.

Audio from inside a Pandas Dataframe in a Jupyter Notebook

2021-01-29T00:00:00+00:00

Jupyter notebooks are fantastic tools to explore data. You can also play audio within them.

In order to make ASR better you often have to review and listen to many audio files. One great way I've found to do this is through a Pandas dataframe in a Jupyter notebook.

As you can see from the picture above, it keeps the audio component hidden until you click on the wav filename which triggers some javascript to make the audio HTML components visible. Kinda neat.

Within Jupyter notebooks you can use the magic tags %%html in which are you able to define some javascript and CSS. See segment below:

%%html
<script>

  function toggle_visibility(i)
  {
      if (i.style.display === "none") {
          i.style.display = "block";
      } else {
          i.style.display = "none";
      }
  }
</script>

<style>
  .filenameclass{
      cursor: pointer;
      text-decoration: underline blue;
      text-underline-position: under;
      color: blue;
  }
  audio{
      width: 200px;
  }
</style>

Next we just need to define a function for creating the HTML with the new audio widget in, this takes a path and returns some HTML with a special unique ID we can use to identify the component:

def path_to_audio_html(path):
    myuuid = 'x'+str(uuid4())[:8]
    filename = path.rsplit('/')[-1]
    return f'''
    <div>
        <div class='filenameclass' onclick="toggle_visibility({myuuid})">{filename}</div>
        <div id="{myuuid}" style="display:none;">
            <audio controls>
              <source src="{path}" type="audio/wav" preload="none">
                Your browser does not support the audio element.
            </audio>
        </div>
    </div>
    '''

Assuming we already have a dataframe (with column to filepath), we just need to call the HTML function from IPython.core.display:

HTML(df.to_html(escape=False,formatters=dict(filename=path_to_audio_html)))

Ideas

It’s interesting where this could lead to, could a more powerful javascript player be used? Or could this even be used for a basic transcription service with a text field to correct or save the annotation?

Bash Tricks

2021-01-22T00:00:00+00:00

One of the great things about bash is just how flexible and configurable it is. In this post I’ll explain some changes that I use frequently.

This is the first blog post. 🎉

This post is mostly for my own benefit, so I can quickly copy these configs onto a new machine and see how the layout styling works.

As a general rule I try not install any packages/plugins or anything which pushes you too far from the stock setup… e.g. zsh!

Bash Changes

Here are the changes I have made to my config in order of most important.

Git Branch Highlighting

I find knowing that I am in a git repo one of the most important pieces of information. On top of that it will tell me what branch I am on. This was inspired by the following post.

# git branches
git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
}
PS1="$PS1\[\033[00;31m\]\$(git_branch)\[\033[00m\]\$ "

Speed Aliases

# speed
alias ve="virtualenv --python=python3.8 ./venv/py38 && source ./venv/py38/bin/activate"
alias qg="git add . && git commit -m 'quick commit' && git push origin"
alias gs="git status"
alias rl="readlink -f"
alias gf='git flow'

ve Typing this in any repo will generate a virtualenv and source it automatically. I use this all the time and is great when experimenting and exploring github repos. Virtualenv is supposed to be replaced now with venv.
qg Stands for quick git. It’s ruining the git commit messages but it assumes you’ll squash later. Also it’s risky-ish since it’s adding everything with the git add . but each to their own.
gs Stands for git status. Used in conjunction with #2. There is already a program in called gs in /usr/bin something to do with pdfs, i’ve never used before. Apparently if you want to not use the alias you can easily skip by escaping… so \gs will return the Ghost shell.
rl When working with a team it’s great to send them the exact full path of the file. I use it so often and readlink -f takes too long.
gf Useful if you use gitflow. gf init

Audio Aliases

# audio
alias wavcount="find -type f -name '*.wav' -print0 | xargs -0 mplayer -vo dummy -ao dummy -identify 2>/dev/null | perl -nle '/ID_LENGTH=([0-9\.]+)/ && (\$t +=\$1) && printf \"%02d:%02d:%02d\n\",\$t/3600,\$t/60%60,\$t%60' | tail -n 1"

wavcount Counts the number of hours:mins:secs of audio data is in a directory recursively.

rob@rob-T480s:~/projects/audio (master)$ wavcount
06:49:34

Android Changes

When I work on Android, I find the following are useful. The first is aver which is a special alias which is exported so that it can take advantage of using a variable $1 in the middle of the command.

# android specific
function aver { aapt dump badging "$1" | grep versionName; }
export -f aver
alias sy='watch -n 2 scrcpy -m 1024 --always-on-top'

aver - Stands for APK version. I can run aver apk_name.apk it prints the APK version information using the aapt tool.

rob@rob-T480s:~/$ aver spotify.apk
package: name='com.spotify.music' versionCode='65804403' versionName='8.5.82.894' platformBuildVersionName=''

sy - Scrcpy (presumably pronounced Screencopy) is a great tool to see the android interface on your machine. The command sy starts scrcpy with watch -n 2 which will try to restart it every 2 secs as soon as it ends. Useful for development purposes.

Which bash configuration file to use?

To .bashrc or .profile? That is the question.

.profile is executed on startup only. .bashrc is executed every time you open a new terminal. I generally use .bashrc since I think it’s more common. I also put the following snippet into my code so I can test when each gets executed. I put this in both .bashrc and inside .profile just append it to the end.

# this prints an echo every time the script is executed (useful to know)
self=$(readlink -f "${BASH_SOURCE[0]}")
basename=$(basename "$self")
echo "$basename was executed"

rob@rob-T480s:~/projects/blog/github.io (master)$ source ~/.profile
.bashrc was executed
.profile was executed
rob@rob-T480s:~/projects/blog/github.io (master)$ source ~/.bashrc
.bashrc was executed

Useful crontabs

I want to purge my ~/Downloads folder for files and directories older than 15 days, but I want to send them to the Trash instead. The trash I want to empty at a longer every 180 days automatically on reboot only.

User: crontab -e:

0 0 * * * find /home/username/Downloads/ -mindepth 1 -mtime +15 -exec gio trash {} \;

Root: sudo crontab -e:

@reboot find /home/username/.local/share/Trash/expunged/ -type f -mtime +180 -exec rm {} \;
@reboot find /home/username/.local/share/Trash/files/ -type f -mtime +180 -exec rm {} \;
@reboot find /home/username/.local/share/Trash/info/ -type f -mtime +180 -exec rm {} \;