What is LSTM
LongShort-Term Memory (LSTM) is a
type of recurrent neural network (RNN)
architecture that is designed to handle
long-range dependencies and overcome
the vanishing gradient problem. LSTMs
are widely used in sequential data tasks
such as time-series forecasting, speech
recognition, and natural language
processing (NLP).
3.
LSTM Architecture
An LSTMcell consists of the following components:
1.Forget Gate (ft
): Decides what information to discard from the cell
state.
2.Input Gate (it
): Determines what new information should be stored.
3.Cell State (Ct
): Stores long-term memory.
4.Output Gate (ot): Controls what information is output from the cell.
• Each gate is controlled using a sigmoid activation function (σ), while the
new candidate memory (Ct
) is computed using a tanh activation.
4.
Forget Gate
The informationthat is no longer useful in the cell state is removed with
the forget gate. Two inputs xt (input at the particular time) and ht-
1 (previous cell output) are fed to the gate and multiplied with weight
matrices followed by the addition of bias. The resultant is passed
through an activation function which gives a binary output. If for a
particular cell state the output is 0, the piece of information is forgotten
and for output 1, the information is retained for future use.
5.
Input gate
The additionof useful information to the cell state is done by the
input gate. First, the information is regulated using the sigmoid
function and filter the values to be remembered similar to the
forget gate using inputs ht-1 and xt. . Then, a vector is created
using tanh function that gives an output from -1 to +1, which
contains all the possible values from ht-1 and xt. At last, the values
of the vector and the regulated values are multiplied to obtain the
useful information.
6.
Output Gate
The taskof extracting useful information from the current cell state
to be presented as output is done by the output gate. First, a vector
is generated by applying tanh function on the cell. Then, the
information is regulated using the sigmoid function and filter by the
values to be remembered using inputs ht 1
−
and xt
. At last, the values
of the vector and the regulated values are multiplied to be sent as
an output and input to the next cell.
7.
LSTM Mathematical Formulation
Fora given time step t, given an input xt
, previous hidden
state ht 1
−
, and previous cell state Ct 1
− , the LSTM
computes:
Forget Gate (Decides what to forget)
Input Gate (Decides what to store)
Candidate Cell State (New memory candidate)
8.
LSTM Mathematical Formulation
CellState Update (Final memory state)
Output Gate (Decides what to output)
Hidden State Update (Final output of LSTM cell)
where
9.
Real-Life Analogy
Hidden stateht Your working memory
→
• Think of it like what you remember from a conversation just now.
• You use it immediately, but it might fade away.
Cell state Ct Your long-term memory
→
• It stores important information you need in the future.
• If something is very important, you reinforce it (update Ct ).
• If something is no longer useful, you forget it (modify Ct using the forget gate).
• For example, if you're reading a book,
• Your hidden state ht is the sentence you just read.
10.
Problem with Long-Term
Dependenciesin RNN
Recurrent Neural Networks (RNNs) are designed to handle sequential data
by maintaining a hidden state that captures information from previous
time steps. However they often face challenges in learning long-term
dependencies where information from distant time steps becomes crucial
for making accurate predictions for current state. This problem is known as
the vanishing gradient or exploding gradient problem.
• Vanishing Gradient: When training a model over time, the gradients
(which help the model learn) can shrink as they pass through many steps.
This makes it hard for the model to learn long-term patterns since earlier
information becomes almost irrelevant.
• Exploding Gradient: Sometimes, gradients can grow too large, causing
instability. This makes it difficult for the model to learn properly, as the
updates to the model become erratic and unpredictable.
11.
Applications of LSTM
•Language Modeling: Used in tasks like language modeling, machine translation and
text summarization. These networks learn the dependencies between words in a
sentence to generate coherent and grammatically correct sentences.
• Speech Recognition: Used in transcribing speech to text and recognizing spoken
commands. By learning speech patterns they can match spoken words to
corresponding text.
• Time Series Forecasting: Used for predicting stock prices, weather and energy
consumption. They learn patterns in time series data to predict future events.
• Anomaly Detection: Used for detecting fraud or network intrusions. These networks
can identify patterns in data that deviate drastically and flag them as potential
anomalies.
12.
Applications of LSTM
•Recommender Systems: In recommendation tasks like suggesting movies,
music and books. They learn user behavior patterns to provide personalized
suggestions.
• Video Analysis: Applied in tasks such as object detection, activity recognition
and action classification. When combined with
Convolutional Neural Networks (CNNs) they help analyze video data and
extract useful information.
13.
LSTM vs RNN
FeatureLSTM (Long Short-term Memory) RNN (Recurrent Neural Network)
Memory
Has a special memory unit that allows it to learn
long-term dependencies in sequential data
Does not have a memory unit
Directionality
Can be trained to process sequential data in
both forward and backward directions
Can only be trained to process sequential data
in one direction
Training
More difficult to train than RNN due to the
complexity of the gates and memory unit
Easier to train than LSTM
Long-term dependency
learning
Yes Limited
Ability to learn
sequential data
Yes Yes
Applications
Machine translation, speech recognition, text
summarization, natural language processing,
time series forecasting
Natural language processing, machine
translation, speech recognition, image
processing, video processing
Lo más leído
Lo más leído
Lo más leído
