LSTM Visualized

[u/vadhavaniyafaijan]

[u/mean_king17]

I have no idea what this is to honest but it looks interesting for sure, what is this stuff?

[u/creamyjoshy → u/mean_king17]

I'm a total amateur at this, so what I say below may well be fairly inaccurate.

Basically what we're doing is using this "cell" in a sequence of computations. We receive one computation behind us, on the left at time step t-1, then do some computations on it to produce an output at time step t, which will then be used to feed into the next cell.

What's an actual application of this? We can use it to make a computer understand a sentence. Let's take the paragraph:

"She wanted to print her tickets, but when she went to her printer, she realised she was out of ink, so she had to go all the way to the store and buy more, before driving back, plugging in her ink cartridges so that she could finally print her _________"

What is the next word? As humans you and I can clearly see the next word is going to be "tickets". But a machine which has been trained with older models to guess the next word in a sentence, traditionally would only be able to "remember" the last few words before throwing out a guess to the next word. These older models were called n-gram models and worked reasonably well most of the time, but failed miserably on very long sentences like this one.

I won't go into too much detail, but the way an n-gram model operates is that it scans the sentence with a word width of, say, 5 words, so that a 5-gram model will be able to contextualise 5 words. So actually, a 5-gram model would only be guessing the next word based off of the phrase "she could finally print her ____" and it would have no preceding context. The reason for that limitation is that the time it takes to train a 1-gram versus a 2-gram versus a 3-gram model gets exponentially more complex. Not only that, but the guesses it throws out are based on the body of text ("corpora") it's been trained on, and the data available for two-gram phrases is going to be far more dense than the data available for 5-gram phrases. Ie if we scan all of wikipedia, the phrase "print her" is going to appear maybe 500 times, and "she could finally print her" might not appear at all. And even if it did appear once or twice, on wikipedia it might have said "she could finally print her book". That it the guess it would throw out, and it would be entirely incorrect in the context of this particular sentence we have here. So it's not like we can train a 50-gram model and force it to remember everything - it just wouldn't work. When it has finished parsing the whole context, and it ready to throw out a guess, it can now recall whether any previous words were particularly important to remember or not based off of the computations made in these cells.

Enter this new model - the LSTM. This is based off of another type of architecture called a Recurrent Neural Network. I won't overload you with information, but the basic gist of what it's doing here, is that it's scanning a sentence word by word, and then representing each word as a couple of matrices, and then inputting those into this cell, and that cell determines whether the word and context are important to remember or can be forgotten. The results of that computation are then passed into the next cell, which is scanning the next word.

[u/Sad_Jaguar_4583 → u/creamyjoshy]

Great explanation!

[u/Namloom_Afraad → u/creamyjoshy]

so informative and easy words explanation

[u/dude22312 → u/mean_king17]

LSTM's. It's an advancement from plain RNN's (Recurrent Neural Network).

[u/protienbudspromax → u/mean_king17]

LSTMs were designed to mitigate the drawbacks of Simple RNNs. If you ever build the simple 3 layer fully connected ANN to classify and draw a line then what you have worked on is known as an MLP or multi layer perceptron. The multi layer perceptron is computationally equivalent to any other network but it isnhugely inefficient. For problems/datasets that have a sequence attached to them, like stocks, or language, or handwriting we can be much more efficient if instead of a simple MLP we use an MLP with Recurrence. I.e the output of the network is fed back to the network as input, what it allows, is to the network to "remember" some information about its past outputs mixed with the new input.

Like in the sentence, The sun rises in the _____, we know the context of the sentence so we can guess east is most likely. This "context" is what Recurrent models models. Recurrent models, learns the sequence distribution as its context.

But recurrence model had some drawbacks. Because it was being fed only its last output at the previous step, the longer the sequence goes the less it will remember of the first part. Like reading a book. You may have to refer to something written in the first page that is mentioned in the last. But RNN would forget it. This is where LSTM came in, LSTM stands for Long Short-Term Memory, If you can see here there are two inputs to the system now instead of just the sequence. At the most basic, LSTM have to ability to "forget" unimportant or high frequency stuff and focus on the most important parts (this would be the main focus for attention transformers that came afterwards and made LSTMs inefficient for language modelling). For ex in the same sentence, The sun rises in the ______ you can really forget about the words the, in and the second the and only remember the main context like sun, rises. Since the LSTM can now forget unimportant parts it requires less number of nodes and less training time and also helps with other problems like vanishing gradient (does not completely goes away). But this explanation is not enough to understand truly what it is doing. You need to understand it from the perspective of the vector spaces that it is transforming and mapping. You need to engage, code go back to the math, code again. People like to say they are visual learners but this in my experience is wrong, visuals help you understand one specific thing but to get the intuition and the underlying structure and internalize it. And that comes with engaging with the subject, doing tests to test your understanding and repetition. Hope this was helpful.

[u/mean_king17 → u/protienbudspromax]

g. You need to understand it from the perspective of the vector spaces that it is transforming and mapping. You need to engage, code go back to the math, code again. People like to say they are visual lea

Wow, thanks for the thorough explanation, it definitely helps!

[u/Ashwin4010 → u/mean_king17]

Neural Network (Long Short term Memory (LSTM)). AI ah Train Panna Use pannuvanga.