![]() Please note, that it really doesn’t matter here if you preprocess using other methods too. So now as we know how to preprocess our data we will get into the actual code for preprocessing steps. This part is covered more in detail in my last post on transformers, so if you are feeling confused here, I would ask you to take a look at that This is intentional as we want to start the target sentence with some start token(so 2 is for token) and end the target sentence with some end token(so 3 is token) and a string of blank tokens(so 1 refers to token). All sentences start with a word whose index in german vocabulary is 2 and they invariably end with a pattern. If you notice that there seems to be a pattern to this particular matrix. Here also the numbers in this matrix correspond to words based on the German vocabulary we will also need to create. The Shifted Target German sentences(Target): A matrix of shape (batch size x target sentence length).Also, do you notice that a lot of sentences end with a word whose index in vocabulary is 6? What is that about? Since all sentences don’t have the same length, they are padded with a word whose index is 6. So for example, 234 in the English vocabulary might correspond to the word “the”. The numbers in this matrix correspond to words based on the English vocabulary we will also need to create. ![]() ![]() The Source English sentences(Source): A matrix of shape (batch size x source sentence length).
0 Comments
Leave a Reply. |