Hi Alessandro,

Great tutorial, thank you. I'm new to deep learning. I see that there are 6 different forecast graphs for the test data above, and the truth values are different in each graph. I assume that you take 6 different sequences of the test data, is that right? Can you do the forecast for all the test data at once?

Also, how do you do inference for the future, beyond the test data? Thanks

Hey, thank you so much for your kind words!

Yes, you are perfectly right, as you can see in the code which I used to produce the figure in the colab notebook linked at the bottom of the article, I am just taking some random slices of my test set and showing the result of the model.

Notice that since I built my dataset as a tf.data.Dataset, the test data can be shuffled easily and the random slices are just extracted via test_windowed.take(6).

It is definitely possible to produce a forecast for all test data at once with this model, but my intuition tells me that it won't be any better than doing a linear regression using only as features the "deterministic" features of future time steps, and where we forecast each time step independently from the others. I might be wrong tho, so that might be a good experiment to run if you're starting out with these topics!

Regarding inference for the future, you do it in the same way you predict for stuff in the test set, i.e., just call model.predict((past_features, future_features)) where past_features and future_features are properly-shaped tensors (see the starting part of this post and the create_dataset function) encoding respectively your knowledge of the past and partial knowledge of the future. Of course, you will not know the ground truth, but as the motto goes "It's tough to make predictions, especially about the future".

Wish you all the best!

Hi Alessandro, thanks for the post. I followed what you said about making inferences for the future but am still having trouble doing such. Would you be able to add code of such? Thank you.

Sure! Although to do so, I must first understand where the problem lies. Which kind of trouble are you encountering? Which kind of data are you trying this on, and is your issue about massaging tensors in the correct shape?

Thank you for the detailed answer.

Hey Alessandro, Thank you for this Great post,
I noticed you have one time-step output, I didn't understand how you push the forecasted values as input to the next forecasting point in the test phase, would you please add that code to see how is going to fit in that pipeline. I will appreciate that.

Dear Hamza RABI , thank you for the encouragement! :-)

The full code used for this article is in the colab notebook linked at the end of the post, so you might be interested in checking that out.

Regarding your question, I didn't feed forecasts back again as features. Rather, LSTM cells have a sort of "memory", so each time you feed them new features (i.e., at each time step) information related to previous forecasts can be preserved in the state of the cells. If you want I can give you some references, but maybe check out this classic post from the legendary Andrej Karpathy for more about this.

Oh I'm sorry I didn't read the code carefully, I was a little bit confused with that one unit Dense layer. but now I checked the output shape in colab and it is (None, 120, 1) so that makes total sense.
thank you again. very helpful post.

Hi Alessandro,

Thank you for your tutorial.
If I don't need future feature, how should I change the model structure.
Thank you very much.

Great post! I have a question regarding the scaling. Don't you consider as the best practice to use only the training set for normalisation? I think the way you did it, you have data information leakage from train to test set. Am I missing something with time series?

Dear George,

oh my gosh, you are perfectly right! This specific mistake is one I really hate when I see it "in the wild", and I made it myself :-(
Yes, info about the test set leaks into training/validation sets via the normalization. Best practice would be to introduce a normalization layer trained only on the test set; will try to fix the article asap... For now, thanks a lot for pointing it out!

Hi, Alessandro, thanks for sharing. I have a question regarding the output dense layer. Since we do a 120 steps ahead forecasting, why there is only one unit in the last output dense layer? I thought it would be something like 'output = tf.keras.layers.Dense(120, activation='relu')(x)'.

Hey Jin Liu! Good question. Notice that in this model we are using recurrent layers (LSTMs). In particular, the decoder_lstm layer has the parameter "return_sequences" set to true. You might want to play with the notebook linked at the bottom of the article, where you can find all the code I wrote for this. In particular, notice that if you look at the printout from a call to model.summary(), the output shape of the LSTM layer is a tensor of dimension (None, 120, 16); neglecting the first axis (which is related to batch dimension) this means that it returns 120 vectors (one for each time step in the future) with 16 components

Finally, the encoder-decoder LSTM Keras tutorial that I expect, the explanation is very clear, thank you very much.

Anyway I've ever seen in some encoder-decoder LSTM architecture image, that the LSTM decoder part receive the input from the encoder LSTM output repeatedly rather than from external future features. Is it also considered as encoder-decoder LSTM? What is the difference? Thanks in advance
https://uploads.disquscdn.c...

After months of looking for the way how the exogenous regressor can be implemented in LSTM, I finally found your post. Thank you good sir for this.

Hi Allessandro,

Great overview of the capabilities of Encoding - Decoding LSTM models.

I have some troubles seeing the whole code and I get the following error message when trying to open the link posted:

Loading...
This site may not work in your browser. Please use a supported browser. More info

Notebook loading error
There was an error loading this notebook. Ensure that the file is accessible and try again.
'TextDecoder' is not defined
https://drive.google.com/dr...

Details

'TextDecoder' is not defined
ReferenceError: 'TextDecoder' is not defined
at Anonymous function (https://colab.research.goog...
at Fa (https://colab.research.goog...
at Da.prototype.next_ (https://colab.research.goog...
at next (https://colab.research.goog...
at b (https://colab.research.goog...

I am unsure if the file still exists or if there is a problem from my end?

Hey Andreas Alberg-Fløjborg! Sorry for the late reply, for some reason your comment was marked as spam (maybe because of the links it contains).

Anyhow, yes, the notebook is still there, and from the error message you see it appears to be some problem on your end. May I ask which browser/os are you using? Are you able to run any kind of code on google colab? ( https://colab.research.goog... )

Thank you for your reply.

It took me some time to figure out as you suggested that the issue was at my end. Apparently the default browser couldn't open google colab and I had to switch to chrome.

I now have full access and can read the entire code.

Great post! Do you have any suggestions on how the model should be created if there are no deterministic features in the future?

Thank you for your kind words! In that case I think you can just use a "vanilla" LSTM, or, even better if you do not have a ton of data, just some ARIMA model. You pass information only up to some "present" time, and then the model should return a sequence of predictions

Hey, thanks for the nice tutorial, although it seems a bit advanced for me, is there any resources to read so I can understand the basics of the encoder decoder architecture applied to LSTM? And by the way nice that you covered the probability distribution prediction, no other tutorial did that :p

Hey Abdessalem Boukil ! Thank you very much for your kind words. So, it depends a bit on which level of understanding you have of both concepts, but for sure I can recommend you "Sequence to Sequence Learning with Neural Networks", by Sutskever et al., and references therein. Alternatively (and perhaps an even better place where to start from) take a look at this blog post https://blog.keras.io/a-ten...

Hope to have helped you; feel free to reach out in case you want to discuss this further!