How to use pretrain networks like vgg19,resnet18?

I would like to use pretrained network like VGG19, RESNET18. The output of this network should be connected to LSTM.

Which layer should be considered to take output from pretrain network?

[Ashu]

Can you give more clarification on how you want to connect the output. Some use case might help here.

[Shilpa Sonawane]

i am working on the topic of lipreading. i have done preprocessing & have videos which consists of mouth region of person. I used VGG19 pretrain network. Now its output i would like to connect with LSTM network. I am confused from which layer of VGG19 the output is taken for furter processing. I referred many papers in which first pretrain netwrok is used and then its output is applied to LSTM network. LSTM network is used as input to the system is video which is frame sequence.

[Shilpa Sonawane]

yes i tried it, thank you.

Answer 1

[Nadia Shaik]

Hi Shilpa,

I understand that you wish to connect the output of CNN to LSTM network. MATLAB supports workflows containing both CNN and LSTM layers.

You can convert the videos to sequences of feature vectors using a pretrained convolutional neural network, such as GoogleNet, to extract features from each frame. And then, train an LSTM network on the sequences to predict the video labels.

Please check the link that contains an example showing the CNN+LSTM workflow https://www.mathworks.com/help/deeplearning/examples/classify-videos-using-deep-learning.html

Similarly, you can use the convolutional neural networks like VGG19, RESNET18 to extract features and then train an LSTM network.

I hope this helps