AutometicTimeUnrollOfFeedbackNetworks

[ParthaEth]

No description provided.

[0bserver07]

Hey @ParthaEth, in my opinion, this would be better of as a pull-request on Keras, the contributors might have either tried it or getting better support.

Also one of the Keras Variational Autoencoder implementations on github does contain an approach to do Recurrent Latent Variable, maybe check those out first.

[ParthaEth]

@0bserver07 Making a PR to the keras repo is the plan. But it would be great to have a team to work with rather than trying it myself. This functionality isn't there in keras at this point of time. I am trying to find a few people who would be interested in doing the project with me here. Do check the conversation here.

about "Also one of the Keras Variational Autoencoder implementations on github does contain an approach to do Recurrent Latent Variable, maybe check those out first."
are you talking about this here? If so then I don't see the 'Recurrent' part. Could you point to any specific code please?

[farizrahman4u]

@ParthaEth Try RecurrentShop

[ParthaEth]

@farizrahman4u any plan on merging this repo into keras itself? Looks like it does what I am trying to program into keras. What was the motivation to have it as a separate repo? If you/we make a PR with this code into keras it will be reviewed by more people and perhaps more tests will be written and things in general will become better. It is also not nice to have things in bits and pieces all over as if keras decides to make a change in its code you will have to keep an eye on to make this repo compatible again.

[ParthaEth]

@farizrahman4u Could you have a look in here as well. Some how I am not able to get my point understood but may be you can rephrase it better.

[farizrahman4u]

"Support depth first RNN stack computation in Keras"

[ParthaEth]

@farizrahman4u exactly that is necessary for this kind of architectures. but we can decide to do depth first computation if we see a usage of feed back layer or a feed back connection. But you've already implemented this I guess so why not merge RecurrentShop into keras? More reviews and tests would make it even better. I guess ppl are alo talking about giving keras official google support so things will be great.

[ParthaEth]

@farizrahman4u sorry to have not included a link where I wanted to point you in my last to last comment. I wanted to actually look at this issue.

[ParthaEth]

Any one else interested in working/ exploring how to bring in support for RNNs in keras? I guess if we have things that can be done in TF but not in Keras then we defeat the purpose of keras being a research platform and make it rather a industrial user platform. The whole idea is to let one build random things and try it out quickly even though it is not well established architecture.

[0bserver07]

Hey @ParthaEth, with the suggestions given from @farizrahman4u who have been a long contributor to Keras, it looks like that partially of what you have suggested is there.
Maybe a pull-request to do that would be cool? To avoid doing redundant work at the mean-time.

It looks like that @farizrahman4u have started another project here as well :) keras-contrib. Maybe a pull-request on that as well would be relevant.

For research, if you're interested in implementing relevant Keras or TF work, there is a great list of Research Topics that are actively being researched and developed (check them out):

1. Chromosome Segmentation
2. Social media botnet detection and analysis
3. Identifying biomedical articles at risk for retraction
4. Multitask and Transfer Learning
5. Layer-wise supervised incremental training of residual networks

[ParthaEth]

@0bserver07 I saw keras-contrib too the purpose of it is I guess to try out new keras features before it goes official. So yeah i guess every PR will go to this repo first. If I understand correctly, @farizrahman4u 's suggestion was to try RecurrentShop, which is nice of course but the idea of having all my network inside a single recurrent layer is what bothers me. I was thinking to automatically unroll RNNs over time when Model(input=I, output=O) is called. Also implementing a feed back layer to let people create feed back over time loops. With all these a new RNN might look something like the following.

input = Input(batch_shape=(2, 5), dtype='float32', name='current_pose_input')

f1 = Feedback(initial_tensor=Zeros(dim))

o = LSTM(10, f1)
o = TimeDistributed(Dense()(o))
f1(o)

m = Model(input=input, output=o)
m.compile('rmsprop', 'mse')

[cpernoud]

@ParthaEth I wonder if there couldn't be a more explicit and expandable solution.

I happened to notice this on Tensorflow 1.1 "Added a new lightweight queue-like object - RecordInput." I was thinking of something like a queue or accessible memory location. The trick of course is it would need to be used in such a way that it was able to read/written to at the right part of the process.

The benefits I can imagine are things like accessing a multiple-length queue to pick the item you want, or the hard-attention kind of explicit memory access. Of course, we could just do the simple 1-length queue to get back to your example.

I'm in favor of your cause here. I'm surprised there's also not a easy way to feedback your last error.

Thanks for getting the conversation started.

[ParthaEth]

@cpernoud Glad that you find this interesting. I am sure it is faced by many researcher working with RNNs. I have a feeling RNN are receiving far less attention than they really should. I am not sure how best to approach this. Honestly I do not have to time at the moment to develop the feed back layer my self. So I just kind of pushed it in a todo list. :)