Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make drop_prob mutable in DropBlock ScriptModules #28

Open
wants to merge 1 commit into
base: torchscript
Choose a base branch
from

Conversation

mdraw
Copy link

@mdraw mdraw commented Feb 5, 2019

This makes it possible to schedule drop_prob during training when used
in JIT mode. The LinearScheduler module still can't be used in JIT
mode yet, but you can change drop_prob from outside of the model, i.e. in
your training loop.

Note that you can still use Python floats in the constructors, but to change the drop_prob buffer values you now need to pass a scalar tensor, e.g.

from dropblock import DropBlock2D
db = DropBlock2D(drop_prob=.5, block_size=3)
m.drop_prob = torch.tensor(.2)

This makes it possible to schedule drop_prob during training when used
in JIT mode. The LinearScheduler module still can't be used in JIT
mode yet, but you can change drop_prob from outside of the model, i.e. in
your training loop.
@mdraw
Copy link
Author

mdraw commented Feb 5, 2019

I've also tried to make LinearScheduler work in JIT (see https://gist.github.com/mdraw/f5d9b24d8c43731756be9245ac4eeffb), but I'm not sure how the error in line 28 can be fixed.

@mdraw
Copy link
Author

mdraw commented Feb 5, 2019

It looks like buffer mutation is not yet supported at all in ScriptModules, unfortunately: https://github.com/pytorch/pytorch/blob/1409a2afc8d81d911dbcfc2d0b210cf62658237b/test/test_jit.py#L5434.

@miguelvr
Copy link
Owner

miguelvr commented Feb 5, 2019

I am questioning the need of having a ScriptModule LinearSchedule, because it is a training tool, not an inference tool.

You don't really want to use the LinearScheduler during inference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants