Skip to content

Added parameter checks for optimizers - #2

Open
lazypanda1 wants to merge 4 commits into
masterfrom
optimizer-fixes
Open

Added parameter checks for optimizers#2
lazypanda1 wants to merge 4 commits into
masterfrom
optimizer-fixes

Conversation

@lazypanda1

@lazypanda1 lazypanda1 commented Mar 15, 2018

Copy link
Copy Markdown
Owner

This PR adds parameter range checks to all optimizers to ensure that end-users do not end up providing invalid values to the optimizers and be confused by the output when there is no actual problem with their model.

For example, running the following program produces NaNs in the output, due to invalid value of rho (>1.0).

import torch
from torch.autograd import Variable

N, D_in, H, D_out = 64, 1000, 100, 10

x = Variable(torch.randn(N, D_in))
y = Variable(torch.randn(N, D_out), requires_grad=False)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(size_average=False)

learning_rate = 1e-4
optimizer = torch.optim.Adadelta(model.parameters(), lr=learning_rate, rho=1.1)
for t in range(2):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    print(t, loss.data[0])
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Output:

0 651.8707885742188
1 nan

I tried adding constraints for all the parameters that I could infer from the corresponding articles, but I am still missing some. Please feel free to suggest what should be bound for the ones which are missing.

This is similar to the bounds check which I added for Adam Optimizer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant