Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tweak the ConvLSTM architecture until it works #14

Merged
merged 42 commits into from
May 28, 2024
Merged

Tweak the ConvLSTM architecture until it works #14

merged 42 commits into from
May 28, 2024

Conversation

rhaps0dy
Copy link
Collaborator

@rhaps0dy rhaps0dy commented May 9, 2024 •
edited
Loading

  • Ability to load from a checkpoint (and continue training)
  • Lots of experiments trying things (probably no need to review them)
  • make ConvLSTM residual

rhaps0dy added 30 commits May 2, 2024 12:01
@rhaps0dy
Implement LSTM
4717422
@rhaps0dy
LSTM reference test
f677f9d
@rhaps0dy
Tests say it's fine.
9a9d233
@rhaps0dy
Fix ConvLSTM config
94be8b4
@rhaps0dy
Check that the CLI parses without launching it
c39576e
@rhaps0dy
Can we straightforwardly compensate for LSTM low variance?
0cd099b
@rhaps0dy
Incorporate the vision shortcut
2a54374
@rhaps0dy
Hackily set up the devbox
155a9df
@rhaps0dy
Remove multiplier
47176e3
@rhaps0dy
Try and make the ConvLSTM more balanced in variance
64e701e
@rhaps0dy
Use horizontal connections for pool-and-inject
aa179e9
@rhaps0dy
pool_and_inject_horizontal
a11fd26
@rhaps0dy
Squash missing head_scale bug
1cc9423
@rhaps0dy
Get rid of generics and work around NiklasRosenstein/python-databind#66
9b4b8dd
@rhaps0dy
Try more things
765482e
@rhaps0dy
Change notebook params
31cc510
@rhaps0dy
Torch initialization maybe
1436bfd
@rhaps0dy
Go back to separated convs
1f45d9e
@rhaps0dy
More architectural variation for the ConvLSTM
760f29f
@rhaps0dy
Fencepad
d4ae3c3
@rhaps0dy
Copy hyperparams from the best lstm runs
d3587f3
@rhaps0dy
Config reflects actual practice
0fd2af6
@rhaps0dy
Wild variations experiment
a5326b8
@rhaps0dy
Run good one for longer
cfd7831
@rhaps0dy
Store RMS gradient of every parameter
e9dc974
@rhaps0dy
Have to commit new variations
bda7fd3
@rhaps0dy
Try updating the actor again
0ec2666
@rhaps0dy
What if it's a residual LSTM
497430b
@rhaps0dy
Try again
ac9b919
@rhaps0dy
Tweak the residual stream
2803e78
rhaps0dy
rhaps0dy commented May 9, 2024
View reviewed changes
pyproject.toml Outdated Show resolved Hide resolved
@rhaps0dy rhaps0dy requested a review from taufeeque9 May 22, 2024 02:28
taufeeque9
taufeeque9 approved these changes May 23, 2024
View reviewed changes
Copy link
Contributor

@taufeeque9 taufeeque9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Haven't checked the experiments file though.

pyproject.toml Outdated Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
@rhaps0dy rhaps0dy merged commit 51ccefb into main May 28, 2024
1 check failed
@rhaps0dy rhaps0dy deleted the aga/lstm-var branch May 28, 2024 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taufeeque9 taufeeque9 taufeeque9 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rhaps0dy @taufeeque9