A simplified NanoGPT - Like Repo to show that RNNs can compete with GPT. This is RWKV "x051a" which does not require custom CUDA kernel to train, so it works for any GPU / CPU. A rewrite of the RWKV LLM created by Peng Bo, and sponsored by Stability and Eleuther AI. Help to create would be appreciated 🙂. For more information, visit the RWKV GitHub Repo, or join the Discord.
Reference:
RWKV.com ❤️
Dependencies:
$ pip install torch numpy (etc.)
Training Information (Etc.)
- (+) Add Here ...
If you have any more ideas, head over to the RWKV Discord, as thanks to Stability and Eleuther AI, as well as various other sponsors, we now have the capacity to run them. For more information, try the community or the Wiki Page.
List all relavent sponsors and contributors. (Eleuther AI, Stability AI, Community etc.)