This is my research for my CS Research Paper and hobby. I'm creating a transformer/LLM from scratch with decoder architecture. I am following O'reilly's NLP book for guidance to create this transformer, which should be a <100 million parameter model, that generates Python code based on prompt simillar to GitHub co-pilot. Resource used for now is 4x Nvidia A100 80GB from Nvidia.
To Hack Club reviewer, I am just copy-and-pasting this text here for all of my repository involved in HC. However, here's basically the thing, I will be explaining my repository via a youtube stream, since that's the easiest way. For certain repositories, it should be fairly straightforward, like modelling in blender, 3d printing, woodworking, etc. hence the time span will only be about 2-3 minutes. However, for my AI projects, more specifically LLM training from scratch projects and courses. The issue is that you guys, reviewers, can't really experience it. Not that you can't, but it's impossible, unless you guys have access to renting cloud supercomputers, such as 8x Nvidia A100s or 8x Nvidia H100s, which are also very hard and tedious to use, as you need to get the right extensions in VS-code, need to get used to Ubuntu, how cloud computers work, and most finally, money, you need to PAY to have access to these computers. Hence, I have decided to do the next best thing, which I had spent 4 hours, for you guys to be able to watch my process of doing everything here, like downloading the raw data by running certain files in my respective repos, and then trying to get the GPUs setted up with specific libraries and hitting run. (pls give me my printer :( I rlly worked hard for this brah). Anyways, below are the respective links to my youtube stream, and their respective timestamps of when I was explaining or showing the experience for each repository/projects. I was also forced to do this across 4 streams, because my streams are only limited to 1 hour since I don't have enough subscribers for 12+ hour streams. For this project:
https://www.youtube.com/watch?v=NQfp2KT9kEU 46:06 to 1:00:00
https://www.youtube.com/watch?v=1OhGDyw3_eo&t=3s 0:00 to 46:36