Tiny hackable CUDA language model implementation

Posted by markusheimerl 4 days ago

Comments

Comment by yobbo 2 days ago

Looks very nice, but I can't find numerical gradient checks, which is helpful when verifying that backward pass is correct:

https://github.com/markusheimerl/gpt/blob/main/transformer/a...

Comment by markusheimerl 1 day ago

I deleted the numerical checks a while back after confirming the backward pass is correct to keep the code base lean - running https://github.com/markusheimerl/gpt/blob/main/transformer/a... is also somewhat of a confirmation that the backward pass is correct, since an analytically incorrect backward pass cant fit perfectly to synthetic data.

Comment by Gred_papa_dance 1 day ago

I need more info:

* where is data (make data) how create new my own data, (questions for chat?) * how create a tokenizer (meybe separate) * how stop the code, how many memory need, how setup size of context etc. * how creating a LORA or learn with new data. * how quantize model?

In my opinion this is great idea but making a Ruby extension will be goot way to increase users using this code.

Comment by markusheimerl 1 day ago

the data gets downloaded via curl from huggingface - sure you can make your own data, simply dump all text you want the model to be trained on into "corpus.txt" and skip "make data".

As the tokenizer adds substantial complexity, this implementation does not include any tokenziation logic and works on raw bytes. Feel free to add your own tokenzier with the help of the coding model of your choice.

You can stop the training using CTRL+C You can train on as little memory as you have. Simply reduce batch size and/or model dimensions in train.c You can change the context window size in train.c via the "seq_len" variable.

Regarding Ruby, LORA and quantization I'll have to refer you to the coding agent of your choice.

Comment by ewew53 1 day ago

Meybe add a simple step betwen start and train:

convert text data to binary data. This help converting a differend data.

(please make 8 bit format, 16, 32 bit format)

Comment by qqqqqlqq 1 day ago

$make run -j 10

CUDA error in attention.c:91: out of memory

Command exited with non-zero status 1

1.38user 0.46system 0:00.75elapsed 246%CPU (0avgtext+0avgdata 226164maxresident)k

0inputs+0outputs (0major+25414minor)pagefaults 0swaps

make: ** [Makefile:34: run] Błąd 1

clang: warning: CUDA version 12.4 is only partially supported [-Wunknown-cuda-version]

(I have ubuntu and 8GB memory NVIDIA GeForce RTX 3050 876MiB / 8192MiB )

Comment by markusheimerl 1 day ago

Reduce batch size in train.c

Comment by ewew53 1 day ago

how to do this? I have this same error

Comment by markusheimerl 1 day ago

https://github.com/markusheimerl/gpt/blob/main/train.c - in this file, search for the line "const int batch_size = 15;" - reduce this number

Comment by oakinnagbe 1 day ago

Nice implementation. Have you thought about supporting LoRA fine-tuning on top of this, or is the design too low-level for that kind of extension?

Comment by markusheimerl 1 day ago

Sure it could be extended to support LoRA finetuning but this implementation has the goal to be as lean and efficient as possible for a pre-training stack as you can be.

Comment by 4 days ago

Comment by qqqqqlqq 1 day ago

It works on arm ?

Comment by markusheimerl 1 day ago

I did run it as a test on the NVIDIA Jetson Orin Nano Super Dev. Kit once - so yea it works on arm like a charm ;)

Comment by FastAnchor 1 day ago

[flagged]