How to Train an LLM From Scratch: A Practical Step-by-Step Guide

Large Language Models (LLMs) power chatbots, code assistants, search, and countless other products. Training one sounds mysterious (billions of parameters, GPU clusters, weeks of compute), but the ideas are learnable, and you can run a real training job on your laptop or a single GPU using open-source tools.

This guide starts from zero: what an LLM is, the vocabulary you will hear in every paper and tutorial, how full training differs from fine-tuning, and a complete hands-on example fine-tuning DistilGPT-2 (a small open-source model from Hugging Face) on your own text with LoRA, a modern, efficient training method.

1. What Is a Large Language Model?

An LLM is a neural network trained to predict the next piece of text (usually the next token) given everything that came before. After seeing enormous amounts of text during training, it learns grammar, facts, reasoning patterns, and style, not by memorizing sentences blindly, but by compressing statistical structure into billions of adjustable numbers called parameters (weights).

Most modern LLMs use the Transformer architecture (Vaswani et al., 2017). Transformers rely on self-attention: each token can “look at” other tokens in the sequence to decide what context matters. That design scales well and is why models like GPT-2, Llama, Mistral, and Phi are all transformer-based.

2. Essential Terms (Glossary)

Term	Plain-English meaning
Token	A chunk of text (word, sub-word, or symbol). “Hello” might be one token; “unhappiness” might split into pieces.
Tokenizer	Converts raw text ↔ token IDs the model understands.
Pre-training	Training from random weights on huge general corpora (web, books). Teaches “language” broadly. Very expensive.
Fine-tuning	Continuing training on a smaller, task-specific dataset to adapt behavior (support tone, domain, format).
Instruction tuning	Fine-tuning on (instruction, response) pairs so the model follows user prompts.
RLHF	Reinforcement Learning from Human Feedback: humans rank outputs; a reward model guides further tuning (used in many chat products).
Loss	A number measuring prediction error. Training minimizes loss.
Epoch	One full pass over your training dataset.
Batch size	How many examples the optimizer updates on at once.
Learning rate	Step size for weight updates. Too high = unstable; too low = slow or stuck.
LoRA	Low-Rank Adaptation: train tiny adapter matrices instead of all weights; fast and light on memory (Hu et al., 2022).
PEFT	Parameter-Efficient Fine-Tuning: family of methods including LoRA (Hugging Face PEFT library).
Checkpoint	Saved model weights at a point in training.
HF Hub	Hugging Face model & dataset hosting, where we download DistilGPT-2.

3. Three Ways People Say “Train an LLM”

1. Pre-train from scratch: You need terabytes of text, hundreds of GPUs, and months. Organizations like Meta, Mistral AI, and EleutherAI do this. Not practical for most individuals.
2. Full fine-tuning: Update every parameter of an existing model on your data. Better quality possible, but heavy VRAM (often 40GB+ for 7B models).
3. PEFT / LoRA fine-tuning: Update ~0.1–2% of parameters. Fits consumer GPUs; ideal for learning and many production adapters. Our hands-on section uses this.

4. The LLM Training Pipeline (Big Picture)

Even when you only fine-tune, it helps to see where your work sits in the full lifecycle:

LLM lifecycle (conceptual flow)

① Collect & clean raw text corpora

↓

② Tokenize → train Transformer (pre-training)

↓

③ Optional: instruction fine-tuning / RLHF

↓

④ Your step: domain LoRA / full fine-tune on custom data

↓

⑤ Evaluate → deploy (API, local, edge)

What happens inside one training step?

1. Load a batch of text sequences.
2. Tokenizer converts them to input IDs + attention masks.
3. Model predicts next token at each position.
4. Compute cross-entropy loss vs. true next tokens.
5. Backpropagation computes gradients.
6. Optimizer (e.g., AdamW) updates weights (or LoRA adapters only).

5. What You Need Before the Hands-On Lab

• Python 3.10+
• GPU recommended (NVIDIA with CUDA). CPU works for DistilGPT-2 but is slow.
• ~4–8 GB disk for model + libraries
• Accounts: Hugging Face (free); accept model licenses if required

Step 0: Create a project folder and virtual environment

mkdir llm-finetune-lab && cd llm-finetune-lab
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install --upgrade pip

Step 1: Install libraries

pip install torch transformers datasets peft accelerate bitsandbytes evaluate

Note: On Windows without CUDA, install CPU PyTorch from pytorch.org. bitsandbytes is mainly for QLoRA on Linux; our DistilGPT-2 example uses standard LoRA in float32/float16.

6. Hands-On: LoRA Fine-Tune DistilGPT-2 on Custom Text

Model: distilgpt2 (82M parameters, GPT-2 architecture distilled by Hugging Face; Sanh et al., 2019; Radford et al., 2019).
Goal: Teach the model a repetitive style (e.g., short “product taglines”) so completions match your pattern.
Method: Causal language modeling + LoRA via peft and transformers.Trainer.

Step 2: Prepare training data

Create train.txt with one example per line (at least 50–200 lines for a visible effect):

Tagline: CloudSync - your files, everywhere, instantly.
Tagline: BrewMaster - coffee that wakes your ambition.
Tagline: FitTrack - steps today, strength tomorrow.
Tagline: CodeNest - build software in calm focus.

Each line is a self-contained training example. The model learns to continue text in the same format.

Step 3: Training script (train_lora.py)

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling,
)
from peft import LoraConfig, get_peft_model, TaskType

MODEL_ID = "distilgpt2"
OUTPUT_DIR = "./distilgpt2-tagline-lora"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

# Load plain text file (one record per line)
dataset = load_dataset("text", data_files={"train": "train.txt"})
dataset = dataset["train"].train_test_split(test_size=0.1, seed=42)

def tokenize(batch):
    return tokenizer(
        batch["text"],
        truncation=True,
        max_length=128,
        padding="max_length",
    )

tokenized = dataset.map(tokenize, batched=True, remove_columns=["text"])

model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
model.config.use_cache = False  # required for gradient checkpointing compatibility

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["c_attn"],  # attention projection in GPT-2 blocks
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Example output: trainable params ~0.3% of total

training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    learning_rate=2e-4,
    weight_decay=0.01,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=torch.cuda.is_available(),
    report_to="none",
)

collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
    data_collator=collator,
)

trainer.train()
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)
print("Saved LoRA adapter to", OUTPUT_DIR)

Step 4: Run training

python train_lora.py

Watch the console: loss should generally decrease across epochs. If loss is NaN, lower the learning rate (try 5e-5).

Step 5: Inference (test your fine-tuned model)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "distilgpt2"
ADAPTER = "./distilgpt2-tagline-lora"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

prompt = "Tagline: SolarGrid -"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=24, do_sample=True, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

You should see completions that resemble your training lines. Quality depends on data size, diversity, and epochs; this is a teaching setup, not production scale.

7. Scaling Up: Same Pattern for Llama / Mistral (7B+)

The steps are identical; only configuration changes:

• Swap MODEL_ID to e.g. meta-llama/Meta-Llama-3-8B (gated; request access on Hugging Face).
• Use target_modules=["q_proj", "v_proj", "k_proj", "o_proj"] for Llama-style architectures.
• Add QLoRA: load model in 4-bit (BitsAndBytesConfig) to fit 8–16 GB VRAM.
• Use chat templates and JSONL with messages for instruction tuning.
• Consider trl.SFTTrainer for supervised fine-tuning with less boilerplate.

Fine-tuning decision flow

Start → Do you have a strong base model on Hugging Face?

No → use an open model (DistilGPT-2, TinyLlama, Phi-3-mini).

Yes → Is VRAM < 16 GB?

Yes → QLoRA + small batch + gradient accumulation.

No → LoRA or full fine-tune; increase batch size carefully.

Evaluate on held-out prompts → iterate data & hyperparameters.

8. Evaluation and Quality Checks

• Perplexity on a held-out set: lower often means better fit (watch for overfitting).
• Human review: sample 20–50 prompts; score helpfulness, factuality, tone.
• Benchmarks (optional): MMLU, HellaSwag for general models; custom JSON tests for your domain.
• Safety: fine-tuning can amplify biases or jailbreaks; test edge cases before deploy.

9. Common Mistakes (and Fixes)

Problem	Likely cause	Fix
CUDA OOM	Batch too large / full precision 7B+	QLoRA, smaller batch, gradient accumulation, shorter max_length
Model repeats training lines	Overfitting, tiny dataset	More data, fewer epochs, dropout, early stopping
No visible change after training	LR too low, wrong layers targeted	Verify trainable params; increase LR slightly; check data format
Garbled output	Tokenizer mismatch	Load tokenizer from same checkpoint as weights

10. Responsible Training Notes

Training on scraped data raises copyright and privacy issues. Fine-tuning on customer data requires consent and secure storage. Open models ship with licenses (Apache 2.0, Llama Community License, etc.); read them before commercial use. Document your dataset, hyperparameters, and evaluation for reproducibility (Mitchell et al., 2019).

Training a frontier LLM from scratch is a datacenter-scale project, but training in the practical sense (fine-tuning an open model on your data) is something you can do in an afternoon. You learned the vocabulary, the lifecycle from pre-training to deployment, and you walked through LoRA fine-tuning of DistilGPT-2 with Hugging Face. From here, swap in a larger base model, structured instruction data, and QLoRA when you are ready for real domain assistants.

References

• Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
• Hugging Face. (2024). PEFT: Parameter-efficient fine-tuning. Hugging Face Documentation. https://huggingface.co/docs/peft
• Hugging Face. (2024). Transformers documentation. Hugging Face. https://huggingface.co/docs/transformers
• Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. Proceedings of the International Conference on Learning Representations. https://arxiv.org/abs/2106.09685
• Mitchell, M., Wu, S., Zanzani, M., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229. https://doi.org/10.1145/3287560.3287596
• Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730-27744.
• Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
• Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv. https://arxiv.org/abs/1910.01108 (DistilGPT-2 follows the same distillation approach for GPT-2.)
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.
• Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., ... & Rush, A. M. (2020). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38-45. https://doi.org/10.18653/v1/2020.emnlp-demos.6

Read More