PyTorch Module
PyTorch Generator
Generate a ready-to-run PyTorch project in seconds.
11 flags 7 files
Template Library
Pick a blueprint and let the config snap it into place.
Distillation — PyTorch
Distillation with PyTorch
Distill GPT-4/Claude/Gemini into a GPT-2 student built from scratch. Full control over every tensor.
15 flags 11 files
Distillation — HuggingFace
Distillation with HuggingFace
Distill API teachers into a pretrained HF student (DistilGPT2, GPT-2, etc.) using HF Trainer.
14 flags 9 files
Fine-tuning — HuggingFace
SFT with HuggingFace TRL
Fine-tune any HF model with SFTTrainer. LoRA/QLoRA, 4-bit/8-bit quantization, packing, chat templates, push to Hub.
33 flags 6 files
Fine-tuning — Unsloth
SFT with Unsloth
Fine-tune 4x faster with Unsloth. 4-bit LoRA, RSLoRA, gradient checkpointing, GGUF export.
30 flags 6 files
RL — HuggingFace
RLHF with HuggingFace TRL
Preference tuning with DPO, GRPO, PPO, ORPO, or KTO. Full TRL trainer configs, LoRA/QLoRA support.
36 flags 6 files
RL — Unsloth
RLHF with Unsloth
DPO, GRPO, or ORPO at 4x speed with Unsloth. 4-bit LoRA, merged/GGUF export, push to Hub.
30 flags 6 files
Data Parallelism
Data Parallelism (DP)
Pure PyTorch distributed data parallelism: replicate model, shard batches, all-reduce gradients across GPUs.
12 flags 8 files
ZeRO Optimizer
ZeRO (Stages 1-3)
ZeRO optimizer/gradient/parameter sharding with pure PyTorch. Select stage 1, 2, or 3 to trade communication for memory.
13 flags 9 files
Tensor Parallelism
Tensor Parallelism (TP)
Split weight matrices across GPUs with column/row parallel linears and attention head splitting. Pure PyTorch.
12 flags 9 files
Pipeline Parallelism
Pipeline Parallelism (PP)
Split model layers across GPUs with micro-batch pipelining. GPipe (fill-drain) and 1F1B schedules. Pure PyTorch.
13 flags 9 files
Sequence Parallelism
Sequence Parallelism (SP)
Shard sequence dimension for LayerNorm/dropout, paired with tensor parallelism. Reduces activation memory. Pure PyTorch.
12 flags 10 files
Expert Parallelism
Expert Parallelism (EP)
Mixture-of-Experts with all-to-all token dispatch, top-k routing, load balancing loss, and shared experts. Pure PyTorch.
16 flags 9 files
HuggingFace Pipeline
HuggingFace Generator
Generate a HuggingFace pipeline project with model loading, inference, and configuration.
6 flags 5 files
GPU Kernel Blueprint
Custom CUDA Kernel
Generate custom CUDA kernels with automatic benchmark harness, Makefile, and PyTorch integration.
4 flags 4 files