Orion-LLM
Built a 107M parameter small language model using a linear attention architecture combining FAVOR+ (Performer) with asymmetric KV projections for sub-quadratic attention. Achieved 23% faster inference than GPT-2 at comparable parameter count; quantized model ran 84% faster. Tech stack: Python, PyTorch.