Martin Loretz
Hey, I'm just a human trying to figure out how this universe works. I love engineering and building stuff, like a computer out of logic gates, a screaming plant or a motorized bobbycar. I'm interested in physics, rockets, artificial intelligence and molecular biology.
Blog
Accelerate CPU based LLM Inference with a Vector Index on the Output Embeddings (up to 28% for llama 3.2 1B)
15/01/2025 - The input and output embedding layers of transformer-based LLMs are massive. For example, GPT-2's embeddings make up 38.5M out of 124M total parameters.
Exploring the activation ratios of different MLP layers
21/09/2024 - I've become interested in understanding how many neurons in a layer (Linear + ReLU) actually contribute to the next layer versus those that remain inactive after the activation function.
Polynomials as weights and activations for MLP's
03/05/2024 - I recently read the Kolmogorov-Arnold Networks paper and I'm fascinated by the idea of moving from fixed activations on the nodes of a MLP to learned ones at the edges. Combined with some other thoughts, I applied this idea to image classification and in this process came up with the idea of polynomial MLP's.
Random thoughts:
If you shake a box of lego long enough, at some point it will be assembled by chance.
Life 4.0 will be able to rewrite the universe by exploiting the simulation and changing the rules of physics.
I think turing was right in the sense that if you can't tell a difference, there is no difference.
Stuff worth checking out:
Talks and awards:
Cool stuff I've built:
