Want to create an interactive transcript for this episode?
Podcast: Chaos Computer Club - recent audio-only feed
Episode: Gompertz Linear Units (GoLU) (sps25)
Description: Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as `GoLU(x) = x Gompertz(x)`, where `Gompertz(x) = exp(−exp(−x))`. The GoLU activation leverages the...