Want to create an interactive transcript for this episode?
Podcast: airhacks.fm podcast with adam bien
Episode: From SIMD to CUDA with TornadoVM
Description: An airhacks.fm conversation with Michalis Papadimitriou (@mikepapadim) about:
GPU acceleration for LLMs in Java using tornadovm,
evolution from CPU-bound SIMD optimizations to GPU memory management,
Alfonso's original Java port of llama.cpp using SIMD and Panama Vector API achieving 10 tokens per second,
TornadoVM's initial hybrid approach combining CPU vector operations with GPU matrix multiplications,
memory-bound nature of LLM inference versus compute-bound traditional workloads,
introduction of persist and consume API to keep data on GPU between operations,
reduction of host-GPU data transfers for improved performance,
comparison with native CUDA implementations and optimization strategies,
JIT compilation of kernels versus static optimization...