Ggml v1.0.0
The ggml 1.0.0 update includes several performance improvements and feature additions, including a new version, removal of deprecated API, and support for YaRN truncation and FP8. It also includes performance work across backends, such as few-row quantized matmuls and MoE fusion on CUDA/Metal, a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels, expanded Hexagon op coverage, and scheduler improvements that avoid uploading unused inputs. Additionally, it includes fixes for various bugs and issues, such as making the host build work with clang-cl on Windows ARM64, adapting examples to the removal of ggml_diag_mask_inf, and adding support for Q5_0 and Q5_1 on Hexagon. Other changes include routing GATED_DELTA_NET to the SpacemiT RVV kernel, adding a Vulkan prefill Intel FA kernel, using the 4x4 simd_gemm tile on ARM NEON, and optimizing prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints. It also includes several fixes and improvements for SYCL, such as making oneDNN FA cache thread-safe and memoizing runtime failures, fixing AOT for more than 4GB, and using cached device-id in set_main_device. Other changes include adding a CUDA debug buffer pool, adding a bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin, skipping empty MoE tiles in MMQ stream-k fixup, and preventing out-of-bounds reads in SOLVE_TRI kernel on Metal. Finally, it includes several other bug fixes and improvements, such as support for batched tiling on Vulkan and Hexagon op coverage expansion.
Read the full article at github.com →