Problem Statement

Currently we use llama.cpp Vulkan build for AMD GPUs. HIP build should have better perf. (we should have benchmark test for that too)

Once #2128 lands, we should be able to use upstream HIP builds for Windows

There is no upstream HIP builds for Linux. We might need to build it ourselves, though I think we need AMD GPUs for that.

cc: @david-menloai @sangjanai @qnixsynapse

Feature Idea

See above