gemma-4-26b and gemma-4-31b opencode interaction issues leading to tool loops/failures · Issue #21034 · anomalyco/opencode · GitHub
Skip to content

gemma-4-26b and gemma-4-31b opencode interaction issues leading to tool loops/failures #21034

Closed as not planned
Closed as not planned
gemma-4-26b and gemma-4-31b opencode interaction issues leading to tool loops/failures#21034
Assignees
nexxeln

Description

@pchuck
pchuck
opened on Apr 4, 2026

Description

Gemma-4-26b and Gemma-4-31b, even with the latest tokenizer fixes, running on the latest patched engines, is not usable in OpenCode

Additional environment details

  • lmstudio: v0.4.9 - includes: "Improved Gemma 4 tool call reliability"
  • engine: metal llama.cpp v2.11.0 - includes "Gemma 4 improvements"
  • models: google, and unsloth/bartowski imatrix k-quants (including most recent updates as of 2026.04.04 w/ tokenizer fixes)

Model Settings

  • context: 64K
  • temperature: 1
  • top-k: 64
  • repeat penalty: 1.1
  • top-p: 0.95
  • min-p: 0.0
  • flash attention: on
  • cache quantization: off

Plugins

none

OpenCode version

v1.3.13

Steps to reproduce

Configure the model in opencode.json...

            "gemma-4-26b-a4b-it": { 
                "name": "gemma-4-26b-a4b-it",
                "limit": {
                    "context": 65536,
                    "output": 65536
                }
            },

Create a simple test file 'test.txt' in the current working directory

Send a simple prompt in OpenCode

read the file test.txt

Observe the failures:

  • tool failures
  • command repetition on tool success (model not seeing responses?)

(see screenshot)

Screenshot and/or share link

Image

Operating System

MacOS 26.4

Terminal

iTerm2

Activity

  1. pchuck
    added
    bugSomething isn't working
    on Apr 4, 2026
  2. github-actions
    assigned
    rekram1-node
    on Apr 4, 2026
  3. github-actions
    added
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on Apr 4, 2026

  4. github-actions commented on Apr 4, 2026

    @github-actions
    github-actionsBot
    on Apr 4, 2026 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

  5. pchuck commented on Apr 4, 2026

    @pchuck
    pchuck
    on Apr 4, 2026
    Author

    Just following up on the possible duplicates flagged by github-actions.

    This issue is not a duplicate of #20995. In that issue, the (smaller) model is unable to make a tool call (model responds with: "I do not have the capability to execute system commands like listing directory contents.") In this issue, the tool calls are attempted.

    This issue is not a duplicate of #20669. In that issue, the tools are called with invalid arguments ("The bash tool was called with invalid arguments ... expected string, received undefined"). In this issue, tool calls succeed and are repeated numerous times.

  6. pchuck commented on Apr 6, 2026

    @pchuck
    pchuck
    on Apr 6, 2026
    Author

    Updated to metal llama v2.12.0 and tested both unsloth/gemma-4-26b-a4b-it@q4_k_xl and unsloth/gemma-4-31b-it@q4_k_xl.

    OpenCode seems to be working now. Model appears to be parsing tool responses properly. Tool use is working without error, and redundant/repeated tool call loops are no longer occurring.

  7. NikolaKrstic94 commented on Apr 7, 2026

    @NikolaKrstic94
    NikolaKrstic94
    on Apr 7, 2026

    Updated to metal llama v2.12.0 and tested both unsloth/gemma-4-26b-a4b-it@q4_k_xl and unsloth/gemma-4-31b-it@q4_k_xl.

    OpenCode seems to be working now. Model appears to be parsing tool responses properly. Tool use is working without error, and redundant/repeated tool call loops are no longer occurring.

    Same here ,can confirm it works much much better.

  8. eikonomega commented on Apr 7, 2026

    @eikonomega
    eikonomega
    on Apr 7, 2026

    I can confirm this same behavior on PopOS/RTX 5090 setup. Also any request that requires a tool call ends up in an infinite loop with Gemma4 26b and 31b models.

  9. bni commented on Apr 7, 2026

    @bni
    bni
    on Apr 7, 2026

    I see this and then the agent just stops
    <edit_file> <file_path>/Users/bni/Documents/freesynd/engine/src/system/sdl/system_sdl.cpp</file_path> <old_string> #include <SDL_image.h> </old_string> <new_string> #include <SDL3/SDL_image.h> </new_string> </edit_file> <edit_file> <file_path>/Users/bni/Documents/freesynd/engine/src/system/sdl/fstexture_sdl.h</file_path> <old_string> #include <SDL.h> </old_string> <new_string> #include <SDL3/SDL.h> </new_string> </edit_file>

    Running latest llama.cpp like this:

    .\llama-server.exe
    --hf-repo unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL --temp 1.0
    --top-p 0.95 --top-k 64
    --alias "unsloth/gemma-4-26B-A4B-it-GGUF" --host 0.0.0.0
    --port 8080 --reasoning on

  10. FricoRico commented on Apr 7, 2026

    @FricoRico
    FricoRico
    on Apr 7, 2026

    VLLM v0.19.0 still showed issues and so did the nightly from yesterday. I'm unsure if this is an OpenCode problem or an inference side problem.

  11. JackKelly commented on Apr 8, 2026

    @JackKelly
    JackKelly
    on Apr 8, 2026

    I can confirm that I'm seeing similar behaviour with opencode, vLLM v0.19.0, and Gemma 4 26B.

  12. BrutchsamaJeanLouis commented on Apr 8, 2026

    @BrutchsamaJeanLouis
    BrutchsamaJeanLouis
    on Apr 8, 2026

    Same with gemma4-31b tokenizer tool parsing is broken in opencode but works in openwebui. latest llama.cpp b8705

    there is also issue 16531 that blocks gemma4 support

  13. Racemase commented on Apr 9, 2026

    @Racemase
    Racemase
    on Apr 9, 2026

    I have ecountered the same issue with 26B and 32B, works great in chat mode via unsloth or even straight llama-server frontend. But tool usage in opencode is really inconsistent. It will run tools for a bit and then require constant "continue" prompts from the user. Seems like a failed tool call chaining or something?

    Have tried it with multiple different quants, built llama.cpp with lastet gemma patches and toknenizers. Definitely seems to be something off with opencode tool parsing?

  14. i-am-rad commented on Apr 9, 2026

    @i-am-rad
    i-am-rad
    on Apr 9, 2026

    Also having issues on gemma 4 31b-it - just gets stuck thinking in opencode. Still trialing other things. I dropped the reasoning parser and same issue.

    vllm serve google/gemma-4-31B-it \
  --host 127.0.0.1 \
  --port 8000 \
  --api-key your_key \
  --dtype bfloat16 \
  --max-model-len 229376 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 1 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

  15. bbrowning commented on Apr 16, 2026

    @bbrowning
    bbrowning
    on Apr 16, 2026

    There's an issue in the Gemma 4 chat templates in handling the system message sent by OpenCode. The chat template expects the system message to be a string, not a list of text content parts. That's at least part of the confusion these models are running into with OpenCode. Here's a local diff that gets the system prompt rendering properly instead of as an inline JSON array:

    --- chat_template.jinja 2026-04-16 18:46:34.506453697 -0400
+++ gemma_4_e4b_chat_template.jinja     2026-04-16 18:36:00.864299411 -0400
@@ -186,7 +186,15 @@
     {%- endif -%}
 
     {%- if messages[0]['role'] in ['system', 'developer'] -%}
-        {{- messages[0]['content'] | trim -}}
+        {%- if messages[0]['content'] is string -%}
+            {{- messages[0]['content'] | trim -}}
+        {%- elif messages[0]['content'] is sequence -%}
+            {%- for item in messages[0]['content'] -%}
+                {%- if item['type'] == 'text' -%}
+                    {{- item['text'] | trim -}}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
         {%- set loop_messages = messages[1:] -%}
     {%- endif -%}

    This may not fix all issues, but it's definitely part of the problem.

  16. bbrowning commented on Apr 17, 2026

    @bbrowning
    bbrowning
    on Apr 17, 2026

    Here's a full chat template with the diff above plus an additional fix that properly handles the empty context blocks OpenCode sends in multi-turn scenarios that was tripping up the Gemma 4 chat template's insertion of the end turn (<turn|>) special token - https://gist.github.com/bbrowning/c584eb2dbd79e4cc9ecedf92eee2d135

    I'm getting decent results from OpenCode even with the smaller Gemma 4 models with the template change.

  17. emansom commented on Apr 17, 2026

    @emansom
    emansom
    on Apr 17, 2026

    Here's a full chat template with the diff above plus an additional fix that properly handles the empty context blocks OpenCode sends in multi-turn scenarios that was tripping up the Gemma 4 chat template's insertion of the end turn (<turn|>) special token - https://gist.github.com/bbrowning/c584eb2dbd79e4cc9ecedf92eee2d135

    I'm getting decent results from OpenCode even with the smaller Gemma 4 models with the template change.

    Good progress! Also working on some fixes, to eventually be upstreamed (if still relevant, and actually beneficial, and upstream agrees etc. etc.) in this branch here: https://github.com/emansom/llama.cpp/tree/gemma4-fixes

    https://github.com/emansom/llama.cpp/commits/gemma4-fixes/

    Give it a try and tell me if it reduces tool call failures! Feel free to send PRs to my fork to improve it further! I'll debug your template improvements mentioned above this weekend and merge those in if need be, my fork also contains a few alterations there.

  18. bbrowning commented on Apr 17, 2026

    @bbrowning
    bbrowning
    on Apr 17, 2026

    Give it a try and tell me if it reduces tool call failures! Feel free to send PRs to my fork to improve it further! I'll debug your template improvements mentioned above this weekend and merge those in if need be, my fork also contains a few alterations there.

    Here's the full diff of the chat template change I made, shown here as a diff against the chat template shipped with the larger Gemma 4 (26B and 31B) models (can be adapted to the smaller ones as well, but the base templates are slightly different):

    --- chat_template_gemma_large_orig.jinja	2026-04-16 19:13:08.903438297 -0400
+++ chat_template_gemma_large_fixed.jinja	2026-04-16 20:21:41.983833992 -0400
@@ -186,7 +186,15 @@
     {%- endif -%}
 
     {%- if messages[0]['role'] in ['system', 'developer'] -%}
-        {{- messages[0]['content'] | trim -}}
+        {%- if messages[0]['content'] is string -%}
+            {{- messages[0]['content'] | trim -}}
+        {%- elif messages[0]['content'] is sequence -%}
+            {%- for item in messages[0]['content'] -%}
+                {%- if item['type'] == 'text' -%}
+                    {{- item['text'] | trim -}}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
         {%- set loop_messages = messages[1:] -%}
     {%- endif -%}
 
@@ -302,19 +310,34 @@
                 {%- endfor -%}
             {%- endif -%}
 
+            {%- set ns_text = namespace(ok=false) -%}
             {%- if message['content'] is string -%}
                 {%- if role == 'model' -%}
-                    {{- strip_thinking(message['content']) -}}
+                    {%- set _s = strip_thinking(message['content']) -%}
+                    {{- _s -}}
+                    {%- if (_s | trim | length) > 0 -%}
+                        {%- set ns_text.ok = true -%}
+                    {%- endif -%}
                 {%- else -%}
                     {{- message['content'] | trim -}}
+                    {%- if (message['content'] | trim | length) > 0 -%}
+                        {%- set ns_text.ok = true -%}
+                    {%- endif -%}
                 {%- endif -%}
             {%- elif message['content'] is sequence -%}
                 {%- for item in message['content'] -%}
                     {%- if item['type'] == 'text' -%}
                         {%- if role == 'model' -%}
-                            {{- strip_thinking(item['text']) -}}
+                            {%- set _s = strip_thinking(item['text']) -%}
+                            {{- _s -}}
+                            {%- if (_s | trim | length) > 0 -%}
+                                {%- set ns_text.ok = true -%}
+                            {%- endif -%}
                         {%- else -%}
                             {{- item['text'] | trim -}}
+                            {%- if (item['text'] | trim | length) > 0 -%}
+                                {%- set ns_text.ok = true -%}
+                            {%- endif -%}
                         {%- endif -%}
                     {%- elif item['type'] == 'image' -%}
                         {{- '<|image|>' -}}
@@ -331,7 +354,7 @@
 
         {%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
             {{- '<|tool_response>' -}}
-        {%- elif not (ns_tr_out.flag and not message.get('content')) -%}
+        {%- elif not (ns_tr_out.flag and not ns_text.ok) -%}
             {{- '<turn|>\n' -}}
         {%- endif -%}
     {%- endif -%}

    The first change in that diff is to handle how OpenCode sends in the system role not as a string (which the template expected) but as a text content part. This was resulting in the entire system prompt from OpenCode getting rendered to the model as a JSON array instead instead of as a simple text prompt.

    The second change in that diff is because OpenCode is not sending in the exact same messages it received, and instead including an empty text content part for the content of every tool call message instead of setting that content field to None or an empty string. So, we have to detect if a message has only empty content and not render the closing turn tag in that case. This fixes an issue where the model was seeing closing turn tags prematurely, while the tool call -> tool response -> tool call -> tool response loop was still ongoing between OpenCode and the model.

    The larger unquantized models actually handled these failure cases reasonably well early in a conversation, but in more complex multi-turn conversations these errors compounded and led to substantial failures. I found these by dumping the model outputs directly from vLLM and dumping the model input inside vLLM to get the full prompt exactly as seen by the model and the full output exactly as generated by the model.

  19. bbrowning commented on Apr 29, 2026

    @bbrowning
    bbrowning
    on Apr 29, 2026

    The official Gemma 4 chat templates have been updated within the last 24 hours with some of the fixes we identified here for OpenCode. I've tested the chat template updates with vLLM and gemma-4-31B-it and things are working very well for me. If you're using OpenCode with vLLM (or any other inference server that uses the model's chat templates), it's worth updating your local copy of the model from HuggingFace again to get the newer changes to chat_template.jinja.

  20. thdxr
    removed
    bugSomething isn't working
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on May 3, 2026
  21. thdxr
    assigned
    nexxeln
    and unassigned
    rekram1-node
    on May 3, 2026

  22. rkuo2000 commented on May 31, 2026

    @rkuo2000
    rkuo2000
    on May 31, 2026

    modifying opencode.json for increasing context window size did not increase ollama model's num_ctx

    journalctl -u ollama -f to check if prompt > num_ctx
    if yes, then

    ollama run gemma4-31b
    /set parameter num_ctx 65536
    /save gemma4-31-ctx64k

    edit ~/.config/opencode/opencode.json to use the newly saved model
    rerun opencode

    Hope this works for you !

  23. Bazmundi commented on Jun 8, 2026

    @Bazmundi
    Bazmundi
    on Jun 8, 2026

    opencode 1.16.2 with ollama 0.30.6 and gemma4:latest exhibits the same problem

  24. bartlettroscoe commented on Jun 28, 2026

    @bartlettroscoe
    bartlettroscoe
    on Jun 28, 2026
    Contributor

    So this does need to be fixed on the llama.cpp side in the same way that it was addressed in vLLM?

  25. houleixx commented on Jun 29, 2026

    @houleixx
    houleixx
    on Jun 29, 2026

    For tool loops, I would try to capture several failed turns and compare them.

    ccglass can show the request body, tool schemas, tool calls/results, response bodies, token/cache/cost, latency, and diffs between turns.

    Project: https://github.com/jianshuo/ccglass

    That may help determine whether Gemma is emitting malformed tool calls, whether OpenCode parses them incorrectly, or whether repeated tool
    results are being fed back into later context.

  26. github-actions commented on Aug 28, 2026

    @github-actions
    github-actionsBot
    on Aug 28, 2026 – with GitHub Actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

  27. github-actions
    closed this as not plannedon Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions