codefuseEval/execution.py:680-683 (byte-identical duplicate at codefuseEval_202503/execution.py:681-684):

if metric == "accuracy" or metric == "f1":
    predictions = [0 if prediction == reference else 1 for prediction, reference in
                   zip(predictions, references)]
    references = [1] * len(references)
...
result = metric_module.compute(predictions=predictions, references=references, **kwargs)

When prediction == reference (the generation is correct), this assigns 0; when they differ, it assigns 1. references is then set to all-1s, and the arrays are handed to metrics/accuracy/accuracy.py's Accuracy._compute:

def _compute(self, predictions, references, normalize=True, sample_weight=None):
    return {"accuracy": float(accuracy_score(references, predictions, normalize=normalize, sample_weight=sample_weight))}

sklearn.accuracy_score counts the fraction where predictions[i] == references[i] == 1, i.e. the fraction of mismatches. This is the exact inverse of the contract documented in metrics/accuracy/README.md ("Accuracy is the proportion of correct predictions among the total number of cases processed"). f1 shares the identical transform and inverts the same way.

What happens (reproduced on the real module)

check_currentmetric() in codefuseEval/execution.py:680-683 (byte-identical at codefuseEval_202503/execution.py:681-684, both confirmed against the repo's current HEAD 09a4f2a) inverts the accuracy/f1 polarity: on a match it assigns predictions=0 and sets all references=1, then hands these to the vendored HuggingFace evaluate Accuracy/F1 classes (metrics/accuracy/accuracy.py, metrics/f1/f1.py, unmodified), whose sklearn.accuracy_score/f1_score then measure the mismatch rate rather than the match rate — independently reproduced against real sklearn: a 10/10-correct model scores accuracy=0.0/f1=0.0, a 0/10 or all-empty-output model scores accuracy=1.0, exactly contradicting metrics/accuracy/README.md's documented contract. Both accuracy and f1 are public, unguarded CLI-selectable metrics (METRIC_SUPPORT in evaluation.py) with no caller-side correction, no pinning test, and no existing issue/PR — any invocation with --metric accuracy or --metric f1 (e.g. against the registered spider_test/text2sql dataset, whose post-processor only extracts SQL text and performs no scoring of its own) hits this. The one-line polarity flip suggested in the report (swap which branch maps to 0 vs 1) is the correct fix, matching the already-correct polarity convention used a few lines away in the pass@k path (evaluation.py:477-479).

Happy to open the PR.