createAI: add an evaluation model for Clef (experimental_evaluate) · Issue #2630 · cloudflare/agents · GitHub
Skip to content

createAI: add an evaluation model for Clef (experimental_evaluate) #2630

Open

Description

@mattzcarey

createAI has no evaluation model, so Clef (@cf/cloudflare/clef, @cf/cloudflare/clef-flash) can't be used with the AI SDK's experimental_evaluate (EvaluationModelV4, ai@7.0.103+).

const ai = createAI({ binding: env.AI });
const { answers } = await evaluate({
  model: ai.evaluation("@cf/cloudflare/clef"), // doesn't exist
  state: "Checkout has been failing for every customer for the last hour.",
  questions: { urgent: { type: "boolean", instructions: "Is this urgent?" } }
});

The modality methods live in packages/agents/src/models/ai-sdk/index.ts and ai-sdk/models/modality.ts; there is no evaluation alongside embedding, image, speech, transcription and reranking. cloudflare/ai#670 has a working EvaluationModelV4 for the old package (maps boolean to Clef's noul, forwards images, exposes confidence and legend in provider metadata) that could be ported.

From cloudflare/ai#670.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions