iframe-proxy

monotykamary · 2025-08-01T14:46:28Z

Summary

What kind of change does this PR introduce? Feature - adds signature tool calling capability for LLMs without native tool support
What is the current behavior? Models without native tool calling APIs cannot leverage Ax's tool execution capabilities
What is the new behavior (if this is a feature change)?
- Adds flag to enable tool execution via signature fields
- Tool schemas automatically inject as optional signature fields
- LLM populates tool fields → Ax executes tools → returns combined results
- Works with any model regardless of native tool support
- Zero breaking changes, opt-in via flag
Other information:
- Comprehensive test suite with 12 passing tests
- Working example provided in
- Type-safe implementation with full TypeScript support
- Compatible with streaming and MCP tools
- Enables tool usage for models like older GPT versions, Claude via text, etc.

- Add signatureToolCalling flag to enable tool execution via signature fields - Implement tool schema injection as optional signature fields - Add signatureToolCalling manager for tool execution coordination - Include tool schema converter for LLM-friendly field definitions - Add comprehensive test suite with 12 passing tests - Provide working example with search and calculator tools - Zero breaking changes, opt-in via signatureToolCalling: true This enables models without native tool calling support to leverage Ax's signature architecture for tool execution directly from LLM responses.

monotykamary · 2025-08-01T14:55:57Z

Add comprehensive dot notation support for nested tool parameters in signature tool calling, enabling intuitive parameter access using tool_name.parameter_name syntax. Includes recursive handling of nested objects and arrays, smart field name sanitization, and robust type inference. - Add jsonSchemaToSignature utility for JSON Schema to signature field conversion - Enhance AxSignature.injectToolFields with dot notation parameter generation - Support deep nesting with arbitrary depth and complex array structures - Implement smart fallback logic for tools without parameters - Add comprehensive test suite covering all edge cases and scenarios - Update examples to demonstrate new dot notation capabilities - Ensure full TypeScript type safety and backward compatibility This enhancement enables advanced signature tool calling with complex nested APIs while maintaining the existing interface and adding powerful new parameter injection capabilities.

monotykamary · 2025-08-01T17:55:13Z

I did have to add some sort of jq-like dot notation design. It's currently set specifically for tool calling, but can be plucked out for the regular signature call design as well, with caveats. I'll just leave a possible design approach for it if needed below:

1. Syntax Design

// Current (flat structure)
const sig = AxSignature.create('userName:string, userEmail:string ->
response:string')

// Proposed (nested structure)
const sig = AxSignature.create('user.name:string, user.email:string ->
response:string')

2. Implementation Approach

A. Nested Object Fields

// Input: Flat signature string with dot notation
const signature = AxSignature.create('user.name:string, user.email:string ->
response:string')

// Output: Structured field objects
// user: {
//   name: { type: 'string', isArray: false },
//   email: { type: 'string', isArray: false }
// }

B. Array Support

// Arrays with dot notation
const signature = AxSignature.create('items[].name:string, items[].
price:number -> total:number')

// Output: Array of objects
// items: [{ name: string, price: number }]

C. Deep Nesting

// Deep nested structures
const signature = AxSignature.create('config.api.timeout:number, config.api.
retryCount:number -> result:string')

// Output: Deep nested object
// config: {
//   api: {
//     timeout: number,
//     retryCount: number
//   }
// }

3. Implementation Strategy

A. Signature Parser Enhancement

// Enhanced parser in sig.ts
class AxSignature {
  public static create(signature: string): AxSignature {
    // New: Parse dot notation for nested structures
    const parsed = parseDotNotationSignature(signature);
    // ... existing logic
  }
}

function parseDotNotationSignature(signature: string) {
  // Parse nested field structures
  // Handle arrays with []
  // Build hierarchical field objects
}

B. Field Structure

interface AxField {
  name: string;           // "user.name" or "user"
  path: string[];         // ["user", "name"]
  type: { name: string, isArray: boolean }
  parent?: string;        // "user" for nested fields
  isNested: boolean;      // true for dot notation fields
}

4. Usage Examples

A. Simple Nested Objects

// Input
const sig = AxSignature.create('user.name:string, user.age:number ->
greeting:string')

// Usage
const result = await generator.forward(llm, {
  user: {
    name: "Alice",
    age: 30
  }
})

B. Arrays of Objects

// Input
const sig = AxSignature.create('products[].name:string, products[].
price:number -> summary:string')

// Usage
const result = await generator.forward(llm, {
  products: [
    { name: "Widget", price: 10.99 },
    { name: "Gadget", price: 25.50 }
  ]
})

C. Mixed Flat and Nested

// Input
const sig = AxSignature.create('user.name:string, user.email:string,
count:number -> response:string')

// Usage
const result = await generator.forward(llm, {
  user: { name: "Bob", email: "bob@example.com" },
  count: 5
})

5. Validation & Error Handling

A. Field Name Validation

// Ensure consistent naming
'user.name' → valid
'user..name' → invalid (consecutive dots)
'user.name.' → invalid (trailing dot)
'user.123name' → valid (numbers allowed)

B. Type Safety

// TypeScript integration
interface User {
  name: string;
  email: string;
}

interface Input {
  user: User;
  count: number;
}

6. Backward Compatibility

A. Flat Fields Still Work

// Existing flat signatures continue to work
const sig = AxSignature.create('name:string, email:string -> response:string')

B. Migration Path

// Gradual adoption
const sig1 = AxSignature.create('name:string') // Flat
const sig2 = AxSignature.create('user.name:string') // Nested

7. Implementation Complexity

A. Parser Changes

• Low: Extend existing signature parser
• Medium: Add nested field resolution
• High: Full hierarchical structure support

B. Runtime Changes

• Field access: fields['user.name'] vs fields.user.name
• Validation: Ensure nested structure matches signature
• Serialization: Handle nested objects in results

8. Potential API Design

// Option 1: Direct dot notation in signature
const sig = AxSignature.create('user.name:string, user.email:string ->
response:string')

// Option 2: Builder pattern for complex structures
const sig = AxSignature.create('user:object -> response:string')
  .addNestedField('user.name', 'string')
  .addNestedField('user.email', 'string')

// Option 3: Object literal style
const sig = AxSignature.create({
  user: {
    name: 'string',
    email: 'string'
  }
} -> 'response:string')

9. Performance Considerations

• Parsing: O(n) complexity for signature parsing
• Validation: O(n) for nested structure validation
• Memory: Minimal overhead for nested field storage
• Serialization: Efficient JSON serialization/deserialization

dosco · 2025-08-02T01:58:33Z

I understand this is about adding tool calling when the api does not support it. And you're also proposing exposing this at the signature level. Lets keep these things separate else it'll complicate things. Can you enable debug and post a trace here of what the user -> llm exchange it with this kind of tool calling so I can quickly grok it.

And separately these questions related to exposing it in the signature.

Can you include some tool calling examples in the comments I'm trying to wrap my head around this a bit more. How does something like this below signature do tool calling?

const sig = AxSignature.create('user.name:string, user.email:string ->
response:string')

We have typescript infer the input / output types from a signature string, does that still work. eg. s(input -> outout) gives you { input: string}, { output: string}. There's also a signature builder fluent api f() but it's easy to add support there.
We have streamingForward and even with forward we do streaming parsing out input does this work with this design what does the signature render into on the prompt?

monotykamary · 2025-08-02T02:57:25Z

Oh sure, the magic here is using the LLM's output as input for the tool calling. As such, there needs to be a manager to detect whether the optional fields are filled in and pass them as tool call inputs in their respective handlers:

The vision is to inject the tool call inputs as the LLM's outputs and basically "listen" for them to push to the tool handlers:

// before
const basicAgent = agent('question:string -> answer:string', {
  name: 'basicAgent',
  definition: 'You are a helpful assistant.',
  ai: llm
  functions: [searchTool, calculateTool],
  // no signatureToolCalling
});

// LLM sees:
// Input: { question: string }
// Output: { answer: string }

// after
const smartAgent = agent('question:string -> answer:string', {
  name: 'smartAgent',
  definition: 'You are a helpful assistant that can search and calculate.',
  functions: [searchTool, calculateTool],
  signatureToolCalling: true, // 🔑 This enables dot notation that gets sanitized to snake_case
});

// LLM sees:
// Input: { question: string }
// Output: {
//   answer: string,
//   search_web_query?: string,        // Dot notation field
//   search_web_limit?: number,        // Dot notation field
//   calculate_expression?: string     // Dot notation field
// }

TypeScript should work... I hope 🤞; I work on neovim so I haven't rigorously tested it. injectToolFields() should return and pass AxSignature<_TInput, _TOutput>.

// Original flat signature
const flatSig = s('name:string, age:number -> greeting:string')
// Type: { name: string, age: number } -> { greeting: string }

// New dot notation signature (tool calling)
const toolSig = s('query:string -> response:string')
// Type: { query: string } -> { response: string, get_weather_location?:
string, ... }

// The dot notation fields are automatically injected as optional
// TypeScript infers the complete type including tool fields

I don't think I've touched anything on the streaming side, but it should just work 🤞. The idea is we inject optional signature fields based on the tool's schema into its current signature. It would be equivalent to us manually adding the tools schema to the signature by hand; so if that didn't work, send halp.

monotykamary · 2025-08-02T03:16:40Z

Here's a simple debug script to help visualize it a bit better:

// debug-signature-tool-calling.ts
import { ai, agent } from './src/ax/index.js';
import type { AxFunction } from './src/ax/ai/types.js';
import { AxSignature } from './src/ax/dsp/sig.js';

// Debug-enabled tool calling example
const searchTool: AxFunction = {
  name: 'searchWeb',
  description: 'Search the web for information',
  parameters: {
    type: 'object',
    properties: {
      query: { type: 'string', description: 'Search query' },
    } as any,
    required: ['query'] as any,
  },
  func: async (args: { query: string }) => {
    console.log(`🔧 [TOOL EXECUTED] searchWeb("${args.query}")`);
    return `Found results for "${args.query}" - Top 3: Wikipedia, official site, news articles`;
  },
};

const calculateTool: AxFunction = {
  name: 'calculate',
  description: 'Perform mathematical calculations',
  parameters: {
    type: 'object',
    properties: {
      expression: { type: 'string', description: 'Mathematical expression' },
    } as any,
    required: ['expression'] as any,
  },
  func: async (args: { expression: string }) => {
    console.log(`🔧 [TOOL EXECUTED] calculate("${args.expression}")`);
    // biome-ignore lint/security/noGlobalEval: Safe for demo purposes
    return `Result: ${eval(args.expression)}`;
  },
};

async function debugSignatureToolCalling() {
  console.log('🚀 === DEBUG: Signature Tool Calling Exchange ===\n');

  // Check if API key is available
  if (!process.env.OPENAI_API_KEY) {
    console.log('❌ Please set OPENAI_API_KEY environment variable');
    console.log('   Example: export OPENAI_API_KEY=your_key_here');
    return;
  }

  // Set up AI
  const llm = ai({ 
    name: 'openai', 
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-4o-mini'
  });

  console.log('📋 STEP 1: Original Signature');
  console.log('   Input:  { question: string }');
  console.log('   Output: { answer: string }');

  console.log('\n📋 STEP 2: After Tool Injection');
  const originalSignature = AxSignature.create('question:string -> answer:string');
  const injectedSignature = originalSignature.injectToolFields([searchTool, calculateTool]);
  const fields = injectedSignature.getOutputFields();
  console.log('   Output: { ');
  fields.forEach((field: any) => {
    console.log(`     ${field.name}: ${field.type.name}${field.isOptional ? '?' : ''}`);
  });
  console.log('   }');

  console.log('\n📋 STEP 3: User → LLM Exchange');
  console.log('   User Question: "What is 15 * 23 and search for the tallest building?"');

  // Create agent with signature tool calling
  const debugAgent = agent('question:string -> answer:string', {
    name: 'debugAgent',
    description: 'Debug agent showing signature tool calling flow',
    definition: 'You are a helpful assistant that can search the web and perform calculations. Use the available tools when needed.',
    functions: [searchTool, calculateTool],
    signatureToolCalling: true,
  });

  console.log('\n📋 STEP 4: LLM Response Detection');
  try {
    const result = await debugAgent.forward(llm, {
      question: 'What is 15 * 23 and search for the tallest building?'
    });

    console.log('\n📋 STEP 5: Final Result');
    console.log('   Result:', result);
  } catch (error) {
    console.log('❌ Error:', error);
  }

  console.log('\n✅ === DEBUG COMPLETE ===');
}

// Run the debug script
if (import.meta.url === `file://${process.argv[1]}`) {
  debugSignatureToolCalling().catch(console.error);
}

🚀 === DEBUG: Signature Tool Calling Exchange ===

📋 STEP 1: Original Signature
   Input:  { question: string }
   Output: { answer: string }

📋 STEP 2: After Tool Injection
   Output: {
     answer: string
     search_web_query: string?
     calculate_expression: string?
   }

📋 STEP 3: User → LLM Exchange
   User Question: "What is 15 * 23 and search for the tallest building?"

📋 STEP 4: LLM Response Detection
🔧 [TOOL EXECUTED] calculate("15 * 23")
🔧 [TOOL EXECUTED] searchWeb("tallest building in the world")
🔧 [TOOL EXECUTED] searchWeb("current tallest building in the world")

📋 STEP 5: Final Result
   Result: {
  answer: '15 * 23 = 345. The tallest building in the world is the Burj Khalifa.'
}

(although ideally like this ^ let me fix up the part manager part)

- Add mutual exclusion logic to disable normal tool calling when signatureToolCalling is enabled - Update createFunctionConfig to return empty functions when signatureToolCalling is true - Integrate SignatureToolCallingManager into AxGen class for signature and result processing - Add comprehensive test suite to verify mutual exclusion behavior - Ensure only one tool calling method can be active at a time

dosco · 2025-08-04T08:13:59Z

Couple things I'm thinking about:

The option signatureToolCalling sounds confusing since thats a implementation detail not a thing the user cares about.
Each underlying AI has a getFeatures method that tells us if it supports function calling or not if it's not then maybe we should just use this.
I didn't check but does this support array values etc.

I want to explore exposing the nested signature fields stuff to users for structured extraction. but I want to keep that separate since it's a bit more complex.

dosco · 2025-08-04T20:02:14Z

Will get a release out today I'm calling it functionCallMode : "auto" | "native" | "prompt"

dosco · 2025-08-05T07:57:06Z

It's out can you test and if needed more any fixes

monotykamary · 2025-08-05T08:02:51Z

awesome, works great so far 🚀

dosco · 2025-08-09T16:51:29Z

I'll look into your structured output idea more this weekend inferring the input / output types from the signature is key to if we do it. Also what model do you use this type of function calling with?

monotykamary · 2025-08-10T03:02:56Z

) * feat: add signature tool calling for non-native tool support - Add signatureToolCalling flag to enable tool execution via signature fields - Implement tool schema injection as optional signature fields - Add signatureToolCalling manager for tool execution coordination - Include tool schema converter for LLM-friendly field definitions - Add comprehensive test suite with 12 passing tests - Provide working example with search and calculator tools - Zero breaking changes, opt-in via signatureToolCalling: true This enables models without native tool calling support to leverage Ax's signature architecture for tool execution directly from LLM responses. * feat: implement jq-like dot notation for tool parameter injection Add comprehensive dot notation support for nested tool parameters in signature tool calling, enabling intuitive parameter access using tool_name.parameter_name syntax. Includes recursive handling of nested objects and arrays, smart field name sanitization, and robust type inference. - Add jsonSchemaToSignature utility for JSON Schema to signature field conversion - Enhance AxSignature.injectToolFields with dot notation parameter generation - Support deep nesting with arbitrary depth and complex array structures - Implement smart fallback logic for tools without parameters - Add comprehensive test suite covering all edge cases and scenarios - Update examples to demonstrate new dot notation capabilities - Ensure full TypeScript type safety and backward compatibility This enhancement enables advanced signature tool calling with complex nested APIs while maintaining the existing interface and adding powerful new parameter injection capabilities. * feat: integrate signature tool calling with mutual exclusion - Add mutual exclusion logic to disable normal tool calling when signatureToolCalling is enabled - Update createFunctionConfig to return empty functions when signatureToolCalling is true - Integrate SignatureToolCallingManager into AxGen class for signature and result processing - Add comprehensive test suite to verify mutual exclusion behavior - Ensure only one tool calling method can be active at a time

dosco merged commit 29f4cdb into ax-llm:main Aug 4, 2025
1 check passed

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

monotykamary commented Aug 1, 2025

Summary

Uh oh!

monotykamary commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotykamary commented Aug 1, 2025

1. Syntax Design

2. Implementation Approach

A. Nested Object Fields

B. Array Support

C. Deep Nesting

3. Implementation Strategy

A. Signature Parser Enhancement

B. Field Structure

4. Usage Examples

A. Simple Nested Objects

B. Arrays of Objects

C. Mixed Flat and Nested

5. Validation & Error Handling

A. Field Name Validation

B. Type Safety

6. Backward Compatibility

A. Flat Fields Still Work

B. Migration Path

7. Implementation Complexity

A. Parser Changes

B. Runtime Changes

8. Potential API Design

9. Performance Considerations

Uh oh!

dosco commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotykamary commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monotykamary commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dosco commented Aug 4, 2025

Uh oh!

dosco commented Aug 4, 2025

Uh oh!

dosco commented Aug 5, 2025

Uh oh!

monotykamary commented Aug 5, 2025

Uh oh!

dosco commented Aug 9, 2025

Uh oh!

monotykamary commented Aug 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

monotykamary commented Aug 1, 2025 •

edited

Loading

dosco commented Aug 2, 2025 •

edited

Loading

monotykamary commented Aug 2, 2025 •

edited

Loading

monotykamary commented Aug 2, 2025 •

edited

Loading