Skip to main content

Overview

This example is based on the evaluator-optimizer pattern, where one LLM generates a response while another provides evaluation and feedback in a loop. This is particularly effective for tasks with clear evaluation criteria where iterative refinement provides better results. Evaluator-optimizer

Example task

This example task translates text into a target language and refines the translation over a number of iterations based on feedback provided by the LLM. This task:
  • Uses generateText from Vercel’s AI SDK to generate the translation
  • Uses experimental_telemetry to provide LLM logs on the Run page in the dashboard
  • Runs for a maximum of 10 iterations
  • Uses generateText again to evaluate the translation
  • Recursively calls itself to refine the translation based on the feedback

Run a test

On the Test page in the dashboard, select the translate-and-refine task and include a payload like the following:
This example payload translates the text into French and should be suitably difficult to require a few iterations, depending on the model used and the prompt criteria you set.