Lesson 2

Fine-tuning Each Node

The fine-tuning process is about adding examples to help your agent learn more about what a successful conversation sounds like. In general, the more examples you can provide the agent with, the better it will be at directing the conversation and speaking the way you intended.

If we go to our “Start Node,” we will see three types of fine-tuning examples. 

  • Dialog Examples 

  • Pathway Examples

  • Condition Examples

In this lesson, we’ll fine-tune a few nodes from our SaltyEvents customer service Conversational Pathway.

TABLE OF CONTENTS▼

    Dialog Examples

    In this field we can pass the agent multiple examples of how a conversation sounds under normal conditions. These are simple to format. Under “Conversation History” you put the user’s speech and under “Expected Assistant Response” you put the agent’s.

    Pathway Examples

    Pathway examples help the routing LLM see when it should select the next pathway.   These are particularly useful when your pathway has many different routes to choose from.

    Condition Examples

    Condition Examples are related to “states” of your pathway. They can be in reference to extracted boolean variables or a prompt describing the call.   For example,  “the customer is not satisfied with the answer you provided,”  could be used in a conditional to help the agent decide which node move to next.

    Example: Start Node Fine-Tuning

    This node is crucial to getting our pathway running correctly, as it identifies the type of caller and then routes them.

    Let’s take a look at the prompt for your start node::

    Introduce yourself as the SaltyEvents Customer Support Agent. First ask the user their name.   Then ask the user what is the nature of their call and how you can help with it. Determine if the user is an attendee of an event, a vendor, or a staff member at the venue, also called a host.   

    This is a decent prompt, but it’s fairly generic. It doesn’t cover a lot of “what if” scenarios or  define the main types of callers.  While our agent is equipped to operate without this information, , we still want to help it out by providing some examples.

    Dialogue Examples

    Conversation History: Hello there. Can you help me with my tickets?

    Expected Assistant Response: Ok,it sounds like you are an attendee of one of our events and have already purchased tickets.

    Conversation History: I'd like to get some pricing about booth spaces for the event in Germany please.

    Expected Assistant Response: Great, it sounds like you're interested in becoming one of our vendors for an event. I can definitely help get you connected to someone for pricing information. 

    Conversation History: Hi there. Can you help me?  

    Expected Assistant Response: I will certainly try to. Can you let me know a bit more about why you're calling today? Then I'll see what I can do.

    Pathway Examples:

    As this is a starter node and has three important pathway routes, it is especially important to provide pathway examples. You should recognize the “Selected Pathway” dropdown fields as the labels from each of our possible pathway routes. 

    Conversational History

    Target Range

    I’m having trouble making a payment for my company's booth space. 

    User is a vendor

    I never received my tickets in my email.

    User is an attendee

    I need to speak with someone about the scheduling of hotel room blocks for our event.

    User is host venue staff

    Fine-tuning Workflows

    Fine-tuning can be done both upfront during pathway development and after your agent has started making real calls.  We recommend setting up a feedback loop of some kind during the initial phases of your pathway to: 

    1. Review your agent’s transcriptions and call logs

    2. Identify ideal dialog and exchanges between your agent and users

    3. Add those examples to the nodes where they occurred 

    How Many Examples Should I Give?

    In most use cases, we aim to have around 10 nodes with fine-tune conversation examples.  More complex implementations could benefit from more,  while simpler use cases can see benefits from just a few fine-tuned nodes.

    Fix Your Prompts First

    Before spending a substantial amount of time fine-tuning your nodes with examples, make sure that your base prompts are optimized. Prompt engineering should always be your first method of improving a node’s performance.

    NEXT

    Putting Custom Tools Within the Pathway Examples

    Putting Custom Tools Within the Pathway Examples

    Putting Custom Tools Within the Pathway Examples

    TABLE OF CONTENTS▼
      Scroll to top