Advanced Features
This module explores advanced features of Bland, including custom tools, web agents, and integration with Twilio accounts.
IN THIS MODULE YOU WILL LEARN
Utilize custom tools within Bland AI
Manage web agents effectively
Integrate and manage Twilio accounts
Perform advanced call and batch operations
Lesson 1
Utilizing Custom Tools
This lesson covers the creation and management of custom tools in Bland.
Learning Objectives:
Create and update custom tools using POST operations
Delete custom tools with DEL operations
List and retrieve details of custom tools using GET operations
Custom tools are critical to most advanced pathways. They are configurations that allow your agent to make web API calls during the conversation with your users. Some possible uses for custom tools include:
Sending message notifications (emails/SMS)
Scheduling appointments (or checking availability)
Updating a CRM or internal knowledge base
Creating support tickets in your company’s issue tracking system
How agents Use Custom Tools
During a conversation, the phone agent is constantly trying to determine when to respond. When the phone agent realizes it’s time to respond, it reviews the tools in its “toolbox” and chooses whichever one is most appropriate for the situation..
Examples of these tools are speak, wait, and button press. When you create a custom tool, it is added to the existing toolbox. This gives the agent an additional decision point.
How to Define a Custom Tool
Custom tools are defined via a Bland API request. Once a custom tool has been defined, it is accessible for use in your pathways.
Your description should explain what the tool does and/or outline any special instructions for its use.
Here are the pre-existing default tools with their descriptions:
Speak: Talk to the person on the other end of the line.
Press Button: Press buttons on the phone. Each character is a different button.
Wait: Wait and go silent for an extended period of time. (Note: use this sparingly.)
Finish: Say a goodbye message and end the call once completed.
Custom tools have the following properties:
name - the agent will see this in the list of tools
description - a short explanation of what the tool does
url - web url to where the API call will be made
method - http method used (POST, GET etc.)
input_schema - a JSON schema describing the input data
speech (optional) - a string that will be spoken by the agent while your tool waits for a response
body - the body of the agent’s external API request.
query (optional) - adds query parameters onto the url of your request
response_data - An array of objects that describe how to extract data from the response. Within the response data, you can create variables that the phone agent can reference in its prompt.
Name and Description
The name, plus the description, help the agent decide which tool to use.
Please note that there are certain words that should not be included in your name or description because they can confuse the AI decision-making process. They are:
input
speak
transfer
switch
wait
finish
press
button
say
pause
record
play
dial
hang
Choosing names that are similar to those of default tools can also confuse the agent. Two to three-word names are preferable.
For this example, we’ll set the name to BookAppointment and the description to “Book an appointment for the customer.”
Input Schema
The input schema defines the shape of the API request and the different inputs the request can take. It also includes an example that helps our system when creating requests.
Two important notes about input schema:
input_schema is converted into the variable "{{input}}," which you can use in the request body/query/headers
To access nested properties, use dot notation: "{{input.property.subproperty}}"
This means that later on you can use "{{input.service}}" to schedule whatever type of appointment requested by the customer.
Essentially, you are describing the structure of the variables that the agent will create.
Here’s what the input schema could look like:
"input_schema": {
"example": { // "example" is a special property that shows an example of what the input object the agent creates should look like
"speech": "Got it - one second while I book your appointment for tomorrow at 10 AM.",
"date": "2024-04-20",
"time": "10:00 AM",
"service": "Haircut"
},
"type": "object",
"properties": {
"speech": "string",
"date": "YYYY-MM-DD",
"time": "HH:MM AM/PM",
"service": "Haircut, Coloring, Trim, or Other"
}
}
Note: If you want to gather detailed letter-by-letter information from the user, raise your interruption_threshold parameter to about 200.
Speech
Since requesting external APIs can take a few seconds, we enable you to define a speech property that the phone agent will say while it makes the request.
This speech could be: “Perfect, I'll schedule that right now. Give me just a second.”
In the case of an agent taking an order for a restaurant, the speech could be: “Thank you, placing that order now.”
Dynamic Speech
You can also have the agent generate dynamic speech based on the conversation. This is done by defining the input.speech in the input_schema.
{
"input_schema": {
"example": {
"speech": "Checking your account details right now John!",
"name": "John Doe",
"email": "johndoe@gmail.com"
},
"type": "object",
"properties": {
"speech": {
"type": "string"
},
"name": {
"type": "string"
},
"email": {
"type": "string",
"format": "email"
}
},
"required": ["speech", "name", "email"]
}
}
Body
This will be the body of your request sent to the external API. It can include prompt variables (and likely will for the majority of conversational pathway applications).
// AI-generated input is created as the `input` Prompt Variable - and the structure is defined by the input schema.
// `input` will match the structure of `input_schema.example` if it is defined.
{
"input_schema": {
"example": {
"name": "John Doe",
"email": "johndoe@gmail.com"
}
}
"body": {
"name": "{{input.name}}",
"email": "{{input.email}}"
}
}
Response Data
Once the API request comes back, response data must be extracted and then passed on to the phone agent.
The data field determines how you extract the data and the name field determines the variable name for reference in the prompt.
Here’s an example:
"response": {
"succesfully_booked_slot": "$.success",
"stylist_name": "$.stylist_name", // if your API returns a JSON object with a key "stylist_name"
}
// If the external API response is:
{
"available_times": [
{
"time": "10:00 AM",
"date": "2022-01-01"
},
{
"time": "11:00 AM",
"date": "2022-01-01"
}
],
"store_hours": {
"open": "9:00 AM",
"close": "5:00 PM"
},
"address_info": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA",
"zip": "12345"
}
}
// You can extract new Prompt Variables like this:
{
"response": {
"available_times": "$.available_times",
"store_hours": "$.store_hours",
"address_info": "$.address_info",
"zip_code": "$.address_info.zip"
}
}
// And then it'll automatically replace them elsewhere (like in the `task`/`prompt`)
{
"task": "The store is open from {{store_hours.open}} to {{store_hours.close}}.",
"prompt": "The store is located at {{address_info.street}}, {{address_info.city}}, {{address_info.state}} {{zip_code}}."
}
Timeout
This is the maximum time in milliseconds that the tool will wait for a response from the external API.
If the external API does not respond within this time, the AI will move on to the next tool.
The default timeout is 10 seconds (10000 milliseconds).
To ensure that the agent always waits for a response, set the timeout to an extremely high value, like 99999999.
Create a Custom Tool With a Post Request
We’re going to use our conversational pathway example from the previous module as an example. Say we also want to update our internal CRM with a “note” when the prospect the agent is speaking to expresses interest in any product.
We will have our custom tool send an API request to our CRM and let the prospect on the phone know that we are jotting down a quick note for them.
Headers
Standard headers here for Authorization and Content-Type
Content-Type: application/json
Authorization: [API Token]
Body
The request body incorporates most of the available properties.
Note: those left blank in the example below aren’t relevant to this use case.
{
"name": "CreateLead",
"description": "Create a lead record for a prospect that seems interested in purchasing.",
"speech": "Give me one second to make a note of your interest in your system.",
"url": "https://my-company-crm.io/leads/",
"method": "POST",
"headers": {"Authorization": "Bearer {{api_key}}"},
"body": {
"name": {{name}},
"email": {{email}},
"product_interest": {{product}}
},
"input_schema": {
"example": {
"name": "John Doe",
"email": "jondoe@gmail.com",
"product_interest": "silver package"
},
"properties": {
"name": {
"type": "string"
},
"email": {
"type": "string",
"format": "email"
},
"product_interest": {
"type": "string"
}
}
}
}
Ensuring Your Tool Is Used Properly
The agent looks at the tool’s name and description. It then looks at the current context of the conversation to decide if it makes sense to use the tool at that time.
How Does Information From the Call Get Passed to My Custom Tool’s API Request?
The agent refers to the input schema and the example within it. These provide context that help it decide what data to pass.
After it has made this decision, the agent extracts the information from the transcript and passes it to the request body.
You can improve the accuracy of the input data by providing a detailed example within the input_schema.
Handling Responses
When the API response from the custom tool comes back, the agent can extract the API response and create variables. This can be done within the response_data property.
Once you’ve given the variable a name, you can reference it in the prompt using double brackets ({{}}).
Note: with the current setup, there might be references to variables that have null values.
Here’s the restaurant example prompt:
You are taking a drive-thru order from a customer. Find out everything they want, like a drive-thru cashier. Continue until they say they’re done. Repeat the full order back to them after that, and then ask if that’s correct. If they confirm that it’s correct, then and only then will you place their order using the PlaceOrder tool. After you place it, tell them their order total and ask them to pull forward. Their order price is {{order_price}}
In this example, the order_price will be null until the data comes back. This is because the prompt is structured to have the agent take the order and then use the PlaceOrder tool before responding with the order price.
By the time the phone agent is asked for the order price by the customer, it will have the information.
%22%20stroke-linecap%3D%22round%22%20stroke-linejoin%3D%22round%22%20stroke-width%3D%2216%22%2F%3E%3Cpolyline%20points%3D%2256%20112%20128%2040%20200%20112%22%20fill%3D%22none%22%20stroke%3D%22rgb(255%2C%20255%2C%20255)%22%20stroke-linecap%3D%22round%22%20stroke-linejoin%3D%22round%22%20stroke-width%3D%2216%22%2F%3E%3C%2Fsvg%3E%0A)