🛠️ How to Fix Hallucinations

At Siena, we understand that sometimes she might experience what we call "hallucinations". These are instances where she might not respond as expected. But don't worry, we've got you covered! Here's a comprehensive guide on how to troubleshoot and fix these issues.

Step 1: Replicate the Issue

Before making any adjustments, it's crucial to first replicate the issue in the Playground. We recommend running the test at least five times. If you are not able to replicate the issue or you get the same correct results each time, it's likely a one-time hallucination, which is common with AI models. However, if the results vary, proceed to the troubleshooting steps. Often, simply updating the data in the automation or Persona can significantly improve the results.

Step 2: Analyze the Intent

If the detected intent is somewhat similar but not exactly the same (for example, refunds and returns), there's no need to create a new automation. Instead, you can simply add more instructions to the existing automation. This will help Siena understand how to handle these scenarios in the future, and it also keeps your library more manageable. You can even add Expressions to ensure Siena consistently selects this automation for relevant topics.

If the detected intent is not relevant to the customer's message, you might need to check for duplicate automations or unnecessary Expressions. Sometimes, a knowledge gap might be the issue. In this case, you'll need to create a new automation for that specific topic or update the title of an existing one.

Step 3: Check the Response

If the intent is correct but the response is wrong, review the instructions and context in the library. Look out for misspelled words, misplaced punctuation, or oddly structured sentences. An instruction for Siena "not to respond" sometimes causes hallucinations too since she is always expected to respond unless you add the "skip response" action. Generally, the aim is to keep the data as clear and concise as possible. The simpler and more direct your instructions, the less likely Siena is to hallucinate.

In some cases, it can be because of knowledge gap, so make sure to fill in Siena with everything that she has to know in order to respond to customers' questions.

Also, remember to organize your context, instructions, and data. This will make it easier to make adjustments later on. Check the Persona instructions and attributes as well. Some attributes, like "resolute" or "helpful", can sometimes cause Siena to be overly helpful and provide incorrect answers. Also, look out for conflicting instructions that might confuse the AI, for example: you have an instruction in your Persona to route "where is my order?" tickets to an agent, but you also have an automation to provide the tracking details to the customer and not route them to an agent. This causes Siena to hallucinate.

Step 4: Test Again

After making each change, test it again in the Playground. Run the tests five times to ensure consistent results. If you're updating multiple automations, run another set of regression tests once all changes have been made. This will ensure that each automation still behaves as expected.

We hope this guide helps you troubleshoot any AI hallucinations you might encounter. Remember, our team is always here to help if you have any further questions or need assistance. Don't hesitate to reach out to us!