Key Takeaways

  • Track token usage ruthlessly: Don't just prompt and hope for the best. Alessio Finelli, a builder working with OpenAI's Symphony, constantly tracks token usage to pinpoint where an AI agent's tools or prompts fall short. This approach lets you identify efficiency bottlenecks before they balloon costs or complexity.
  • Actively "purge" outdated instructions: AI models, Finelli notes, have a natural "tendency to like add rather than remove" information. If you're not actively managing your agent's workflow.md files, they'll bloat with old, sometimes contradictory advice, making your agents confused and less effective. Regularly prune unnecessary lines.
  • Improve the tooling layer, not just the prompt: When your AI agent consistently misses the mark, the problem might not be your prompt. Finelli suggests that a significant gap between expected and actual outcomes often signals a need to refine the underlying tools or integrations the agent uses. As he says, “If the reality is very far away from your expectations there's probably something in the tooling layer that you can do to improve.”
  • Small sentences, heavy weight: Forget the sprawling "magic skills file" or the lengthy business problem-solving markdown. Finelli emphasizes that clear, concise instructions carry more weight than verbose explanations, preventing model confusion. "I think everybody just wants to have the magic skills file that does everything for them," he reflects. “I think the reality is like now more than ever like small sentences of like very a lot of weight.”
  • Follow Alessio Finelli's AI Prompting Troubleshooting Checklist: When an autonomous agent goes "off the rails," Finelli has a direct, five-step system for getting it back on track, from switching providers to decomposing complex problems.

The Alessio Finelli's AI Prompting Troubleshooting Checklist

Type: checklist

Name: Alessio Finelli's AI Prompting Troubleshooting Checklist

Components:

  • 1. Switch Providers: If available, try using a different AI model or provider.
  • 2. Restart Conversation: Begin a new conversation or reset the agent's context.
  • 3. Tweak Prompt: Refine the existing prompt, adding specific instructions or constraints.
  • 4. Decompose Problem: Break down the complex task into smaller, more manageable sub-problems to reduce ambition.
  • 5. Use Speech-to-Text (Yapper's API): For longer or more complex instructions, use speech-to-text to articulate thoughts more fully, often leading to additional helpful details.

When This Works (and When It Doesn't)

This checklist is particularly useful when an AI agent deviates from expectations or fails to execute a task correctly, requiring an iterative refinement of your interaction and a simplification of the problem space for the agent. This framework truly shines when you're debugging an autonomous agent designed for sequential tasks or complex workflows, especially where its internal state and instruction set evolve over time, much like the systems Finelli builds with OpenAI's Symphony. It's less directly applicable to basic, single-shot prompt engineering for simple text generation, though core principles like "tweak prompt" remain valuable. Its real power emerges when an agent maintains "memory" or a persistent instruction set, making troubleshooting stateful behavior critical.

What to Do With This

Imagine you've built an AI agent with Linear to draft customer support responses, but it's suddenly adding excessive apologies and asking redundant questions, frustrating your users. Here's how to apply Finelli's checklist this week:

1. Switch Providers: If you're on OpenAI's GPT-4, test your prompts and tools against Claude 3 Opus or even Gemini 1.5 Pro, if available. Sometimes, a different model's inherent biases or strengths can resolve the issue immediately.

2. Restart Conversation: Clear the agent's memory entirely. Force a full context reset. This ensures no lingering, misfiring instructions from previous interactions are influencing its current, bad behavior.

3. Tweak Prompt: Refine the main prompt. If it's over-apologizing, add explicit instructions: "DO NOT apologize unless the customer explicitly states dissatisfaction." If it's asking redundant questions, add: "Confirm prior information first; do not ask for details already provided in the ticket history." Remember Finelli's point: small sentences, heavy weight.

4. Decompose Problem: Is your agent trying to do too much? Instead of handling all support requests, can it focus only on, say, password resets and billing inquiries, passing more complex issues to a human or a specialized agent? Reduce its scope to reduce its chances of error.

5. Use Speech-to-Text: If the problem persists, articulate your ideal customer support interaction aloud. Describe the tone, the information flow, and the exact steps. Use a tool like Yapper's API to transcribe your thoughts, often uncovering nuances you might have missed typing out, leading to more precise constraints for your agent.