Skip to main content
rendevo

AI Performance Monitoring

Track and analyze how well your AI booking assistant is performing.

Prerequisites

Enable AI Performance Learning

Location: Settings → AI & Automation

Toggle: "Enable AI Performance Learning"

What it does:

  • Enables collection of AI performance data
  • Required for all monitoring features
  • Allows AI to improve automatically
  • GDPR-compliant - opt-in only

When OFF:

  • AI still responds to customers
  • No performance data collected
  • No metrics or analytics available
  • Cannot provide feedback
  • AI cannot improve from patterns

When ON:

  • AI performance tracked
  • Metrics available in Settings
  • Can provide feedback on mistakes
  • AI learns from patterns automatically
  • Can analyze performance trends

AI Performance Tab

Location: Settings → AI & Automation → AI Performance Tab

Overview Metrics

Key performance indicators at a glance.

Total Conversations

What it is: Number of unique customer conversations handled by AI

Calculation: One conversation per customer contact, counts first message to booking or escalation

Good numbers:

  • Growing week over week
  • Matches your marketing efforts

Concerning patterns:

  • Sudden drop (integration issue?)
  • Zero conversations (AI disabled? No messages received?)

AI Resolution Rate

What it is: Percentage of conversations where AI successfully booked without escalation

Formula: (Successful bookings / Total conversations) × 100

Benchmarks:

  • 60-70%: Good for new users or complex services
  • 70-85%: Excellent for established users
  • 85%+: Outstanding performance
  • Below 60%: AI needs training or threshold adjustment

What affects it:

  • Service complexity
  • Knowledge Base quality
  • Confidence threshold setting
  • Type of questions customers ask

Average Confidence

What it is: Average AI confidence score across all interactions

Range: 0-100%

Interpretation:

  • 85-95%: AI is confident in most decisions
  • 75-85%: Normal, healthy range
  • Below 75%: AI is often uncertain, may need more training

If consistently low:

  • Add more information to Knowledge Base
  • Improve service descriptions
  • Lower confidence threshold
  • Review common customer questions

Escalation Rate

What it is: Percentage of conversations escalated to human

Formula: (Escalated conversations / Total conversations) × 100

Target range: 15-30%

Analysis:

  • Below 15%: AI might be too aggressive, check for mistakes
  • 15-30%: Healthy balance
  • 30-50%: Conservative, AI playing it safe
  • Above 50%: Threshold too high or knowledge base needs improvement

Analyzing Performance

Running Analysis

  1. Go to Settings → AI & Automation → AI Performance Tab
  2. Select date range (7 days, 30 days, 90 days)
  3. Click Analyze Performance
  4. Wait for analysis to complete (10-30 seconds)
  5. Review results

What Gets Analyzed

Intent Classification

Shows: What customers are asking for

Categories:

  • Booking requests (most common)
  • Service inquiries
  • Availability questions
  • Pricing questions
  • Policy questions
  • Complaints or issues
  • Other

How to use:

  • High "Policy questions"? Add policies to Knowledge Base
  • Many "Service inquiries"? Improve service descriptions
  • Lots of "Other"? Customers asking unique questions - review manually

Failed Knowledge Base Searches

Shows: Questions AI couldn't answer from Knowledge Base

Common patterns:

  • Customers using different terminology
  • Questions not covered in Knowledge Base
  • Ambiguous questions

Action items:

  • Add missing information to Knowledge Base
  • Include customer terminology (synonyms)
  • Clarify ambiguous policies

Example:

  • Pattern detected: "How much is a trim?" (3 times)
  • Your service name: "Haircut"
  • Action: Add to Knowledge Base: "Trim is the same as our Haircut service"

Confidence Distribution

Shows: How confident AI is across different interactions

Ideal distribution:

  • 60-70% of interactions above 85% confidence
  • 20-30% between 70-85%
  • 10% or less below 70%

Red flags:

  • Most interactions below 80%: AI needs more training
  • Everything above 95%: Either perfect setup or threshold is too low

Automatic Improvements

Synonym Detection

The system automatically detects when customers use different terms for your services.

How it works:

  1. System monitors conversations (when consent enabled)
  2. Detects frequently used terms (3+ occurrences)
  3. Finds similar official service names
  4. Auto-applies high-confidence matches (≥70%)

Example scenario:

  • Your service: "Brazilian Blowout"
  • Customers say: "keratin treatment" (4 times in one week)
  • System detects: High similarity between terms
  • Auto-applied: Now customers can say "keratin treatment" and AI understands

Runs: Daily at 3 AM (automatic background process)

You don't need to do anything - AI just gets smarter

Viewing detected synonyms: (Feature planned) Will show in AI Performance tab

Feedback System

When to Provide Feedback

Report AI mistakes to help it improve:

  1. Wrong service booked

    • Customer asked for Service A, AI booked Service B

  2. Wrong date or time

    • Customer said "tomorrow", AI booked wrong day
    • Customer said "2 PM", AI booked different time

  3. Wrong information given

    • AI gave incorrect price
    • AI stated wrong policy
    • AI provided outdated info

  4. Shouldn't have escalated

    • AI escalated a simple request
    • Question was straightforward

  5. Should have escalated

    • AI handled complex request poorly
    • Customer needed human assistance

How to Submit Feedback

  1. Go to Inbox
  2. Open the problematic conversation
  3. Click "AI Feedback" button (near top of conversation)
  4. Select issue category:
    • Wrong information
    • Wrong service
    • Wrong date/time
    • Other
  5. Describe the problem (free text)
  6. Submit

What happens next:

  • Feedback logged and analyzed
  • Patterns detected across multiple feedbacks
  • System improvements deployed automatically
  • You can see trends in Performance tab

Inbox Conversation Indicators

AI Confidence Badge

Location: On each message in Inbox

What it shows:

  • Green (85-100%): High confidence
  • Yellow (70-84%): Medium confidence
  • Red (Below 70%): Low confidence

How to use:

  • Review red messages more carefully
  • Yellow messages are usually fine but worth checking
  • Green messages generally reliable

"Requires Human" Flag

Location: Conversation list in Inbox

What it means:

  • AI couldn't handle the request
  • Confidence below threshold
  • Complex or unusual request
  • Customer specifically asked for human

Filters:

  • Use "Attention" filter to see all escalated conversations
  • Shows priority messages needing your response

"AI Handled" Badge

Location: Conversation list

What it means:

  • Conversation successfully resolved by AI
  • Booking created or question answered
  • No action needed from you

Filters:

  • Use "AI Handled" filter to review successful automations
  • Good for quality checking

Performance Optimization

Weekly Review Routine

Every Monday morning:

  1. Check AI Performance Tab metrics

    • Resolution rate trending up or down?
    • Escalation rate acceptable?
    • Any sudden changes?

  2. Run weekly analysis

    • Review intent distribution
    • Check failed KB searches
    • Note common patterns

  3. Review "Attention" conversations

    • Why did AI escalate these?
    • Were escalations appropriate?
    • Can Knowledge Base prevent similar escalations?

  4. Spot-check "AI Handled" conversations

    • Quality of responses
    • Accuracy of bookings
    • Customer satisfaction indicators

  5. Update Knowledge Base

    • Add answers to new questions
    • Clarify ambiguous information
    • Update any changed policies

Monthly Deep Dive

Once per month:

  1. Compare month-over-month metrics

    • Are you improving?
    • Any seasonal patterns?
    • Impact of changes you made?

  2. Review all feedback submitted

    • Common mistake patterns?
    • Specific areas needing improvement?

  3. Adjust confidence threshold (if needed)

    • Too many escalations? Lower threshold
    • Too many mistakes? Raise threshold

  4. Test major scenarios

    • Send test messages
    • Verify AI handles correctly
    • Check message templates still appropriate

Troubleshooting Poor Performance

Low Resolution Rate (Below 60%)

Possible causes:

  1. Confidence threshold too high
  2. Services not well-described
  3. Customers asking questions not in Knowledge Base
  4. Integration issues (messages not reaching AI)

Solutions:

  • Run weekly analysis to identify patterns
  • Lower confidence threshold by 5-10%
  • Enhance Knowledge Base with missing info
  • Check integration status

High Escalation Rate (Above 50%)

Possible causes:

  1. Confidence threshold too high
  2. Complex services requiring human judgment
  3. Knowledge Base doesn't cover common questions

Solutions:

  • Review escalated conversations for patterns
  • Lower confidence threshold
  • Add common questions to Knowledge Base
  • Consider if your services are too complex for AI

Low Average Confidence (Below 75%)

Possible causes:

  1. Service descriptions unclear
  2. Knowledge Base insufficient
  3. Customers using terminology you didn't anticipate

Solutions:

  • Improve service descriptions
  • Add more detail to Knowledge Base
  • Include customer slang and synonyms
  • Review failed KB searches for missing info

Zero or Very Few Conversations

Possible causes:

  1. AI disabled in settings
  2. Integration disconnected
  3. No customer messages received
  4. Webhook issues

Solutions:

  • Check AI enabled toggle
  • Verify integration status (green "Connected")
  • Send test message to your business number
  • Review integration troubleshooting guide

Privacy & Data Collection

What IS Collected (when consent ON):

✅ AI confidence scores ✅ Resolution rates and escalation rates ✅ Intent classifications (booking, question, etc.) ✅ Failed knowledge base search terms ✅ Feedback ratings and comments ✅ Performance metrics over time

What is NEVER Collected:

❌ Customer names, emails, or phone numbers ❌ Actual conversation content ❌ Appointment details (dates, times, services) ❌ Any personally identifiable information (PII) ❌ Business-specific pricing or policies

Optional Global Contribution

Toggle 2: "Help Improve rendevo for Everyone"

When enabled:

  • Anonymized aggregated metrics shared
  • Business type only (e.g., "beauty_wellness")
  • Helps improve AI for all users
  • You can see what was shared in audit logs

You benefit from global improvements even if you don't contribute

Related Articles

Monitor, analyze, improve - the key to great AI performance! 📊