shopify-ai-backup/TESTING_MISTRAL_LOGGING.md

Testing Mistral Response Parsing with Enhanced Logging

How to Test

1. Set Up Mistral as Planning Provider

1. Log in to the admin panel at /admin
2. Go to the "Provider Configuration" section
3. Set "Planning Provider" to "Mistral"
4. Make sure you have a valid MISTRAL_API_KEY environment variable set
5. Configure at least one Mistral model in the models list

2. Create a Test Plan

1. Go to /apps (or /builder)
2. Create a new app/session
3. Send a message that should trigger the planning phase
4. Watch the logs

3. View the Logs

Docker Logs:

# View all logs
docker logs <container-id> -f

# Filter for Mistral logs only
docker logs <container-id> -f 2>&1 | grep "\[MISTRAL\]"

# Filter for plan logs
docker logs <container-id> -f 2>&1 | grep "\[PLAN\]"

# Filter for both
docker logs <container-id> -f 2>&1 | grep -E "\[MISTRAL\]|\[PLAN\]"

ttyd Terminal (if available):

# In the container terminal
tail -f /path/to/server.log | grep -E "\[MISTRAL\]|\[PLAN\]"

What to Look For

1. Request Being Made

Look for:

[MISTRAL] Starting API request { url: '...', model: '...', ... }
[MISTRAL] Request payload: { model: '...', messagesCount: 3, ... }

Check:

• Is the URL correct? (https://api.mistral.ai/v1/chat/completions)
• Is there an API key? (should show first 8 characters)
• Is the model name correct?
• Are messages being sent?

2. Response Received

Look for:

[MISTRAL] Response received { status: 200, ok: true, ... }
[MISTRAL] Full API response: { "choices": [...], ... }

Check:

• Is status 200?
• Is ok: true?
• What does the full response look like?

3. Content Extraction

Look for:

[MISTRAL] Choices array: [...]
[MISTRAL] First choice: { ... }
[MISTRAL] Message object: { ... }
[MISTRAL] Content value: "actual content here"
[MISTRAL] Content type: "string"

Check:

• Does choices array exist?
• Does first choice exist?
• Does message object exist?
• Does content value exist and is it a string?

4. Reply Extraction

Look for:

[MISTRAL] Extracted reply: {
  reply: "...",
  replyType: "string",
  replyLength: 1234,
  isEmpty: false,
  isNull: false,
  isUndefined: false,
  isFalsy: false
}

Check:

• Is reply a string?
• Is replyLength > 0?
• Are isEmpty/isNull/isUndefined all false?

5. Plan Processing

Look for:

[PLAN] Mistral result received (raw): { hasResult: true, ... }
[PLAN] Before sanitization: { rawReply: "...", ... }
[PLAN] After sanitization: { sanitizedReply: "...", ... }

Check:

• Is hasResult: true?
• Does rawReply contain content?
• Does sanitizedReply match rawReply? (or is content being removed?)

6. Message Creation

Look for:

[PLAN] Creating message object with reply: { reply: "...", ... }
[PLAN] Final message details: { messageId: "...", messageReply: "...", ... }
[PLAN] Plan message completed successfully { ... }

Check:

• Does the message object have a reply?
• Is messageReplyLength > 0?
• Does it complete successfully?

Common Issues and What Logs Will Show

Issue: "No API key"

[MISTRAL] API key missing

Fix: Set MISTRAL_API_KEY environment variable

Issue: "API returns error"

[MISTRAL] Response received { status: 401, ok: false, ... }
[MISTRAL] Error response body: "Unauthorized"

Fix: Check API key validity

Issue: "Empty response"

[MISTRAL] Full API response: { "choices": [] }
or
[MISTRAL] Content value: undefined

Fix: Check API documentation, may need different model or parameters

Issue: "Content removed by sanitization"

[PLAN] Before sanitization: { rawReply: "...", rawReplyLength: 1234 }
[PLAN] After sanitization: { sanitizedReply: "", sanitizedReplyLength: 0, wasModified: true }

Fix: Check sanitizeAiOutput() function, may be too aggressive

Issue: "Reply lost in message creation"

[PLAN] After sanitization: { sanitizedReply: "...", sanitizedReplyLength: 1234 }
[PLAN] Creating message object with reply: { reply: "", replyLength: 0 }

Fix: Check the assignment between sanitization and message creation

Expected Successful Output

For a successful Mistral plan request, you should see:

[PLAN] Starting plan message handling { sessionId: '...', ... }
[PLAN] Trying provider { provider: 'mistral', model: '...' }
[PLAN] Using Mistral provider { modelHint: '...', messagesCount: 3 }
[MISTRAL] Starting fallback chain { preferredModel: '...', chainLength: 1, models: [...] }
[MISTRAL] Trying model: mistral-...
[MISTRAL] Starting API request { url: '...', model: '...', ... }
[MISTRAL] Request payload: { model: '...', messagesCount: 3, ... }
[MISTRAL] Response received { status: 200, ok: true, ... }
[MISTRAL] Full API response: { "choices": [{ "message": { "content": "..." } }], ... }
[MISTRAL] Response data structure: { hasChoices: true, choicesLength: 1, ... }
[MISTRAL] Choices array: [{ ... }]
[MISTRAL] First choice: { message: { content: '...' }, ... }
[MISTRAL] Message object: { content: '...' }
[MISTRAL] Content value: Plan details here...
[MISTRAL] Content type: string
[MISTRAL] Extracted reply: { reply: '...', replyType: 'string', replyLength: 1234, isEmpty: false, ... }
[MISTRAL] Successfully extracted reply { replyLength: 1234, replyPreview: 'Plan details...' }
[MISTRAL] Plan succeeded on first try { model: '...' }
[PLAN] Mistral result received (raw): { hasResult: true, hasReply: true, replyLength: 1234, ... }
[PLAN] Before sanitization: { rawReply: '...', rawReplyLength: 1234 }
[PLAN] After sanitization: { sanitizedReply: '...', sanitizedReplyLength: 1234, wasModified: false }
[PLAN] Provider succeeded { provider: 'mistral', model: '...', replyLength: 1234, ... }
[PLAN] Creating message object with reply: { reply: '...', replyLength: 1234, ... }
[PLAN] Final message details: { messageId: '...', messageReplyLength: 1234, cleanReplyLength: 1234, ... }
[PLAN] Plan message completed successfully { sessionId: '...', provider: 'mistral', ... }

Reporting Issues

When reporting issues, please include:

1. The full log output filtered by [MISTRAL] and [PLAN]
2. Screenshots of the UI showing the empty response
3. The admin panel configuration (provider settings, model settings)
4. Environment variables (without exposing the actual API key)

This will help identify exactly where in the chain the response is being lost.