Restore to commit 74e578279624c6045ca440a3459ebfa1f8d54191
This commit is contained in:
southseact-3d
180
CONTAINER_LOGGING.md
Normal file
180
CONTAINER_LOGGING.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Container Diagnostics and Logging
|
||||
|
||||
## Overview
|
||||
This application includes comprehensive container diagnostics and logging to help troubleshoot issues in production.
|
||||
|
||||
## Log Files
|
||||
|
||||
### Diagnostic Logs
|
||||
Located in: `/var/log/shopify-ai/`
|
||||
|
||||
Files:
|
||||
- `diagnostics.log` - System startup, configuration, and runtime diagnostics
|
||||
- `healthcheck.log` - Health check results and service status
|
||||
|
||||
### Accessing Logs
|
||||
|
||||
#### Via Docker
|
||||
```bash
|
||||
# View diagnostic logs
|
||||
docker logs -f shopify-ai-builder
|
||||
|
||||
# Follow log volume
|
||||
docker exec -it shopify-ai-builder tail -f /var/log/shopify-ai/diagnostics.log
|
||||
|
||||
# View health check results
|
||||
docker exec -it shopify-ai-builder cat /var/log/shopify-ai/healthcheck.log
|
||||
```
|
||||
|
||||
#### Via Named Volume
|
||||
```bash
|
||||
# Access logs volume
|
||||
docker run --rm -v shopify_ai_logs:/data alpine cat /data/diagnostics.log
|
||||
|
||||
# Copy logs to local machine
|
||||
docker run --rm -v shopify_ai_logs:/logs -v $(pwd):/output alpine cp -a /logs /output/
|
||||
```
|
||||
|
||||
## Log Levels
|
||||
|
||||
Logs use the following severity levels:
|
||||
- **INFO** - Normal operation and informational messages
|
||||
- **WARN** - Warning messages (non-critical issues)
|
||||
- **ERROR** - Error messages (critical failures)
|
||||
- **DEBUG** - Detailed debugging information
|
||||
|
||||
## What's Logged
|
||||
|
||||
### Startup Diagnostics
|
||||
- Container ID and hostname
|
||||
- OS and kernel information
|
||||
- CPU count and model
|
||||
- Total/used/available memory
|
||||
- Disk usage
|
||||
- Network interfaces and IP addresses
|
||||
- Environment variable validation
|
||||
- Filesystem permissions
|
||||
|
||||
### Runtime Monitoring
|
||||
Every 2 minutes:
|
||||
- CPU usage percentage
|
||||
- Memory usage (used/total/percentage)
|
||||
- Disk usage percentage
|
||||
- System load average
|
||||
|
||||
Every 5 minutes:
|
||||
- Chat service status (port 4000)
|
||||
- TTYD service status (port 4001)
|
||||
- Process health checks
|
||||
- HTTP endpoint responsiveness
|
||||
|
||||
### Service Health Checks
|
||||
- Port listening status
|
||||
- HTTP endpoint response
|
||||
- Process memory and CPU usage
|
||||
- Process uptime
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### High Memory Usage
|
||||
If logs show `⚠ High memory usage`:
|
||||
1. Check current usage: `docker stats shopify-ai-builder`
|
||||
2. Review recent message sizes
|
||||
3. Check for memory leaks in server.js
|
||||
4. Consider increasing container memory limits
|
||||
|
||||
### Service Not Responding
|
||||
If health check fails:
|
||||
1. View diagnostic logs for errors
|
||||
2. Check if ports are accessible: `netstat -tuln | grep -E ':(4000|4001)'`
|
||||
3. Check process status: `ps aux | grep -E 'node|ttyd'`
|
||||
4. Restart container: `docker restart shopify-ai-builder`
|
||||
|
||||
### Disk Space Issues
|
||||
If logs show `⚠ High disk usage`:
|
||||
1. Check workspace size: `du -sh /home/web/data`
|
||||
2. Clean up old sessions
|
||||
3. Check log file sizes: `du -sh /var/log/shopify-ai`
|
||||
4. Rotate logs manually if needed
|
||||
|
||||
## Log Rotation
|
||||
|
||||
Diagnostic logs are automatically rotated when they exceed 10 MB:
|
||||
- Original file is backed up with timestamp: `diagnostics.log.20250114_120000.bak`
|
||||
- New log file is started
|
||||
- Old logs are retained until manually cleaned
|
||||
|
||||
## Enhancing Logging
|
||||
|
||||
To add custom logging:
|
||||
|
||||
In `entrypoint.sh`:
|
||||
```bash
|
||||
log "Your custom message"
|
||||
|
||||
# With diagnostic logger
|
||||
if type diag_log &>/dev/null; then
|
||||
diag_log "INFO" "Your diagnostic message"
|
||||
fi
|
||||
```
|
||||
|
||||
In `healthcheck.sh`:
|
||||
```bash
|
||||
health_log "INFO" "Your health check message"
|
||||
```
|
||||
|
||||
## Monitoring Dashboard
|
||||
|
||||
For a real-time monitoring dashboard:
|
||||
|
||||
```bash
|
||||
# Follow diagnostic logs with color highlighting
|
||||
docker exec -it shopify-ai-builder tail -f /var/log/shopify-ai/diagnostics.log | grep --color=auto -E 'ERROR|WARN|INFO'
|
||||
|
||||
# Monitor all logs
|
||||
docker logs -f shopify-ai-builder 2>&1 | grep --color=auto -E 'ERROR|WARN|usage|tokens'
|
||||
```
|
||||
|
||||
## Common Issues and Log Patterns
|
||||
|
||||
### Token Usage Tracking
|
||||
Look for: `[USAGE]` tags
|
||||
```
|
||||
[2025-01-14 10:30:00] [INFO] [USAGE] Usage summary loaded: {...}
|
||||
[2025-01-14 10:30:05] [INFO] [USAGE] Started aggressive usage polling
|
||||
[2025-01-14 10:30:10] [INFO] [USAGE] Stopped usage polling
|
||||
```
|
||||
|
||||
### Service Startup
|
||||
Look for service startup messages:
|
||||
```
|
||||
[2025-01-14 10:00:00] [INFO] Chat service started with PID: 123
|
||||
[2025-01-14 10:00:02] [INFO] ✓ chat: Port 4000 listening
|
||||
[2025-01-14 10:00:02] [INFO] ✓ chat: HTTP endpoint responding
|
||||
```
|
||||
|
||||
### Resource Issues
|
||||
Look for warning markers:
|
||||
```
|
||||
[2025-01-14 10:00:00] [WARN] ⚠ High memory usage: 95%
|
||||
[2025-01-14 10:00:00] [WARN] ⚠ High disk usage: 85%
|
||||
```
|
||||
|
||||
## Export Logs for Support
|
||||
|
||||
To export all logs for debugging:
|
||||
|
||||
```bash
|
||||
# Create a logs export directory
|
||||
mkdir -p ./logs-export
|
||||
docker cp shopify-ai-builder:/var/log/shopify-ai/. ./logs-export/
|
||||
|
||||
# Export Docker container logs
|
||||
docker logs shopify-ai-builder > ./logs-export/container-logs.txt 2>&1
|
||||
|
||||
# Create a compressed archive
|
||||
tar -czf logs-export-$(date +%Y%m%d).tar.gz logs-export/
|
||||
|
||||
# Send to support
|
||||
# Attach logs-export-20250114.tar.gz to your support ticket
|
||||
```
|
||||
Reference in New Issue
Block a user