234 lines
6.6 KiB
Markdown
234 lines
6.6 KiB
Markdown
# Token Usage Tracking - Implementation Guide
|
|
|
|
## Overview
|
|
This document explains how token usage tracking works in the application and how to test it.
|
|
|
|
## Architecture
|
|
|
|
### Token Recording Flow
|
|
1. **AI Request Made** → User sends a message (plan or build)
|
|
2. **Token Consumption** → AI provider processes request and reports usage
|
|
3. **Recording** → Server records tokens via `recordUserTokens(userId, tokens)`
|
|
4. **Persistence** → Tokens stored in `tokenUsage` object and persisted to disk
|
|
5. **UI Update** → Frontend fetches updated usage via `/api/account/usage`
|
|
6. **Display** → Progress bar updates to show new percentage
|
|
|
|
### Key Components
|
|
|
|
#### Server Side (`chat/server.js`)
|
|
|
|
**Token Storage Structure:**
|
|
```javascript
|
|
tokenUsage[userId] = {
|
|
month: '2026-01', // Current month key
|
|
usage: 15000, // Tokens used this month
|
|
addOns: 50000 // Bonus tokens purchased
|
|
}
|
|
```
|
|
|
|
**Core Functions:**
|
|
- `recordUserTokens(userId, tokens)` - Records token usage
|
|
- `getTokenUsageSummary(userId, plan)` - Returns usage summary
|
|
- `ensureTokenUsageBucket(userId)` - Ensures bucket exists for user
|
|
- `persistTokenUsage()` - Saves to disk
|
|
|
|
**API Endpoints:**
|
|
- `GET /api/account/usage` - Returns current usage summary
|
|
- `POST /api/test/simulate-tokens` - Test endpoint for simulating usage
|
|
- Body: `{ "tokens": 1000 }`
|
|
- Returns: Updated usage summary
|
|
|
|
#### Client Side (`chat/public/builder.js`)
|
|
|
|
**Core Functions:**
|
|
- `loadUsageSummary()` - Fetches usage from `/api/account/usage`
|
|
- `updateUsageProgressBar(summary)` - Updates UI with new usage data
|
|
|
|
**Update Timing:**
|
|
- After message completion (2 second delay for server persistence)
|
|
- After plan message (immediate)
|
|
- On page load
|
|
- On manual refresh
|
|
- While OpenCode is running: poll every 60 seconds
|
|
|
|
**HTML Elements:**
|
|
- `#usage-meter-title` - "Usage" label
|
|
- `#usage-meter-percent` - "X% used" text
|
|
- `#usage-meter-fill` - Progress bar fill element
|
|
- `#usage-meter-track` - Progress bar container
|
|
|
|
## Testing
|
|
|
|
### Method 1: Test Endpoint
|
|
Use the test endpoint to simulate token consumption without running actual AI models.
|
|
|
|
**Using curl:**
|
|
```bash
|
|
# Simulate 1000 tokens
|
|
curl -X POST http://localhost:4000/api/test/simulate-tokens \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tokens": 1000}'
|
|
|
|
# Simulate 5000 tokens
|
|
curl -X POST http://localhost:4000/api/test/simulate-tokens \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tokens": 5000}'
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"ok": true,
|
|
"message": "Simulated 1000 tokens",
|
|
"tokensAdded": 1000,
|
|
"summary": {
|
|
"month": "2026-01",
|
|
"used": 15000,
|
|
"limit": 50000,
|
|
"remaining": 35000,
|
|
"percent": 30,
|
|
"addOn": 0,
|
|
"plan": "hobby"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Method 2: Test HTML Page
|
|
Open `http://localhost:4000/test_token_usage.html` in your browser.
|
|
|
|
**Features:**
|
|
- Visual progress bar matching builder page
|
|
- Input field to specify token amount
|
|
- Real-time usage statistics
|
|
- Debug output showing raw API response
|
|
- Automatic refresh after simulation
|
|
|
|
**Usage:**
|
|
1. Navigate to the test page
|
|
2. Enter desired token amount (default: 1000)
|
|
3. Click "Simulate Token Usage"
|
|
4. Watch the progress bar update
|
|
5. Click "Refresh Usage" to manually fetch latest data
|
|
|
|
### Method 3: Real AI Usage
|
|
Test with actual AI requests on the builder page:
|
|
|
|
1. Open builder page: `http://localhost:4000/builder?session=<session-id>`
|
|
2. Send a plan message (uses OpenRouter)
|
|
3. Wait for response
|
|
4. Observe usage bar update after ~2 seconds
|
|
5. Send build message (uses OpenCode)
|
|
6. Wait for completion
|
|
7. Observe usage bar update again
|
|
|
|
## Verification Checklist
|
|
|
|
### Token Recording
|
|
- [x] `recordUserTokens()` accepts userId and tokens
|
|
- [x] Tokens are rounded up (ceil)
|
|
- [x] Tokens are added to bucket.usage
|
|
- [x] Changes are persisted immediately
|
|
- [x] Console logs confirm recording
|
|
|
|
### Usage Summary
|
|
- [x] Summary includes: month, used, limit, remaining, percent, addOn, plan
|
|
- [x] Percent calculation: `(used / limit) * 100`
|
|
- [x] Remaining calculation: `limit - used`
|
|
- [x] Handles zero/null limits gracefully
|
|
|
|
### API Endpoints
|
|
- [x] `/api/account/usage` returns summary
|
|
- [x] `/api/test/simulate-tokens` records and returns summary
|
|
- [x] Endpoints handle missing userId
|
|
- [x] Responses include all expected fields
|
|
|
|
### UI Updates
|
|
- [x] Progress bar width updates based on percent
|
|
- [x] Percent text shows "X% used"
|
|
- [x] Tooltip shows "used / limit tokens"
|
|
- [x] Update occurs after AI completion
|
|
- [x] Update occurs after simulation
|
|
|
|
## Token Limits by Plan
|
|
|
|
```javascript
|
|
const USER_PLANS = {
|
|
hobby: {
|
|
tokens: 50000, // 50k tokens/month
|
|
multiplier: 1
|
|
},
|
|
starter: {
|
|
tokens: 2500000, // 2.5M tokens/month
|
|
multiplier: 1
|
|
},
|
|
business: {
|
|
tokens: 5000000, // 5M tokens/month
|
|
multiplier: 1
|
|
},
|
|
enterprise: {
|
|
tokens: 10000000, // 10M tokens/month
|
|
multiplier: 1
|
|
}
|
|
}
|
|
```
|
|
|
|
## Debugging
|
|
|
|
### Server Logs
|
|
Look for these log patterns:
|
|
```
|
|
[USAGE] Recorded 1000 tokens for user-abc123. New total: 15000
|
|
[USAGE] Usage summary loaded: { month: '2026-01', used: 15000, limit: 50000, ... }
|
|
```
|
|
|
|
### Client Console
|
|
Look for these log patterns in browser console:
|
|
```
|
|
[USAGE] Usage summary loaded: { month: "2026-01", plan: "hobby", ... }
|
|
[TEST] Simulation response: { ok: true, tokensAdded: 1000, ... }
|
|
```
|
|
|
|
### Common Issues
|
|
|
|
**Issue: Usage bar doesn't update**
|
|
- Check network tab for `/api/account/usage` request
|
|
- Verify response contains `summary` object
|
|
- Check console for JavaScript errors
|
|
- Verify element IDs match: `usage-meter-fill`, `usage-meter-percent`, etc.
|
|
|
|
**Issue: Percentage is wrong**
|
|
- Verify token recording: check server logs
|
|
- Check calculation: `(used / limit) * 100`
|
|
- Ensure limit > 0
|
|
- Check for integer overflow (unlikely)
|
|
|
|
**Issue: Test endpoint returns 401**
|
|
- Ensure user is logged in
|
|
- Check cookie/session
|
|
- Verify `X-User-Id` header or `chat_user` cookie exists
|
|
|
|
## Security Notes
|
|
|
|
The `/api/test/simulate-tokens` endpoint:
|
|
- Should be disabled in production
|
|
- Currently accepts any token amount
|
|
- No authentication beyond user session
|
|
- Can be used to artificially inflate usage
|
|
|
|
**Production Considerations:**
|
|
- Remove or restrict test endpoint
|
|
- Add rate limiting
|
|
- Add admin-only flag
|
|
- Log all simulations for auditing
|
|
|
|
## Future Enhancements
|
|
|
|
1. **Real-time Updates** - WebSocket/SSE for live usage updates
|
|
2. **Granular Tracking** - Track by model/provider/feature
|
|
3. **Historical Data** - Store and display usage trends
|
|
4. **Alerts** - Notify when approaching limit
|
|
5. **Usage Analytics** - Dashboard with charts and insights
|
|
6. **Token Estimation** - Show estimated cost before sending
|
|
7. **Batch Operations** - Test multiple scenarios at once
|
|
8. **Usage Export** - Download usage data as CSV/JSON
|