Please enable JavaScript.
Coggle requires JavaScript to display documents.
Threshold - Coggle Diagram
Threshold
Llama-2 70B
Time-to-first token
Threshold: 50-120ms
Severity Score
Normal: <=150ms
Warning: 150 - 300ms
Critical: 300ms
Time Per Output token
Severity Score
Normal
300 tokens/sec
Warning
150 - 300 tokens/sec
Critical
< 150 tokens/sec
End to End Latency
Normal
<=2 sec
Warning
2 - 4 sec
Critical
4sec
Throughput
Normal
15 RPS
Warning
7 - 15 RPS
Critical
<= 7 RPS
CPU Utilization
Normal
<=60%
Warning
60 - 80 %
Critical
80%
Memory Utilization
Normal
70 %
Warning
70 - 85 %
Critical
85%