Please enable JavaScript.
Coggle requires JavaScript to display documents.
Knowledge Distillation - Coggle Diagram
Knowledge Distillation
-
- what to match?
for less distillation loss
-
-
-
-
-
6) relational info
기존 IKD보다, RKD loss가 더 효과적
-> angle / distance loss 고려
-
2) on-line distillation; deep mutual learning
-> collection of small simple student network
-> supervised loss + mimicry loss (공식 구성)
-
-
- Network Augmentation
for training small network
preventing underfitting