Please enable JavaScript.

Coggle requires JavaScript to display documents.

Cross-Iteration Batch Normalization(2020) - Coggle Diagram

- - - - $$\mu_{t-\tau}= \mu_{t-\tau}(\Theta_{t-\tau})+ \frac{\partial \mu_{t-\tau} (\Theta_{t-\tau})} {\partial \Theta_{t-\tau}} (\Theta_t - \Theta_{t-\tau})+ O(||\Theta_t-\Theta_{t-\tau}||^2)$$
      - $$v_{t-\tau}= v_{t-\tau}(\Theta_{t-\tau})+ \frac{\partial v_{t-\tau} (\Theta_{t-\tau})} {\partial \Theta_{t-\tau}} (\Theta_t - \Theta_{t-\tau})+ O(||\Theta_t-\Theta_{t-\tau}||^2)$$
  - - - $$v^l_{t-\tau}(\Theta_t) \approx v^l_{t-\tau}(\Theta_{t-\tau})+ \frac{\partial v^l_{t-\tau} (\Theta_{t-\tau})} {\partial \Theta^l_{t-\tau}} (\Theta^l_t - \Theta^l_{t-\tau})$$
      - $$\mu^l_{t-\tau}(\Theta_t) \approx \mu^l_{t-\tau}(\Theta_{t-\tau})+ \frac{\partial \mu^l_{t-\tau} (\Theta_{t-\tau})} {\partial \Theta^l_{t-\tau}} (\Theta^l_t - \Theta^l_{t-\tau})$$
  - - - $$\overline{\mu}^l_{t,k}(\Theta_t)= \frac{1}{k} \sum_{\tau=0}^{k-1} \mu^l_{t-\tau}(\Theta_t)$$
      - $$\overline{v}^l_{t,k}(\Theta_t)= \frac{1}{k} \sum_{\tau=0}^{k-1} \max[v^l_{t-\tau}(\Theta_t), \mu^l_{t-\tau}(\Theta_t)]$$
      - $$\overline{\sigma}^l_{t,k}(\Theta_t)= \sqrt{\overline{v}^l_{t,k}(\Theta_t)- \overline{\mu}^l_{t,k}(\Theta_t)^2}$$