Token trajectories
Data: dynamic points clouds \(P_w(l), w \in [|1, V|], l \in [|1, L|]\), where \(V\) is the size of the vocabulary, \(L\) is the total number of layers and \(w\) is a token. Each sentence containing \(w\) define a trajectory. Are there meaningful patterns (such as disambiguation) or is it the case that after layer 1 all token specific information is lost and token are simple placeholders for computations ?
The same analysis can be conducted with position in the sequence (first word, second word etc).
