AIPedia LogoAIPedia
Simulations

Attention Heatmap

Transformer ၏ self-attention — query token က key tokens များပေါ်တွင် weights ကို မည်ကဲ့သို့ ဖြန့်ဝေသည်ကို ကြည့်ပါ။ Row တစ်ခုကို နှိပ်ပြီး attention distribution ကို ကြည့်နိုင်သည်။ attention mechanism term နှင့် တွဲလေ့လာပါ။

Attention Matrix
7 × 7
Row = query token (ကြည့်တဲ့ token) · Column = key token (အာရုံစိုက်ခံရတဲ့ token)
"The" → "The" = 20.0%"The" → "cat" = 17.2%"The" → "sat" = 17.2%"The" → "on" = 9.9%"The" → "the" = 9.9%"The" → "mat" = 17.2%"The" → "." = 8.5%"cat" → "The" = 12.0%"cat" → "cat" = 24.1%"cat" → "sat" = 17.8%"cat" → "on" = 12.0%"cat" → "the" = 12.0%"cat" → "mat" = 12.0%"cat" → "." = 10.3%"sat" → "The" = 11.7%"sat" → "cat" = 15.7%"sat" → "sat" = 23.5%"sat" → "on" = 11.7%"sat" → "the" = 11.7%"sat" → "mat" = 15.7%"sat" → "." = 10.0%"on" → "The" = 12.7%"on" → "cat" = 12.7%"on" → "sat" = 12.7%"on" → "on" = 25.6%"on" → "the" = 12.7%"on" → "mat" = 12.7%"on" → "." = 10.9%"the" → "The" = 9.9%"the" → "cat" = 17.2%"the" → "sat" = 17.2%"the" → "on" = 9.9%"the" → "the" = 20.0%"the" → "mat" = 17.2%"the" → "." = 8.5%"mat" → "The" = 12.0%"mat" → "cat" = 12.0%"mat" → "sat" = 17.8%"mat" → "on" = 12.0%"mat" → "the" = 12.0%"mat" → "mat" = 24.1%"mat" → "." = 10.3%"." → "The" = 12.0%"." → "cat" = 12.0%"." → "sat" = 12.0%"." → "on" = 12.0%"." → "the" = 12.0%"." → "mat" = 12.0%"." → "." = 28.1%Thecatsatonthemat.Thecatsatonthemat.
Sentence
Attention head
Temperature

Temperature နိမ့်လေ attention က sharp (one token ပေါ်စုလေလဲ)၊ မြင့်လေ distribution က uniform နီးလေလဲ ဖြစ်သည်။

Attention of “cat”
The
12.0%
cat
24.1%
sat
17.8%
on
12.0%
the
12.0%
mat
12.0%
.
10.3%