Attention in machine learning

An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sumof the values, where the weight assigned to each value is computed by a compatibility function of thequery with the corresponding key.

The frontal and parietal cortex: Eye movements and attention

Predictive coding is related to attention

The normalization model of attention. Model proposes attention is mostly accomplished by multiplying input by an attention field. Furthermore, the propose a model of attention that incorporates divisive normalization (code on paper)

Some results are consistent with the appealingly simple proposal that attention increases neuronal responses multiplicatively by applying a fixed response gain factor (McAdams and Maunsell, 1999; Treue and Martinez-Trujillo, 1999), while others are more in keeping with a change in contrast gain (Li and Basso, 2008, Martinez-Trujillo and Treue, 2002; Reynolds et at., 2000) or with effects that are intermediate between response gain and contrast gain changes (Williford and Maunsell, 2006)

We propose that this computational principle endows the brain with the capacity to increase sensitivity to faint stimuli presented alone and to reduce the impact of task irrelevant distracters when multiple stimuli are presented.

The three basic components of the model are: the stimulation field, the suppressive field, and the attention field

https://www.youtube.com/watch?v=nA5LVjAqkt8