A rubber is a viscoelastic Polymer (also called elastomer). What makes it viscoelastic is most often that the polymer is cross-linked (though not too cross-linked, as that can lead to rigid materials).
Traditionally, cross-linking was done by exposing natural latex to sulfur, a process known as vulcanization.
Although rubbers are viscoelastic, there is really a continuum between solid and viscoelastic, and some are closer to solids, with others are more clearly viscoelastic.
Silly putty is interesting (apart from fun), because it has viscoelastic properties, but the polymers it's made of are not cross-linked, they are just very long!
Viscoelastic Behavior of Rubbery Materials
Glass transition temperature
There is a temperature, called the glass transition temperature, below which a cross-linked polymer stops being viscoelastic (and thus a rubber), and becomes glassy, and hard.
Above the glass transition temperature, the polymer chains are loose and floppy, and that's why a rubber classifies as a soft material.
Rubbers are also thermoplastic.