In simple terms, it is understood that a statistical system at equilibrium tends towards its most probable state.
Then degeneracy is used in reference to any one of a class of distinct but equivalent physical states of elements of a system: equivalent, say, with respect to their energy or some other intrinsic value.
A simple example would be a die with six sides, five of which have a single dot and the sixth has six dots. The one-dot state would be said to be five-fold degenerate since there are five distinct sides which are otherwise equivalent.
Even though the die is seemingly composed of only two different states, it obviously has six distinct sides or states, and each of those has equal probability of coming up when the die is rolled. Since we can't tell the five one-dot sides apart, it appears as though the one comes up five times more often than the six.
Degeneracy in states can be understood as a loss of feature such that the multiple states of an item become indistinguishable from each other. In the die example all but one face of the die has lost some of its distinguishing features, that is, an individual number of dots that is used to differentiate the faces. What remains is the one dot feature that makes five of six faces indistinguishable.