Why do we only have three types of receptors?

From our earlier discussion, it is obvious that the more types of receptors we have (the greater the number $n$), the higher the resolution of our internal representation of the energy spectral distribution of the light signal, and the more colors that can be distinguished. Then the question is, why didn't our visual system evolve to have more than $n=3$ different types of receptors for uniquely distinguishing more colors?

In our auditory system, we do have many different types of detectors each tuned to respond most optimally a narrow range of frequencies. And similarly in our smell and taste sensory systems there are many different types of detectors each for detecting a particular smell or taste. But why don't we have many more types of retina cone cells ($n \gg 3$) each for one particular band of wavelength?

The answer to this question is obvious if we consider the cost. We should realize that vision takes place in a two-dimensional visual field (the surface area of the retina), compared to other senses such as hearing and smell which essentially take place at a point (of zero dimension). In order to be able to sense the color of every position in the 2D field by a set of $n \gg 3$ detector, there would have to be many such receptors, too many to be possibly contained in the retina of limited area. Moreover, the brain would also have to processing much more visual information coming from the receptors in the eye.

But due to the comparison among the three types of receptors we do have, we perceive many more different colors than just the three colors of red, green and blue. All these different colors are represented internally in our brain as different ratios between the three types of sensors or neurons tuned for the three primary colors. In other words, there are not ``grandmother cells'' each for a different color. Instead, a specific color is represented internally by a set of basic features, the three primary colors. During the long process of evolution, Nature made an optimal tradeoff between the performance (number of colors we perceive) and the cost (number of receptors needed for the task) by having three types of receptors to see many more colors than most of the animals.