OK, here it comes - another lengthy tome from me.
I master CDs for a living and this problem is one of the biggest issues in the field. We call it the "Volume Wars" and it has become a growing problem over the years, particularly since digital audio has come to the mainstream.
With vinyl records you were limited by how wide of a groove could actually be cut in the disc. Wider was louder, but also reduced the playing time. A technique known as compression has been around since the beginning of audio, and it basically reduces the loudest volume levels and also raises the quietest. This makes the audio apparently louder, hence it raised the "apparent loudness" of an audio track. This began to be used when cutting vinyl records to make the music sound louder, but still keep the groove narrow and the playing time as long as possible per side.
Once we got into digital audio and the Compact Disc, compression had advanced to be able to make audio sound as loud as anybody wanted. Thus began the volume wars - everyone wanted their records to sound louder than everyone else's. Labels got on this bandwagon and figured that they would make all of their artist's records louder than any competitor. They also (incorrectly) figured that their records would be louder on the radio.
The reason that this is incorrect is that radio stations have also been engaged in their own volume wars for years. Every station wants to be the loudest, the scheme being that as you go through the dial the loudest station will jump out and you will be hypnotized by the glorious, powerful sound - thus freezing in your tracks while sitting there listening to all of their sponsor's commercials. So the radio stations were playing leap-frog with each other by adding compression to their broadcast signal. An added benefit of compression was that it made every song sound exactly at the same level. The result of this is that the overly compressed discs that the labels were sending to the radio stations were being "double compressed" by the stations themselves. This has the effect of actually reducing the level and power of the audio.
(Reason #4,138 as to why record labels are big and stupid.)
Add to this the proliferation of inexpensive and powerful digital audio tools being used even by kids in their bedrooms and the problem becomes even more compounded. I have been asked by rock bands to master their CD so loud that it will actually distort the consumer CD players that the disc will be played in. When trying to explain this I usually get the "yeah, but it has to go to eleven, man!" attitude.
The problem is that now that everyone has pushed the volume levels up to this point, no one seems willing to let their records be quieter. It appears that there is no turning back. Oh yeah, I forgot to mention that the result of squashing all of the loud and soft levels into one constantly loud level is that the dynamic range of the music is reduced to almost nothing. In other words, no dynamics - a crucial element of music which has been utilized to great effect for thousands of years.