Further Thoughts on Double-Blind Listening Comparisons at Matched Levels [Issue No. 12]

Reluctant as we may be to give up the habit of casual audio-salon type A/B comparisons followed by instant expertizing, a more disciplined approach nearly always results in different conclusions.

We keep coming back to this subject because it is so absolutely essential to valid equipment reviewing. There is, of course, no proselytizer as fervent as a convert, and it is a matter of record that once-long ago, we would like to think-we were heathens ourselves when it came to our listening practices. Today we have little tolerance for what we have come to call the restaurant-reviewer type of audio journalist, whose dogmatic assertions about the sound of A vs. B are based either on memory or on casual dipping into both plates, as it were ("my companion had the trout, which I also tasted"), with nothing but his exquisite taste as documentation. That kind of opinion is worthless and a waste of everybody's time, except occasionally in the case of loud speakers with vastly different sonic signatures.

The conversion from conventional/traditional audiophile wisdom and comfortably vague listening criteria to the world of the unforgiving ABX comparison is a painful process for many and unbearable for some. The very idea that the sound of a $6000 and of a $600 amplifier may be indistinguishable from each other-as long as neither one is clipping, and the volume levels are matched within 0.1 or at least 0.15 dB-will be treated with stubborn and nonnegotiable denial by many otherwise reasonable aficionados.

You might as well ask a Mennonite to join the Marines; it is simply unthinkable in terms of the prevailing belief system.

The designer and manufacturer of the $6000 amplifier will, needless to say, defend their vested interest in the high-end mystique with their last breath, and the owner of such an amplifier has no choice but to defend his buying decision, but why are all, or nearly all, the high-end audio pundits so upset by ABX testing? No one so far has suggested that all $6000 amplifiers sound exactly like all $600 amplifiers; our own ABX comparisons of CD players as reported in this issue have revealed tiny differences where none were even expected; so why all the defensiveness and denial even in the ostensibly impartial segments of the audio community? It almost seems as if The High End were some kind of fraternal order or secret society requiring each member to protect and defend the credibility of all other members regardless of merit or even elementary truthfulness. We have in fact been told on several occasions by various well wishers that we would put ourselves out of business by flying in the face of this brotherhood with our bad-news ABX findings, in other words by telling the high-end audio industry and its audiophile customers what they do not want to hear. What nonsense! The facts are more exciting than fiction, or at least more gratifying in the long run, and there will always be a demand for a reliable source of facts. Even the purveyors of fiction will consult it-maybe only behind closed doors-from time to time. Pandering to fantasies and preconceived notions is a provenly successful business, as we all know, but that does not make "telling it like it is" a losing business, does it? Arguments, good and bad, against ABX-ing.

Since our initial remarks on double-blind listening tests in Issue No. 10 and our reviews in No. 11 reporting a few somewhat tentative ABX comparisons, we have been exposed to just about every argument, pro and con, about the validity of such testing and have added sundry tidbits to our previous insights. Happily, the lamest, most pathetic objection to the ABX method, namely that the switching system is not transparent and therefore covers up the differences, has not resurfaced lately, otherwise we would have to go into the boring subject of how the ABX RM-2 relay module passes square waves (surprise-perfectly), how the results with laborious plugging and unplugging by hand correspond to those obtained with the automated comparator (surprise-they are the same), and so forth. We would much rather address some of the more sophisticated criticisms to come our way.

One of the latter is that an ABX listening test is a tense and anxiety-producing "final exam" type of situation, possibly more so to certain individuals than others but always with an element of pressure and competition, especially where a peer group compares scores afterwards. It is easy to choke in such circumstances, so the argument goes, and to be less perceptive about small differences in sound than under more relaxed conditions. We are fairly sympathetic to that line of reasoning, but the rebuttal is obvious: take all the time in the world, do it alone without anyone watching you-the ABX comparator is perfectly designed for one-man self-testing-and see if the results are significantly different. In our own experience they are not.

What we find to be a greater danger, as we have previously stated, is that intense concentration on small differences produces fatigue, which in turn tends to generate resentment (why am I doing this?), which in turn leads to wild guessing to speed up the test and end it. Valid ABX testing requires sincerity and dedication because it is hard work and sometimes downright unpleasant.

Then there is the familiar song and dance to the effect that you have to live with a piece of equipment before you are able to recognize its specific sonic characteristics and tell it apart from others. Everybody has heard that one, and there is an element of plausibility there, but the rebuttal is again easy: go ahead and live with both A and B, as long as you like, provided you listen to them at exactly matched levels. There is no official time limit to an ABX test; it is entirely permissible to let it go on for months. Bob Carver tells the story of an interesting long-term listening test involving an esoteric, cult-brand tube amplifier. After he had duplicated the transfer function of this amplifier in one of his moderately priced transistor amplifiers by means of his "t-mod" technique, he physically disguised both units in such as way that their inputs and outputs were available but their identities concealed (unless of course a spoilsport made a serious effort to peek). He then left the disguised amplifiers with the skeptical owner of the tube job and asked him to keep a notebook on his listening impressions.

This was in effect a double-blind comparison without an ABX switcher (double because the one man who knew the identities of A and B was not even on the premises). Many months later Bob examined the notebook and determined that the owner had not been able to distinguish his own tube amplifier from the t-mod with higher reliability than is obtainable with sheer guessing. So much for long-term living with the equipment--when they sound the same, they sound the same, and when they sound different, the best way to hear it is still the good old quick-switching A/B method.

To state our reaction to the conventional/traditional audiophile point of view in very general terms, we are very suspicious of strong opinions about the differences between two pieces of equipment when such opinions vanish into thin air as soon as the brand identities are concealed and only the sounds can be compared. We are likewise suspicious of strong opinions about new vs. old equipment when the old equipment is no longer available as a point of reference. On the other hand, we are perfectly willing to admit that a genuine difference exists between A and B even when only one listener out of a hundred can reliably tell them apart in an ABX test. Fair enough? But then why, oh why...

There remains the nagging question of why so many honest and highly competent audio practitioners-as distinct from the phonies, mystics, creeps and crazies-also believe, on the basis of long experience, in certain equipment superiorities which then turn out to be un-provable in ABX or similar listening tests. We have only very tentative answers to that one.

Undoubtedly, preferences having to do with circuit design philosophy, construction details and even cosmetics enter into the picture, but that is a cop-out explanation.

Another is subconscious rooting for and against various brands for various emotional reasons. The $64 question, however, is whether or not these people actually hear in the sound of A something unique or distinctive that disappears in an A/B comparison at matched levels. Well, we have a theory of sorts-and it will have to remain just a theory.

A possible, though far-out explanation.

When an ABX test is being set up and the levels have not been perfectly matched yet-say there is still a 0.4 dB difference-it is ridiculously easy to tell A and B apart and to ascribe sonic personalities to each, even in cases where subsequent level matching within 0.1 dB erases all audible differences. Furthermore, that 0.4 dB mismatch is often interpreted as a quality difference rather than as a volume difference.

Now, different individuals listen to their stereo systems at different levels (even if the difference is only 1 or 2 dB), and when they change a piece of equipment they do not necessarily reestablish the exact same level within 0.1 dB. That could be a source of strong opinions about all sorts of level-related (i.e., not intrinsic or design-related) sonic characteristics. Our theory is that the same audio component A, played at two slightly different levels such as 88 dB and 89 dB, becomes in effect two different components A and B as far as subjective Gestalt is concerned. This is a psychological insight, not a scientific determination, and as such can be punched full of holes, but it does offer some sort of clue to why the same piece of equipment might be described as a little hard-sounding by the 89 dB listener and as velvety smooth by the 88 dB listener.

A/B threshold estimates.

One more thing. We are sometimes asked under what conditions we would expect two fairly similar pieces of equipment to be clearly distinguishable in a correctly set up ABX test; in other words, what the thresholds might be.

Bob Carver claims that, in his bridge nulling test, the null must be -38 dB or deeper for the two amplifiers to sound exactly the same. Dave Clark, another expert whom we trust, believes that events below the -34 dB level with respect to the signal are likely to be inaudible. Thus both of these authorities put the threshold of audible anomalies in the 1.25% to 2% range. Purists will howl; high-end audio salon owners will blanch; but remember, we are only the messengers!

-------

[adapted from TAC, Issue No. 12]

---------

Also see:

Hip Boots: Wading through the Mire of Misinformation in the Audio Press -- Straight Wire ; John Atkinson in Stereophile ; Dick Olsher in Stereophile ; Harry Pearson in The Absolute Sound

Various audio and high-fidelity magazines