Listening Crews
Home Up



The reliability and validity of results yielded by Diagnostic Rhyme Test (DRT), Diagnostic Acceptability Measure (DAM), Paired Acceptability Rating Method (PARM), and other speech evaluation measures under development at Dynastat are large ensured by the intrinsic characteristics of the measures themselves. But, the level of precision ultimately achieved with these measures also depends on the manner in which they are used, most particularly on the manner in which listener factors are controlled.

Inter- and intra-individual variation in listener performance are potential sources of error in all speech evaluation measurements. Effective control of these factors is thus essential to precise measurement of speech intelligibility, overall acceptability, and speaker recognizability. Being fully cognizant of this principle, Dynastat employs extremely rigorous procedures for the selection, training, monitoring, and maintenance of its listening crews.


The Listener Selection Process

In view of Dynastat's substantial investment in each listening crew member, listener candidates are subjected to very careful scrutiny. A comprehensive biographical survey and personal interview are used to identify individuals whose personal characteristics and circumstances indicate them to be adaptable to the demands of the testing situation and likely to remain with Dynastat for an extended period. Individuals are further screened by means of a pure-tone audiometric screening test and various speech perception tests. Measures of the candidate's self consistency and conformity with established norms provide the criteria for crew membership on a trial basis. Approximately 30% of all bona fide applicants survive this level of screening.


The Listener Training Process

Following admission to trial membership, listener candidates participate in both routine and special testing sessions for an extended period during which time their performance on the DRT, the DAM, the DCT, and other measures are closely monitored. Either erratic performance or erratic attendance during this period may provide grounds for dismissal.


The Listener Monitoring Process

After admission to permanent status, new crew members participate as alternates in all research and service testing operations, but their data are normally analyzed only as part of the continuing monitoring process. Once stabilized and calibrated, they participate as useful crew members in cases of absence of tardiness by senior crew members. As vacancies occur they ascend to the status of senior crew membership.


Crew Maintenance

It has been Dynastat's experience that the listening crew member maintains the highest level and greatest stability of performance when he/she is a test participant on a regular basis with a carefully regulated work load. At Dynastat, all crew members normally participate in 4-5-hour testing sessions at least three times a week. Within the testing session there are regular rest breaks, such that the listener's "duty cycle" averages approximately 50%.

Listeners are not usually provided feedback regarding the accuracy of their responses to test materials. Although the DRT is uniquely immune to such effects, rewards for the correct responses can, with some speech discrimination tests, induce the listener to respond on the basis of factors other than his unbiased perception to the speech test materials (e.g., familiarity with the corpus of test materials or with specific randomizations of them). Where listener responses are influenced by such extra-signal or contextual factors, the test results necessarily provide a less valid reflection of the intrinsic characteristics of the system or device being evaluated.


Crew Factors and Experimental Design

Dynastat normally maintains two or more crews for purposes of routine testing and research. There are several advantages associated with this policy. In particular, the effects of absences by one or more crew members are minimized, since individuals of highly similar response characteristics re available as replacements. The availability of more than one crew also permits greater flexibility with regard to experimental design.