Usability and Format Design—Rubens and Rubens

The overall purpose of the two true experiment usability tests conducted by Rubens and Rubens (R&R) was to identify the differences in format and design that can affect a technical document’s ease of use in terms of retrieval time and task completion (213). 

Study 1: Manual Design and Performance

The purpose of Study 1 was to test three versions of the same manual to determine if one was easier to use. If this could be identified, then R&R would proceed with designing additional tests assessing features of the most usable manual (219). R&R pretested their research instruments—the original and modified manuals and the task-based questions—prior to their studies; however, very little information was given about the pretest itself (213). Two people participated, but neither person was described. Despite the lack of details about the pretest, R&R claimed it revealed strengths and weaknesses about the manuals.

Neither their hypothesis nor their research question(s) were stated explicitly; however, it can be inferred based on their literature review and research design that they assumed changing the format and design of the original manual will make it a more usable document based on time-to-productivity and task support (215-19). This is a cause-and-effect relationship.

The original and modified manual formats were described in detail (214-17). The type of manual and their content were not identified, but the withholding of this information can be justified since the object of study is formatting not content. Subjects were selected from unspecified classes at an unnamed college (220). 87 subjects participated in Study 1, meeting the 10:1 subject-to-variable ratio. There were four variables in total. Two dependent variables included: search and retrieval time (interval) and performance scores (interval). Two independent variables included: manual type (nominal) and question type (nominal). An equal number of manuals was randomly distributed to the subjects for data collection. Those who received Manual B, the original, were part of the control group; those with Manuals A and C were the treatment groups.

Data was collected using a 10-question task-based test. Questions varied in difficulty from simple to complex as defined by R&R. As each subject took the test with his or her assigned manual, the time it took to complete a task was measured and a score was given for each correct answer. R&R did a correlational analysis of their interval and nominal data by comparing manual types to performance scores, manual types to question types, and manual types to time (221-23). Unfortunately, R&R did not provide variance or standard deviation. Only means were provided for all variables. Statistical significance was only calculated for question type to time and performance score, but the questions were not the focus of the study.

R&R’s generalization from their analysis: formatting techniques do not always create usable manuals, but they can influence performance.

Study 2: Reference Strategies and Performance

The purpose of Study 2 was to determine how reference aids affect retrieval time and which type contributes to ease of use. R&R hypothesized the formatting of the reference aid and manual will be responsible for quicker retrieval times, however this is not explicitly stated again (224). 

The original manual and Manual A from Study 1 were focused on because their performance scores were higher. The same task-based questions were reused. Three levels of reference aids with two variations were the major addition to the second study, however, it was not indicated whether these instruments were pretested (226). Subjects were selected from unspecified classes at an unnamed college again, but the selection process was not specified (227). This time, 104 subjects participated whose demographic makeup was slightly different from Study 1’s but still met the 10:1 subject-to-variable ratio. The same dependent and independent variables were used from Study 1 with the addition of one more independent variable: reference strategy (nominal). Manuals and reference aids were distributed randomly.

Again, data was collected using the same 10-question task-based test to measure time and score (227). Again, data was correlated between the dependent and independent variables, and again, only the means and select statistical significance for question type to time and performance score were provided.

R&R’s generalization from their analysis: a variety of simple and task-oriented retrieval aids is most effective for manual design to improve performance.

Problems with the Research Design

1. It is unclear whether subject selection was randomized. Randomization is integral to a true experiment’s internal validity.
   
2. Hypotheses are implied leaving R&R’s expectations of the results of treatments up to interpretation.
   
3. Report of the data analysis seems selective. Variance and standard deviation were absent. Significance seemed irrelevant.
   
4. It is unclear whether Study 2 had a control group. 

Rubens, Phillip and Brenda Knowles Rubens. “Usability Testing and Format Design.” Effective Documentation: What We Have Learned From Research. Ed. Stephen Doheny-Farina. Cambridge: MIT Press. 1988. 213-233.