This study investigated performance differences between three different web-based navigation models: linear, persistent, and semi-persistent menu structures. Forty-four college students were placed into one of the three navigation conditions and completed information-finding tasks. No significant differences were found among the models with respect to success in completing tasks or overall completion time. Use of other navigation aids built into the browser such as the HOME and BACK buttons was also measured, as well user satisfaction and perceived usability. No differences among models were found here either. Results of this study tentatively indicate that the three navigation models, as tested, performed equally well and further that designers should perhaps give fair consideration to each of the models pending the purpose and audience of the site being designed. Further research is needed, since several potentially confounding factors were identified.
Introduction
Everyday thousands of people around the world make it a part of their day to locate some piece of information on the World Wide Web (WWW). Sometimes users of the WWW are browsing, simply looking at various web sites to find something that interests them, while other users access the WWW with a specific task in mind.
Web sites vary a great deal in terms of the audience being served and the purpose of the content being provided. There are sites that target children, adolescents and adults; educators, business executives, and researchers; vacationers, television junkies, and sports fanatics. The list goes on as long as there are topics and people who are interested in them. Of all these sites that are currently in use, some are intended to be used occasionally or even rarely, while there are other sites, such as corporate Intranet sites, that are used daily for very specific, job-critical tasks.
Regardless of the usage context for a particular web site, all web sites must employ some kind of navigation scheme to aid the user in traveling from one page to another. Designers of web sites currently enjoy little in the way of published, empirically supported guidelines specific to navigation issues. Site navigation is an essential element of an effective web site and there is a pressing need for researchers to begin investigating the issues and questions surrounding web-based navigation models. Most sites today are still built based either on personal experience and taste, an incompatible medium (i.e., paper), or at best on usability data with limited generalizability.
The purpose of this study was to investigate the effectiveness of three popular navigational models used in on the WWW. Specifically this study had hoped to begin laying the foundation for future studies dealing with web sites and how people perform using them.
Review of Related Literature
Though the research base on navigating within a WWW environment is limited, there are a number of studies which have been done in the last twenty years that have important implications for designing navigation schemes for web-based information systems. The varying research areas of influence include studies done on paper-based navigation, pre-web hypertext models, and web site usability. Hierarchical and persistent menu strategies on the web are a relatively new area of interface design interest. Only recently have researchers begun to address the question of menu development as applied to a hypertext environment, and in particular the World Wide Web. Empirical research is just beginning to emerge, mostly in the form of dissertations and research-oriented courses and seminars.
Effectiveness and Challenges of Hierarchical Information Structures
Research done by Chang (1995) indicates that hierarchical structures result in the most efficient navigation and information retrieval. In designing specific hierarchical menu structures, depth (the number of levels) and breadth (the number of items per page) were shown to be prime considerations by Harney (1990). Other factors that have been shown to be of importance in hierarchical menu design include:
One of the few recommendations one can locate in the research comes from Sano (1997) where he states that an effective and intuitive online information structure has two critical characteristics. First, the information which users need the most should be within two to three levels away from their starting point. Second, grouping of related items is essential to avoiding long lists of information that the user must scroll through.
Structural Cues
Paper documents have long used a wide array of navigation aids to help users find the information they need. Tables of contents have been shown to be effective navigation aids in paper documents. Indices, vertical tabs, and other devices have also been used effectively in paper-based documents.
Many web-based information systems continue to use individual vertical lists of links for each level in a hierarchy. Frick, et al. (1995) and Corry, et al. (1997) found that these lists worked well in meeting user expectations and resulted in high navigation performance on a university web site. In the same design, exemplars were added to each link to aid the user in understanding more about what each link would really provide them. Users in this study reported that these exemplars were helpful. Spool (1997) also reports that link descriptors have been shown in usability testing to be extremely effective aids in helping users to decide whether the link is in fact what they want or not.
Another navigation tool used extensively in online systems is the expanding/contracting hierarchy. Such lists of choices have been shown to be effective in the Windows operating system, as well as in the Apple Computer OS. Early hypertext systems such as Superbook have also been shown to use expanding/contracting outlines successfully (Egan, et al. 1989). Specific to the web, Nation, et al. (1997) conducted an evaluation of an expandable/contractible table of contents tool called WebTOC. The tool is a Java applet that displays a dynamic hierarchy on the left of the screen with the content to the right side. Early results show that the applet can assist users in navigating web sites. Lastly, expanding/contracting tables of contents have also been shown to decrease browsing times when compared with fully expanded versions (Chimera et al. 1994).
Another navigation aid that is popular on the web is what we call hierarchical chains. These chains are structural cues in that they communicate to the user exactly where in the hierarchy they are and where they are currently. Usually a horizontal line of hypertext items separated by colons or arrows, each item in the chain is a level in the hierarchy that is hot-linked, allowing the user to return with a single click to any previously visited level. Examples of this navigation strategy can be seen at two popular and heavily usability-tested web sites. They are:
Additional Design Considerations
After determining the overall hierarchical design, other issues that need to be addressed concern types of menu structure (embedded versus explicit) and sub-menu design. Some research indicates that embedded menus produce more accurate searches, permit fewer screens viewed, and are preferred overall by users (Koved & Shneiderman, 1986). Menu design can also be divided into three levels: alphabetical, categorical, and random. In terms of overall efficiency for novice users, there is no significant difference between alphabetical and categorical (Coll, Coll, and Nandavar, 1993).
Another consideration for web designers concerns the type of tasks that need to be supported. Navigation aids in a hypertext environment should be developed according to the tasks of the user. In terms of information search tasks, navigation via indexes seems to be the preferred method of users (Edwards and Hardman, 1989, cited in Smith, 1997). Developers should also provide information to the user informing them of their current progress, such as a path history (Jones, Farquhar and Surry, 1995). In terms of sub-menu design, studies by Scheurman and Peck (1991) indicate that users of sub-menus that provide a return to main menu option, versus a return to previous sub-menu option, spend less time on sub-menus and therefore are more efficient.
Other issues of importance that need to be considered are usability and its measurement, which have included various factors in the past such as: time to completion, number of errors, and deviations from optimal paths (Smith, 1994, cited in Smith, 1997). Other considerations could include finding required information, identifying the meaning of a link, and obtaining the required information when it is requested (Smith, 1997).
There are currently a number of different navigation strategies that designers implement in web-based information systems. Some of the many navigation options available to web designers include:
Methods
Research Design
The methodology used in this study was a classical experimental design with subjects randomly assigned to one of three treatment conditions. These three treatments were three web sites, which contained same information but different navigation menus.
The dependent variables in this study were the subjects= performance on the task component of the study, the number of times the user voluntarily used the back button, and each subject=s attitude. Attitude variables were constructed to collect subjective data about (a) appeal of the designs, (b) their usability, and (c) perceptions of "getting lost" in the web sites.
Pilot Tests
Six pilot tests were completed prior to actual data collection. The pilot tests were used to determine the usability of data collection instruments, to determine the difficulty of the tasks, and to ensure that data collection procedures worked satisfactorily.
Instruments tested during pilot testing included a background questionnaire, task list, and the attitude survey. The results of the pilot tests were used to ensure the background and the attitude questionnaires were understandable and usable. The pilot test was also used to finalize the task list. Over 30 tasks were tested during the pilot tests. The difficulty and usability of each task was determined based on the whether the subjects found the correct answer and the time it took to find the answer. The easiest tasks were deleted from the task list, and 24 tasks were selected for use in the actual study.
Instruments
The background questionnaire consisted of five questions about the subjects= web and computer experiences. The task instrument consisted of 24 items. The task items were standard usability tasks that users were asked in order to find specific information in the web site. The tasks were evenly distributed among the web site. The attitude questionnaire was an instrument consisting of 18 Likert-type questions and one open-ended question for general comments. The 18 items consisted of positively and negatively worded statements about the navigation model used, which were intended to measure its: (a) appeal, (b) usability, and (c) "getting lost" issues. In response to each statement, participants indicated which one of five ordered responses, ranging from Astrongly agree@ to Astrongly disagree@.
Prior to individual sessions with each subject, the researcher needed to set up the computer and browser for the test. The monitor screen resolution was set to 640 X 480. The browser was opened and maximized. The Adirectory buttons@ were disabled. The URL was set to the treatment tested in that session. The AHOME@ button on the navigation bar was set to the Home Page URL of the treatment. The Amemory cache@ and Adisk cache@ were set to zero so that web server logs would contain navigation moves from page to page. The IP number of the computer workstation was recorded for later web server log analysis. Finally, the link history list was reset.
Subjects
Potential participants were screened beforehand to help determine who would be appropriate for the research study. The major criterion that was used to select subjects was little or no prior experience with the Indiana University School of Education web site. Subjects were excluded who were already familiar with this web site, since prior knowledge might confound the results. The resulting 44 subjects were undergraduate and graduate students and public school teachers, who were randomly assigned to the three treatment conditions (n = 14, 15, 15 respectively).
After completing the background questionnaire, each subject was asked to find answers to the 24 information tasks (e.g., What is the phone number of the External Relations Office? Find the Web page for the Education Library. Find information pertaining to Graduate Admissions.). The tasks were randomly ordered for each subject, who was given up to three minutes to find the answer. When the subject believed she had found the answer, she notified the researcher. If the answer was not found within three minutes, the subject gave up, or if the subject came to the wrong place in the web site, it was counted as an incorrect response. The researcher also recorded task completion time, and navigation buttons used.
Immediately following the completion of the 24 task questions, each subject was asked to complete the attitude questionnaire. Most subjects completed the experiment in 45 minutes to an hour.
Results
Reliabilities of the attitude scales were estimated by Cronbach=s alpha for appeal (0.90), usability (0.89) and getting lost (0.70). The number of completed tasks, the total time for completing the tasks, the total number of voluntary back button clicks, and the attitude scales were analyzed by using the one-way ANOVAs. The statistical significance level used for hypothesis testing was set a priori to .05.
There was no significant difference among group means on number of tasks completed successfully.
Table 1. ANOVA for the number of tasks completed successfully
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
There was no significant difference among the three designs on the total amount of time to do all 24 tasks.
Table 2. ANOVA for the total task completion time in seconds.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
Table 3. ANOVA for the number of voluntary back button clicks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
There was no significant difference among groups with respect to the appeal of the navigation model.
Table 4. ANOVA for subjective perception of appeal
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
Table 5. ANOVA for subjective perception of usability
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
Table 6. ANOVA for subjective perception of Agetting lost@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
We should emphasize that the web site for the School of Education (which is similar to Design A in our study, with its bottom-of-page navigation system removed) was previously developed and revised through extensive usability testing, making it as easy as possible for users to find answers to typical information seeking tasks such as those in our study. The persistent or semi-persistent menus, which were added in Designs B and C in our study, apparently did not significantly enhance its perceived usability or appeal, which was very good from the start. With web sites where little or no usability testing is used in the design process, we can speculate that persistent or semi-persistent menu systems might make a bigger difference.
We strongly believe that more research is needed before we can be sure that no differences exist among these different navigational models, because several usability factors became apparent during the study. Factors that pertained to a majority of respondents are discussed below, and each of these behaviors was either observed by the research team or specifically mentioned by the subjects.
First, most subjects were not initially cognizant of the menu schemes in Designs B and C. In Design B, after six to seven questions subjects typically started to use the sidebar menu scheme that was provided in the left column. Most subjects mentioned that they did not notice the sidebar menu function until after the tasks had started. Also, it took answering a few questions before the subjects understood how the side bar in Design B was applicable to their task-searching behavior. In Design C, several subjects mentioned that they did not notice the hierarchical menu at the top until they had navigated the structure for several levels. It took considerably more time for subjects in Design C to notice and begin to use the menu chain at the top of the page versus the time required for subjects in Design B with the sidebar at the left of the screen. One exception to the time factor in Design B was a subject who was a media specialist who immediately understood and consistently used the menu.
A majority of subjects mentioned that the exemplars were the most beneficial aspect of navigation. Design A contained exemplars for all its menus. Designs B and C contained Design A with the addition of menu choices on the side or top, respectively, which did not contain exemplars due to limitations in screen real estate. We can speculate that if users visit a site frequently, they would learn the meanings of the labels used for the menu items and be less dependent on exemplars for deciding which part of the hierarchy they need to go to. Our study was relatively short (less than an hour), and subjects were screened to ensure that they were not familiar with its structure.
In addition to the exemplar confound, most subjects used the home button for navigation at the beginning of each task. As part of each treatment, the home button was set to go to the home page for that design. For most subjects, it was much easier to navigate to the home page using the home button and then to use the exemplars to find specific pieces of information. This would be atypical for most web sites, and hence may have confounded our results.
Most subjects mentioned that during Design B the sidebar menu should have been contained on the home page as well as the other pages in the document. Subjects stated that having the sidebar menu on sub-levels and not on the main level was confusing. They did not understand how the side bar menu was helpful until they discovered it, typically after six to seven questions.
Subjects were confused by the dead links. In order to keep subjects from leaving the experimental web designs used in the study, links to web pages outside the navigation models were disabled intentionally to prevent confounding by other inconsistent navigation schemes used on external pages. It was not practical to reproduce the entire School of Education web site for each of the three treatment conditions, which would have consisted of literally thousands of web pages developed by different departments and units. Consequently, several subjects gave answers to the task they were completing based on the link name with exemplars itself instead of clicking and following the link.
Conclusion
In summary, the results were somewhat disappointing in that we expected to find designs B and C to be more efficient with respect to time to find answers to information seeking tasks. If choices are visually present to navigate by means of shortcuts to other parts of a menu hierarchy, then this increases the compactness of the hypermedia system (i.e., shorter distances between nodes). It also increases the flexibility, since users have multiple ways to move around a site. However, our data from this study do not support these expectations (with the exception that Design B may decrease reliance on the use of the back button for navigation, since the differences approached statistical significance). One could conclude that there are indeed no differences among the three types of navigation models, and hence could use other design criteria for making a choice among them. For example, if screen space is needed to minimize scrolling, then traditional non-persistent menu structures are more efficient with respect to use of screen real estate. The purpose of the web site might also dictate which model is preferable. For example, Yahoo is a large clearinghouse for links to the web (a Ayellow pages@ function). For such a large, deep hierarchy of choices, the use of semi-persistent hierarchical chains (or textual maps, as represented in our Design C) occupies less screen real estate than a sidebar menu and yet provides flexibility for moving around the hierarchy.
We suggest that future research should be done, given constraints we faced in this study and the possibility of confounded results due to those constraints.
Bibliographic References
Chang, C. T. (1995). A study of hypertext document structure and individual differences: Effects on learning performance. Master Abstracts International, 1465, AAC - 9624305.
Chimera, R. and Shneiderman, B. (1994). An exploratory evaluation of three interfaces for browsing large hierarchical tables of contents, ACM Transactions on Information Systems, 12(4), 383-406.
Coll, J. H., Coll, R., & Nandavar, R. (1993, August). Attending to cognitive organization in the design of computer menus: A two-experiment study. Journal of the American Society for Information Science, 393-397.
Corry, M., Frick. T. and Hansen, L. (1997). User-Centered Design and Usability Testing of a Web Site: An Illustrative Case Study. Education Technology Research and Development, 45(4), 65-76.
Egan, D., Remde, J. R., Gomez, L. M., Landauer, T. K., Eberhardt J., Lochbaum, C. C., Formative (1989) Design-Evaluation of Superbook, ACM Transactions on Information Systems, 7(1), 30-57.
Frick, T., Corry, M., Hansen, L. Maynes, B. (1995). Design-research for the Indiana University Bloomington World Wide Web: AThe Limestone Pages.@ See URL: http://education.indiana.edu/faculty/iuwebrep.html
Harney, C. L. (1990). Performance tradeoffs in computer menu design. Dissertation Abstracts International, 495, AAC - 1342814.
Jacko, J. A. (1993). Models for hierarchical menu design: Incorporating breadth and depth, task complexity, and knowledge structure of the user. Dissertation Abstracts International, 4861, AAC - 9127488.
Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995, July-August). Using metacognitive theories to design user interfaces for compute-based learning. Educational Technology,12-22.
Koved, L., & Shneiderman, B. (1986, April). Embedded menus: Selecting items in context. Communications of the ACM, 312-318.
McKnight, C., Dillon, A., Richardson, J. A., A comparison of linear and hypertext formats in information retrieval. In R. McAleese & C. Green (Eds.), Hypertext: The state of art. Oxford, England: Intellect Books, 1990.
Nielsen, J. (1996). The rise of the sub-site. Alertbox Column. See URL: http://www.useit.com/alertbox/9609.html
Rouet, J., Lovonen, J., Studying and learning with hypertext: empirical studies and their implications. In Rouet J. F., Lovonen J. J., Dillon, A., Spiro, R. J. (Eds.): Hypertext and Cognition. New Jersey: Lawrence Earlbaum, 1996.
Sano, D. (1996). Designing large-scale web sites: A visual design methodology. New York, NY: John Wiley & Sons.
Schuerman, R. L., & Peck, K. L. (1991, Summer). Pull-down menus, menu design, and usage patterns in computer-assisted instruction. Journal of Computer-Based Instruction, 93-98.
Smith, P. A. (1997). Virtual hierarchies and virtual networks: Some lessons from hypermedia usability research applied to the World Wide Web. International Journal of Human-Computer Studies. [Online]. Available: http://ijhcs.open.ac.uk/index.html.
Spool, J. M., Scanlon, T., Schroeder, W., Snyder, C., & DeAngelo, T. (1997). Web site usability: A designer=s guide. Andover, MA: User Interface Engineering.