Development of Web-based IR Systems: A Review

Information retrieval (IR) has traditionally been the domain of librarians and information professionals. IR systems have been used almost exclusively by such search experts for several reasons, such as the number of search systems available, cost, and the complexity of use requiring command language searching. However, with the rapid growth of the Internet, together with tools like World Wide Web (also known as WWW, or simply the Web), there have been significant changes and improve­ments in online information retrieval environments. These include a broad and diverse existence of both IR systems and various user interfaces and functions. This paper briefly reviews the developmental history of Web-based IR systems that are available for library reference services.

Development of Web-based IR Systems
The first IR systems allowing online searching were introduced in the early 1970s. During the 1960s, experimental IR systems were developed by libraries for the storage and retrieval of their own in-house information resources. In 1964, the US National Library of Medicine (NLM) offered on-demand batch searching (i.e., off-line) of the MEDLARS system.1 By the following year, Lockheed (Dialog), System Development Corporation (SDC), and Chemical Abstracts Service (CAS) began work on their own online search services.2 In 1968, the first online search started from the State University of New York/Biomedical Communications Network (SUNY/BCN) in Albany to the MEDLARS database in Bethesda, Maryland, using dedicated lines.3 By 1969, the first packet-switched data communications network (ARPANET) had begun test operation at University of California in Los Angles (UCLA).4 The main aim was to link academic, research and military establishments in the US.

In the early 1970s, the information retrieval (IR) industry began to shift from off-line, batch-processing services to the development of online services that process requests for information entered at remote terminals in a time-sharing mode.5 The first major online search service was the NLM’s MEDLINE, the online version of MEDLARS system, which began operation in 1971.6 It was quickly followed by the commercial online search services offered by Dialog in 19727 and SDC Orbit in 1973.8 Mead Data Central introduced the LEXIS service in 1973.9 Several other commercial online search services were introduced in swift succession. The West Publishing Company introduced Westlaw in 1975.10 European host services, such as European Space Agency -Information Retrieval Service (ESA-IRS), and British Library Automated Information Service (BLAISE), were launched during the mid-1970s.11 In 1977, Bibliographic Retrieval Services (BRS) began commercial online service.12 The Dow Jones News/Retrieval service, the first online business information service, was also introduced in the same year.13

These early IR systems used complicated command language interfaces. Williams14 reviewed IR interfaces and examined both batch and online searching software, some of which included user aids such as online thesauri. Williams15 again briefly reviewed the state of the art as of late 1977 and identified several trends for the future. Among these were a more “transparent user interface” to help users avoid the need for understanding all the specific differences of databases, systems, command languages, vocabularies and access protocols. She argued that such systems would greatly increase the usability of online databases both by professionals and end-users. Hawkins16 reviewed the history of commercial online searching until 1981, including interface aspects. He noted that IR systems required search experts because of the complexity of the interfaces and differences between systems. Micho and Lee17 identified similar problems.

The first intensive marketing to end-users came in the early 1980s after IBM introduced its PC.18 Several online services introduced new products and services to attract the end-user market. Some services offered simplified versions of their search interfaces and subsets of their databases, available only after business hours and at a significant discount. BRS/After Dark and Dialog’s Knowledge Index offered low-cost access to anyone with a dial-up terminal to a variety of databases in evenings and weekend hours.19 At the same time, “front-end” software packages were developed to make online searching more accessible to end-users. The capabilities and purposes of these packages varied considerably. Some offered merely automatic dial-in and logon facilities, some provided extensive augmentation of one system or of one or a few databases, and others offered more limited assistance for multiple systems and databases.20 These front-ends, however, had not been well accepted, mostly because the underlying systems were difficult for end-users to use successfully in searching.21 In the 1980s, efforts were also made to apply expert systems and natural language techniques to online searching. Several operational prototypes were made, but they all had very little influence on commercial online searching.

In the 1980s, online search systems used a variety of interaction styles to make online searching more accessible to users. For example, After Dark subscribers used menus; Knowledge Index subscribers could choose to search with simplified commands or with menus.22 BRS/BRKTHRU offered a subset of commands which was easier to learn. EasyNet offered a common command language to facilitate searching on  various retrieval systems. ProSearch provided the user with the option to choose one language from a predefined set of host languages and to use it to search all online hosts accessed by the search system.23 Crawford and Edwards24 described a front-end software that took advantage of direct manipulation interface to search information on Dialog. The FIRSTUSER used a combined menu and form filling interface.25

There was a continued development of commercial search services in the 1980s. Mead Data Central introduced its NEXIS service in 1980.26 In the same year, DataStar was launched by a consortium led by Radio Suisse, and became one of the most important online vendors for business and bio-medical information in Europe.27 In 1984, H. W. Wilson began offering online access to its popular indexes via WILSONLINE.28 By the mid-1980s, several online commercial search services were available to libraries. Commercial search services had also increased in sophistication and in capabilities offered, and continued to add databases regularly. By 1983, Dialog offered over 170 databases; SDC and BRS offered more than 70 databases each.29 In the mid-1980s, databases were also published on CD-ROMs for mounting at user sites. They had a dramatic impact on end-user searching in a way that was free at ‘point of search.’ Some CD-ROMs also offered menu-based interfaces which also helped end-user searching.

There was much commercial activity amongst online service vendors in the 1980s. Pergamon Infoline, a Maxwell Company acquired Orbit in 1987.30 In 1988, Knight-Ridder purchased Dialog Information Services.31 Maxwell bought BRS Information Technologies and renamed the whole group as Maxwell Online in 1989.32 However, despite end-user movement in the 1980s, high price and unfriendly search interfaces largely restricted online searching to professional searchers. In all this time, very little was done to enhance the search interfaces. Rather, online search services concen­trated on loading more and more databases.

In the early 1990s, a few attempts were made to enhance the IR interfaces. In 1992, Westlaw introduced Westlaw Is Natural (WIN), the first natural language application in the commercial online environment.33 Mead Data Central quickly followed with its FREESTYLE in 1993.34 Dialog’s similar offering, TARGET was also launched in the same year.35 All these three systems provided ranked lists of retrieved documents. The proliferation of online search services continued into the 1990s. OCLC launched its EPIC service in 1990.36 In 1991, OCLC introduced FirstSearch, a menu-based search service designed for end-user searching.37 Proquest Direct from UMI were introduced in 1995.38 In the same year, Profound, a business-oriented search service, was introduced by MAID.39 Online Search services were also sold and consolidated in the 1990s. Maxwell Online became InfoPro Technologies in 1992.40 Thomson acquired ISI in the same year.41 In the following year, Knight-Ridder acquired the DataStar service from Radio Suisse.42 Reed-Elsevier bought LEXIS and NEXIS services from Mead Data Central in 1994.43 In the same year, InfoPro Technologies sold its three divisions – Orbit, BRS, and BRS Software – to Questel, CD Plus, and DataWare respectively.44 CD Plus changed its name to Ovid Technologies in 1995.45 West Publishing Company merged with Thomson Legal Publishing to form West Group in 1996.46 The Dialog Corporation was formed through the merger of MAID and Knight-Ridder in 1997.47 In 2000, Thomson acquired Dialog’s Information Services Division, including Dialog, DataStar, and Profound services.48

Since mid-1990s, several Web-based search services began their operation. Some online services reconfigured their services for Web implementations that replaced or coexisted with earlier versions. OCLC’s FirstSearch is an example of an existing search system which was reconfigured for Web access. The Web-based version of FirstSearch was released in 1996.49 This replaced FirstSearch’s original menu-driven interface. The NLM replaced its conventional search service with two Web-based offerings: PubMed and Internet Grateful Med.50 DataStar Web was launched in December 1996.51 ISI introduced a Web interface to its citation databases in 1997.52 Dialog released Web-based DialogWeb and DialogClassic on the Web in 1997 and 1998 respectively as alternatives to its Classic Dialog service.53 The Proquest Direct service introduced its Web access in 1996.54 Ovid Technologies released the Ovid Web Gateway in the same year.55 By the following year, WilsonWeb was launched as the Web-based successor to the WILSONLINE.56 Dow Jones introduced Web version of News/Retrieval service in the same year.57 West Group launched the Web-based access to Westlaw in 1998.58 Today, almost all major IR systems have Web access to their services.

A fundamental characteristic of Web-based IR systems is that they are inherently interactive and provide a variety of ways for users to interact with both information and systems. Xie and Cool59 identified the following advantages of Web IR systems: (a) guide user access to a variety of databases; (b) facilitate multiple search strategies; (c) assist using thesaurus terms; (d) offer interactive help facilities; (e) offer multiple manipulations of output; and (f) provide iterative movement of links. Despite these advantages, many Web-based IR interfaces are still difficult to use. This results in confu­sion, frustration, and failure for both novice and experienced users.

There have been some studies comparing the effectiveness of Web and non-Web interfaces in IR systems. Koehler and Mincey60 compared the dial-up access and Web-based access methods, and concluded that FirstSearch Web was a major improvement over the dial-up access. Bates61 compared Web-based packages Dialog Web and DataStar Web, with the Classic, ASCII, dial-up version of Dialog. She noted the benefits of the Web-based version, but also considered the Web-based product to be less efficient and responsive for the experienced users than the ASCII product.

Barker62 compared online searching on DataStar using the Classic command language interface with access via the Web interface DataStar Web. The functionality of the two interfaces was compared in terms of entering the system, selecting a database, searching, output and display, terminating the session, error messages and help pages, and support, training and documentation. She concluded that although both interfaces offer access to the same databases, there were significant differences. Many of these differences may affect search performance by both novice and experienced users.

Xie and Cool63 also compared Web and non-Web interfaces and found that some of the functions of Web-based interfaces outperformed non-Web interfaces, but at the same time they were not universally preferred. Experienced searchers preferred both greater user control and greater ease of use in the search process. They concluded with an argument that greater attention should be paid to the tension between user control and ease of use in the design of effective and useful interactive IR systems.

The rate of growth of all aspects of IR industry has been truly extraordinary since the beginning in the early 1970s. Williams64 estimated that from 1975 to 2000, the number of online databases increased from 301 to 12,417; prod­u­cers had increased from 200 to 4,017; and vendors had grown from 105 to 2,891. In addition, the estimated number of online searches had increased from approximately 7.5 million in 1982 to 90 million in 1998.65 It is hardly an understatement to say that the world of online searching has changed drama­tically with the development of the Web. As Saffady66 pointed out, “the Web has prompted the development of new online search service, forced existing services to reconfigure their offerings and improve their user interfaces, and catalyzed other industry trends, such as end-user searching and the develop­ment of new pricing models.”

This review suggests that there has been a general lack of attention given to interface issues by database vendors. Early IR systems used very structured command mode interfaces. The users of these early systems were largely professional searchers as intermediaries in library settings. Then attempts have been made to develop better interface design for end-users using features such as menu-selection, form fill-in, natural language, and direct manipulation interfaces. The development of the Web-based IR systems has already left a tremendous mark on user access to such systems. First, the Web has made vast quantities of information resources available globally. Second, it has made cheaper and wider end-user access to various online databases.

Despite recent development of Web-based IR interfaces, little empirical research has been carried out about the usability of these systems with respect to interface features and functionalities. More­over, reviews of Web IR systems have been uncritical in their discussion of the search interface, or have been limited to subjective comments. Although few studies suggested the need for improving search interfaces, they fail to provide specific recommendations. Thus, the impor­tance of evaluating the interfaces and end-user searching is crucial for the future development of the technology and its use in information access. Some of the problems that IR faces today can obviously benefit from Human-Computer Interaction (HCI) research. In particular, the maturity of user-centred design can help developing more usable IR systems and assessing how these systems are adapting to user needs.


1.      Bourne (CP) (1980). Online systems: history, technology and economics. Journal of the American Society for Information Science, 31(3), 155-160

2.      Chowdhury (GG) (1999). Introduction to Modern Information Retrieval. London: Library Association Publishing

3.      Saffady (William) (2000). The state of library automation at 2000. Library Technology Reports, 36(1), 67-101

4.      Lynch (CA) and Preston (CM) (1990). Internet access to information resources. In: Williams (ME) ed. Annual Review of Information Science & Technology, 25. Amsterdam: Elsevier Science, 263-315

5.      Lancaster (FW) (1978). Towards Paperless Information Systems. New York: Academic Press

6.      McCarn (DB) (1980). MEDLINE: an introduction to online searching. Journal of the American Society for Information Science, 31(3), 181-192

7.      Walker (G) and Janes (J) (1999). Online Retrieval: A Dialogue of Theory and Practice. Englewood, Colorado: Libraries Unlimited

8.      Borgman (CL), Moghdam (D), and Corbett (PK) (1984). Effective Online Searching: A Basic Text. New York: Marcel Dekker

9.      Hartley (RJ), Keen (EM), Large (JA), and Tedd (LA) (1990). Online Searching: Principles and Practice. London: Bowker-Saur

10.  West Group (2003). About Westlaw. Available at:

11.  Orton (D) (1995). Online Searching in Science and Technology: An Introductory Guide to Equipment, Databases and Search Techniques. London: The British Library

12.  Rowley (JE) (1987). Organising Knowledge: An Introduction to Information Retrieval. Hants: Gower

13.  Moulton (JC) (1979). Dow Jones News/Retrieval. Database, 2(1), 54-64

14.  Williams (ME) (1974). Generation and use of machine-readable bibliographic databases. In: Cuadra (CA), Luke (AW) and Harris (JL) eds. Annual Review of Information Science & Technology, 9. Washington, DC: American Society for Information Science, 221-284

15.  Williams (ME) (1978). Online retrieval – today and tomorrow. Online Review, 2(4), 353-366

16.  Hawkins (DT) (1981). Online information retrieval systems. In: Williams (ME) ed. Annual Review of Information Science & Technology, 16. New York: Knowledge Industry Publications, 171-208

17.  Micho (WH), and Lee (J) (1987). End-user searching of bibliographic databases. In: Williams (ME) ed. Annual Review of Information Science & Technology, 22. New York: Elsevier Science, 227-263

18.  Tenopir (Carol) (1995). Impacts of electronic reference on instruction and reference. In: Shirato (L) ed. The Impact of Technology on Library Instruction: Papers and Session Materials Presented at the 21st National LOEX Library Instruction Conference, 14-15 May, Racine, Wisconsin. Ann Arbor, MI: Pierian Press, 1-8

19.  Janke (RV) (1984). Online after six: end-user searching comes of age. Online, 8(6), 15-29

20.  Borgman (CL), Case (DO) and Meadow (CT) (1989). The design and evaluation of a front-end user interface for energy researchers. Journal of the American Society for Information Science, 40(2), 99-109

21.  Harman (D) (1992). User-friendly systems instead of user-friendly front-ends. Journal of the American Society for Information Science, 43(2), 164-174

22.  Kennedy (B) (1993). Comparing menu systems for end-users: After Dark, Knowledge Index, and FirstSearch. Online, 17(4), 52-57

23.  Efthimiadis (EN) (1990). Online searching aids: a review of front ends, gateways and other interfaces. Journal of Documentation, 46(3), 218-262

24.  Crawford (RG) and Edwards (ME) (1985). A prototype mouse-based interface to drug-related information. Online Review, 9(6), 471-487

25.  Crawford (RG) and Becker (HS) (1986). A novice user’s interface to information retrieval systems. Information Processing & Management, 22(4), 287-298

26.  Provenzano (D) (1981). NEXIS. Database, 4(4), 30-41

27.  Convey (John) (1989). Online Information Retrieval: An Introductory Manual to Principles and Practice. London: Clive Bingley

28.  Seale (C) (1989). Getting the most out of WILSONLINE. Database, 12(2), 55-61

29.  Borgman (CL), Moghdam (D), and Corbett (PK) (1984). Op cit.

30.  Lambert (N) (2000). Orbit and Questel-Orbit: farewell and hail. Searcher, 8(2), 73-76

31.  Hartley (RJ), Keen (EM), Large (JA), and Tedd (LA) (1990). Op cit.

32.  Orton (D) (1995). Op cit.

33.  Paris (LA) and Tibbo (H R) (1998). Freestyle vs. Boolean: a comparison of partial and exact match retrieval systems. Information Processing & Management, 34(2-3), 175-190

34.  Basch (R) (1997). Online news: twenty years of online headlines – reprinted from the newpages of earlier Online and Database magazines. Online, 21(1), 67-70

35.  Losee (RM) and Paris (LA) (1999). Measuring search-engine quality and query difficulty: ranking with Target and Freestyle. Journal of the American Society for Information Science, 50(10), 882-889

36.  Whitcomb (L) (1990). OCLC’s EPIC system offers a new way to search the OCLC database. Online, 14(1), 45-50

37.  Kennedy (B) (1993). Op cit.

38.  Saffady (William) (2000). Op cit.

39.  Basch (R) (1997). Op cit.

40.  Questel-Orbit (2003). About Questel-Orbil. Available at:

41.  Institute for Scientific Information (2003). Company Timeline. Available at:

42.  Dialog Corporation (2003). Key Dates for the Dialog Corporation. Available at:

43.  Reed-Elsevier (2003). Company History. Available at:

44.  Orton (D) (1995). Op cit.

45.  Ovid Technologies (2003). The History of Ovid Technologies. Available at:

46.  Thomson Corporation (2003). History of the Thomson Corporation. Available at:

47.  Large (A), Tedd (LA), and Hartley (RJ) (1999). Information Seeking in the Online Age: Principles and Practice. London: Bowker-Saur

48.  Thomson Corporation (2003). Op cit.

49.  Perlman (Gary) (2000). The FirstSearch user interface architecture: universal access for any user, in many language, on any platform. In: Proceedings of the Conference on Universal Usability, 16-17 November, Wasington, DC. New York: ACM, 1-8

50.  Saffady (William) (2000). Op cit.

51.  Barker (AL) (1997). DataStar Web: living up to the hype? An evaluation of the interface and search system. In: Online Information 97: Proceedings of the 21st International Online Information Meeting, 9-11 December, London. Oxford: Learned Information, 213-222

52.  Oxley (H) (1998). ISI spins a Web of Science. Database, 21(2), 37-40

53.  Dialog Corporation (2003). Op cit.

54.  Basch (R) (1997). Op cit.

55.  Ovid Technologies (2003). Op cit.

56.  H. W. Wilson (2003). Historical Highlights from H. W. Wilson’s Past. Available at:

57.  Marcus (J) (1998). Dow Jones on the Web. Database, 21(1), 76-76

58.  West Group (2003). Op cit.

59.  Xie (H) and Cool (C) (2000). Ease of use versus user control: an evaluation of Web and non-Web interfaces of online databases. Online Information Review, 24(2), 102-115

60.  Koehler (WC) and Mincey (D) (1996). FirstSearch and NetFirst-Web and dial-up access: plus ca change, plus c’est la meme chose? Searcher, 4(6), 24-28

61.  Bates (Mary Ellen) (1997). Knight-Ridder on the Web: a brave new world for searchers? Searcher, 5(6), 28-37

62.  Barker (AL) (1998). DataStar Web: a comparison with ‘classic’ DataStar command language searching. Online & CD-ROM Review, 22(3), 55-67

63.  Xie (H) and Cool (C) (2000). Online interface comparison: features and functionalities. In: Proceedings of the 21st National Online Meeting, 16-18 May, New York. Medford, NJ: Information Today, 513-522

64.  Williams (ME) (2001). Highlights of the online database industry and the Internet: 2001. In: Proceedings of the 22nd National Online Meeting, 15-17 May, New York. Medford, NJ: Information Today, 1-4

65.  Williams (ME) (2000). Highlights of the online database industry and the Internet: 2000. In: Proceedings of the 21st National Online Meeting, 16-18 May, New York. Medford, NJ: Information Today, 1-6

66.  Saffady (William) (2000). Op cit.

  Related Posts