Journal of Digital Information, Volume 1 Issue 3
Article No. 11, 1998-09-18
To cite this paper please include the details above in the full reference
Themes: Digital libraries

Peer reviewed paper

Printable version available.

Abstract

During 1996, the number of scholarly periodicals available in electronic form increased rapidly. The Café Jus project took advantage of this critical mass of electronic journals to mount a major user study with taught postgraduate students, research students and staff in various disciplines at Loughborough University. The main conclusions were that low-level technical problems are still a deterrent to the use of electronic journals; that people prefer not to read at length on screen, but printing out is slow; that commercial publishers tend to follow the lead of technology rather than consider the convenience of their users; that at present there is a significant need for user training, exacerbated by the variety of publishers' interfaces and their speed of change; and that free journals using HTML are preferred to commercial journals using PDF for convenience of reading, but they are likely to be regarded as of lower academic quality. The implications of these results for publishers and for the future of electronic journals are discussed. 
 
 

1  Introduction

Alongside and, in a number of cases, preceding the development of dual print/electronic versions of existing journals by commercial publishers, scholars in a range of academic fields have begun to found new, usually electronic-only, journals. Unlike commercial electronic journals, which are normally priced along similar lines to their printed counterparts, or even tied to the print subscription, most of these journals are distributed free of charge over the Internet. In a number of cases, the founders of these journals see them as offering an alternative publication route, in place of some of the increasingly costly specialist products of commercial publishers (e.g. Harnad 1994, Odlyzko 1996). However, it has been observed that most of the free electronic journals have so far attracted rather few submissions, and therefore publish relatively few articles per year (Harter and Kim 1996). It should also be noted that the two models discussed here -- free journals on the Internet, and electronic versions of existing commercially published journals, sold on subscription, under site licensing arrangements, or by 'pay-per-view' -- are not the only possible models of scholarly communication systems. A full discussion of many possibilities is given in Okerson and O'Donnell (1995). The two discussed here actually exist, however, and are therefore susceptible to user studies.

The rapidly expanding numbers of both commercial and free electronic journals have emphasised the need for user studies, whilst also increasing the possibility of carrying them out. Many libraries are now taking steps to provide their users with access to relevant electronic journals. One pressing question, as discussed by Kidd (1997), is how best to do this. In 1995, Loughborough University Library began to make available, via its World Wide Web pages, a collection of links to electronic journals. Titles were selected initially from the free journals found on the Internet. The criteria for choice were that the journals were, first, peer-reviewed and, second, related to the teaching or research interests in the University. At the same time, it became clear that there would be a major growth in the availability of electronic versions of commercially produced journals during 1996 (Woodward 1997). The Library began negotiations with publishers to allow trial use of some of their electronic journals, but this initiative was overtaken, in part, by a new project started by the Higher Education Funding Councils (HEFCs) in the UK. Their Pilot Site Licence Initiative (PSLI) was designed to allow access by universities to the electronic versions of journals produced by the Institute of Physics and Academic Press (and, subsequently, the two Blackwell imprints) on favourable terms (Bekhradnia 1995). In addition, Chapman & Hall allowed Loughborough University free access to its electronic journals for the duration of the Café Jus project.

As a consequence of these developments, a total of some 300 refereed electronic journals were available to staff and students at Loughborough by mid-1996. The Library restructured its Web electronic journal pages to take account of this increased number and diversity of types. Separate pages were created for free electronic journals, free sample issues of electronic journals to which the University did not subscribe, the prototype Blackwell's Navigator service, and for the commercial electronic journals to which the University did subscribe. The last-named page had links to the home pages of each of the publishers involved. (This was because each commercial publisher had its own access control system which required entry via its own home page.)

By 1996, conditions at Loughborough were appropriate for implementing  a detailed study of users' reactions to electronic journals. A range of titles was now available across a variety of subjects, with easy access from anywhere on the campus. In anticipation of this development, the Café Jus (Commercial and Free Electronic Journals User Study) project was initiated in 1995, with funding from the British Library Research and Innovation Centre. Its main aim was to examine what problems readers with differing subject and computing backgrounds experienced when using electronic journals. A supplementary aim was to compare readers' reactions to free and commercial electronic journals. The freely available journals can generally be accessed by a direct hypertext link from the library's Web page to the journal's own home page, and they are usually in HTML; commercial journals require registration on the publisher's home page, and the full texts are generally in PDF. Were the former considered as on a par with the latter by academic readers?
 

2  Methodology

A structured questionnaire was developed which could be used in conjunction with hands-on access to an electronic journal. It is shown in Appendix 1 (**if you have Acrobat Reader, go directly to Appendix 1 ). The questions were arranged under three headings - journal content, journal appearance and facilities offered by the journal. This questionnaire was tested with volunteers from the 1995/96 cohort of master's students in four departments - Computer Studies, Human Sciences, Information and Library Studies, Physical Education & Sports Science. The students were introduced to the electronic journals in groups, which met in the computer-based teaching laboratories of the Department of Information & Library Studies. An initial presentation on the project together with documentation was provided; then each student accessed an electronic journal, noting down responses to the questionnaire as their reading progressed. If they had time, they could move on to another journal. Dealing with the students in groups made it possible to provide advice and help in real time, and also ensured completion of the questionnaires.

The pilot study indicated that the planned organization of the investigation was acceptable, but that there were minor ambiguities in some of the questions (more especially those dealing with journal facilities). After modifications to the questionnaire, the main study was launched with master's students from the 1996/97 cohort. The same four departments provided students, along with two more - Civil Engineering and Economics. Appendix 2 shows the full lists of titles accessed by at least one student during the pilot and main studies.

For comparison, it was decided to obtain feedback on electronic journal usage from research students and academic staff. In this part of the project, users were given individual instruction in the use of electronic journals, and then left to use them on their own at their own desks. The method of obtaining data here was via a log sheet on which participants were asked to keep a record of their usage of electronic journals. Volunteers were sought mainly via electronic mail messages inviting participation, but this proved to be a good deal less effective as a recruiting tool than the group approach used with master's students.
 

3  Results

One change made between the pilot and the main study was to introduce a question about the computer expertise of the respondents. This was to help identify any major differences between groups of students that might affect their handling of electronic journals. The overall distribution of expertise in the main study was found to be: limited expertise - 20%; moderate expertise - 50%; considerable expertise - 30%. There was no significant difference in this distribution for male/female or UK/overseas groups. In subject terms, only Computer Studies - where all students had considerable expertise - differed from the norm.

Apart from the master's students, feedback was obtained from eight staff on 38 attempts they made to access electronic journals, and on 29 attempts made by four research students. In both cases, the great majority of the accesses were to commercial journals. Overall, staff and research students were more experienced than the master's students both in using journals and in handling information technology.
 

3.1  Results: master's students

3.1.1  Relevance and quality

It is not surprising that the contents of commercial online journals were thought to be of similar standing to those of printed journals, since they were essentially parallel publications of the same material. The likelihood of being acquainted with the electronic journal increased between the pilot and the main study, in step with the increasing number of titles becoming available. The number of respondents who already read the printed version of the electronic journal they were examining almost doubled from the pilot study (18%) to the main study (32%).
 

3.1.2  Access

Most students expected to be able to download full-text articles that interested them to the screen. This often proved to be time-consuming. Some longer articles took so long to download that the session timed out, and students had to log in again. In its favour, IDEAL gives file sizes, and this assisted students in deciding whether or not to download. Not all the publishers provide this kind of help.

The following comments by students, in answering a question about whether they encountered difficulties, indicate the type of access problems that were found.

• "Generally no, although the fact that the IDEAL username box isn't long enough for 'Loughborough' to fit in could cause confusion."
• "Finding it [the publisher's electronic journal site] was relatively easy, but it did take some time to load - about 5 minutes [and this was at 9:00 am!] It then took another few minutes to get from the IDEAL menu to the listing of journals."
• "Yes! Nowhere on the front screen tells you to press the SUBMIT button to get into the system. I presumed you just had to hit the ENTER key. It took 10 - 15 minutes to get to the first journal article!"
• "I could not get into the full text of one article; the message that came up was 'Adobe Type Manager 3.6. or newer, must be installed. Acrobat Reader will now quit.' This happened twice. We started to access the system at 2:00 pm; it is now 2:50 pm and I have not been able to see a full text article."
• "No, but we did have assistance."

• The means for gaining entry to an electronic journal can be far from obvious, and trying to guess how to obtain access can be time-consuming.
• There is a need for prior knowledge (e.g. who the publisher is; what password should be used).
• Not all journal titles immediately indicate the limits or nature of the journal coverage (e.g. PACS-R), and a reader cannot gain an intuitive picture as with a printed journal.
• Users dislike publishers who assume that all readers have access to the latest software. They especially detest publishers who upgrade their interface requirements without adequate prior notice.
• Most of the users felt that they would have had problems accessing an electronic journal for the first time without additional guidance.

3.1.3  Layout and navigation

Students clearly had some difficulty in understanding how some features operated. To look at this further, it was necessary to look at titles which had been examined by a number of readers independently. The two titles selected for this purpose were Construction Management and Economics and Journal of Sports Sciences. An analysis of these showed that most confusion reigned over the question of whether hypertext links existed within articles. There was also some uncertainty in replies relating to the presence of hypertext links to other documents, whether the journal allowed browsing through tables and diagrams, and whether it supported searching across articles.

Although most students regarded the various icons as easy to use, there was often some confusion until they became accustomed to them. One problem was that the icon might not immediately call to mind the operation concerned. For example, some failed to guess that a man with a spanner represented 'services'. Another problem was misinterpretation of the activity involved. For example, some expected the 'back' button to return them to the contents page rather than the previous page of text. It was also not immediately clear to many how to exit from an article in Acrobat. In addition, the help facilities were not found to be very helpful. Indeed, a number of the students did not manage to find them. As one commented, "Could not find them - they should be obvious to novice users!"

The various problems encountered by groups of users trying to read the electronic journals can best be indicated by the account appended below (in abbreviated form) of a session undertaken with a group of Computer Studies students. The session was more fraught than most, even though the students had considerably more computer expertise than the average.

The first problem in this session was that one of the commercial publishers had changed the layout and appearance of its site (without prior notification). One student was thrown off the system completely with the message, 'System error - Netscape Type 2.02', and had to log in again. Another student said he had been going round and round trying to find the search engine. The alphabetical list of journals was squashed into the bottom left-hand corner of the screen, and was almost too small to be read. In addition, the green lettering on a white background proved difficult to decipher. Another student got into the Home page of the selected journal, but could not find the contents pages, nor could he find a way of accessing a document. This problem was finally resolved by a telephone call to the publisher. A further student received the message, 'The file ATM Font Database must be in the system folder. Please re-install ATM. Acrobat Reader will now quit.' He thought this meant he should move to another machine, but it was suggested he should try another document on the same machine. He got into that satisfactorily, but then received the message, 'An error occurred and has been logged.' He then found that he could not retrieve the article he wanted. After several further attempts, he commented, 'This is getting on my nerves.' A fellow student encountered the same error message, and tried every journal on the list. The same thing happened each time. He tried logging in again, and failed to gain access at all. Several students wanted to print articles, but had to wait so long that they eventually gave up.

3.1.4  Perceived advantages to the electronic journal

• Finding specific information (e.g. appropriate articles) is seen as being easier than for printed journals. Student comments included:
• "The electronic version has excellent searching capabilities - that feature is extremely helpful and would save time in research."
• "Easier to find related articles."
• Electronic journals avoid some of the problems of using printed journals. Student comments included:
• "Can manipulate the electronic version: can view lots of journals without having to go out of your mind trying to find all the paper versions in the library."
• "The advantage of the electronic version is that it is readily available at any time as compared to the paper version, which may be in use by another person when you want to read it."

 

3.1.5  Perceived disadvantages to the electronic journal

• Electronic journals are not always easily usable by novices. Student comments included:
• "There are no instructions or guidelines immediately accessible to help you find out how to use the electronic journal, and to move around the system. To be honest, it's not very user-friendly."
• Some operations take more time. Student comments included:
• "It is easier to turn the page of paper journal."
• "At the moment it is too slow to connect, and disconnection occurs too easily."
• "Takes too long to move from one journal to another."
• Reading from the screen is less pleasant. Student comments included
• "To read a screen is tiring."
• "The graphics were very grey and boring."

3.1.6  Comments from discussions in sessions

• The use of different passwords for different publishers was confusing.
• Some journal material was in a different format depending on when it had been input, with the older material being more restricted in its form of presentation.
• Some journals had been scanned in. This led to a generally indistinct appearance of text and graphics on screen.
• Changes made by publishers without warning are difficult even for computer-literate readers to disentangle.
• The more complex the journal, the greater the potential software problems. For example, one engineering journal required additional software (the Shockwave plug-in for Netscape) in order to view the animation. The lab machines had the wrong version, so the students lost the value-added element of this journal.
• Most respondents did not understand 'bookmarks' and 'thumbnails'.

Apart from these, the overwhelming comment was that accessing and using many of the electronic journals (primarily the commercial ones) would have been difficult, or even impossible, without the handouts and oral guidance given during the sessions.
 

3.1.7  Comparison of electronic and printed journals

3.2  Results: research students and staff

3.2.1  Access

The provision of a list of journals grouped into subject categories was not found very helpful. Complaints ranged from an excessive number of subject headings to the inclusion of irrelevant titles. It was felt that access requirements overall were far from intuitive (this included use of Adobe Acrobat).
 

3.2.2  Format

One summary comment on access and format may be quoted: "I wish that [a specific commercial publisher] would go away and die somewhere - or at least their e-journal Web designers!"
 

3.2.3  Printout

In almost all cases, the time required for downloading and printing material was regarded as excessive. The explanation for this seems to lie in the large size of PDF files -- for most papers , 0.5-1.0 Mbyte -- and congestion on the network. One respondent noted that he allowed for this by making the process a background activity, while he continued with all his other computer activities.

Research students essentially echoed the comments of the staff, but with two particular emphases of their own. One related to back issues. Research students are particularly concerned with in-depth coverage of the literature. Hence they were frustrated by the paucity of back issues, and by the problems of searching across back issues where they did exist. The other query concerned the introduction of non-essential links, often to general Web sites, in the text of an article. A conscientious student who worked systematically through these would, it was noted, soon become irritated by the waste of time. One example quoted was an article about archival materials in the White House. The author had written: 'My day job is at the University of Michigan.' The link related neither to the subject of the article, nor in any specific sense to the author.
 

4  Conclusions

• Low-level problems, especially delays in gaining access and impediments to moving about within and between journals, are a major demotivating factor in the use of electronic journals. Some of these problems are within the control of the publisher: not least, the nature of the interface. Others are not. For example, Academic Press improved server access efficiency appreciably between the pilot and the main studies. However, traffic between the USA and the UK grew, so delays from this cause continued.
• Lengthy on-screen reading, when an article has to be read in depth, remains unpopular. This is not helped by less than optimum decipherability of on-screen text and graphics. Such illegibility is something that publishers should be able to change without excessive difficulty. Efficient and rapid printing capabilities are vital, and will continue to be so for the foreseeable future. Printing out from electronic journals currently leaves much to be desired, especially in terms of delays, though the problems are reduced for those with access to Postscript printers. In principle, this can be helped by appropriate arrangements for local caches of the most-used journal issues, as is now normal practice at Loughborough.
• In terms of access, free journals tend to be more favourably assessed than commercial journals. This seems to be largely due to the fact that they need nothing more than a Web browser. Although it is a good printing device, the Acrobat software used by several commercial publishers does not yet adequately support user-friendly journal usage. The longer history of a number of free electronic journals, along with the dedicated attention they receive from their progenitors, means that, for them, some of the teething problems for first-time readers have already been ironed out. Respondents in this survey assessed the quality of free journals as somewhat lower than that of commercial journals. It seems from the comments that this was partly because free journals looked somewhat different, and had a different mix of contents from the commercial journals. Purely in terms of reader acceptance, there seems to be no reason why well-edited free journals should not compete with commercial journals for the attention of scholars.
• Commercial publishers currently appear to be more interested in pursuing their own learning curve than in assisting their customers. After several years of uncertainty about their involvement in electronic journals, they are now finally taking the plunge - only, it seems, to make the classic mistake of allowing technology, rather than human factors, to drive their efforts. They not only seem to have rather little knowledge of how readers operate in an electronic environment, they also seem not to be building on knowledge long since gained from the existing print-based environment.
• Use of electronic journals is going to require a considerable training exercise, especially if publishers keep changing what they are doing. Subscription agents and other intermediaries, such as BIDS and OCLC, are beginning to provide common interfaces for several publishers' journals. If they succeed this will mitigate some of the user problems identified in this study, but not all. The remaining difficulties facing readers are such that any future scenario must surely envisage the provision of more assistance and training by librarians and information scientists in their various institutions. Furthermore, the need for training will be continuous. To read print materials, a person has to learn to read once. But the continuous change in hardware, software, systems and interfaces means that in the future users will need to relearn frequently how to access the literature.
• One important factor for readers of electronic journals may be the rapidity of change, as differences found between the pilot and main studies of this project suggest. On the one hand, this is leading to the electronic provision of an increasing number of titles, which is evidently a plus factor. On the other, it is leading to continuing change in the interfaces. Though increasing sophistication is to be welcomed, change in an interface is quite likely, in the first instance, to make it less user-friendly. In addition, a reader who had adapted to one interface may be required to relearn what to do. Continuing change is therefore unlikely to be good news for readers who are concerned with the content of journals, rather than their handling. A comment on electronic journals from one of the postgraduate students at Loughborough may be relevant here:
• "They are interesting to play with, but normal people read printed journals."

1. Do not launch an electronic-journal service publicly without testing it with a pilot group of real users first; take note of their experiences in using it to redesign it before public launch. Do not allow yourself to be pushed by the technologists; remember the user.
2. Do not have too many graphics in your Web page design. Although it is possible for users to disable graphics in their Web browser, näive users will not know that, and will suffer low delivery speeds owing to the graphics.
3. Do not have unnecessary extra steps in the chain of Web pages leading from your home page to the actual full-text articles. At busy times each page in the chain may take several seconds to deliver.
4. Once the service is launched, do not make frequent changes to the interface or software. If you must change them, test the new versions again as in (1) above, give plenty of warning on your Web pages that they are going to change on a certain date, and leave an explanatory note about the changes on your home page for some time after making the changes.
5. Make journals available through all the various 'one-stop shop' vendors now being developed. In this way, each university can choose one such vendor as standard, and then be able to present all its electronic journals to its users with a standard interface and a single password. If some publishers boycott the one-stop shops, this advantage is lost and the progress of electronic journals in general will be retarded, to the detriment of all publishers.
6. Think seriously about the format in which full texts are presented. Adobe Acrobat/PDF is the current de facto standard, but it is slow to deliver articles, its interface is not as intuitive as it should be, and page integrity means that the article pages are the wrong shape for a VDU screen. Double-column layouts are particularly troublesome for users. PDF is a good document delivery system, and will probably remain standard during the transitional dual-publishing era, but for real electronic publishing, HTML, SGML, HyTime or XML will ultimately be used.

• Factors which are under the control of the publisher. These include most matters relating to design. Proper use of colour and good legibility of text are two obvious examples. Similarly, most of the navigational difficulties mentioned previously could be alleviated if publishers devoted more attention to them. Some needs, such as better backrun provision, may be costly for an individual publisher to resolve. Cooperative activity (e.g. in the case of backruns, with libraries, or with projects like JSTOR (1998)) might be appropriate.
• Factors which are partly under the control of the publisher. In these, cooperation is likely to be particularly important. For example, the inconvenience of retrieving journals by publisher's name and password might be resolved by cooperation between publishers and subscription agents. The overriding problem in this category, however, is how publishers will handle continually changing software/hardware without requiring their readers continually to learn new methods. In particular, changes should be downwardly compatible. To the extent that this implies a need for standards, it again requires increased cooperation on the part of publishers.
• Factors which are not under the control of the publisher. Network delays provide an obvious example in this category. Increasing traffic and inadequate bandwidth are a general problem in which scholarly publishers have only a limited influence.
• The present survey suggests that free electronic journals are not seen as a totally different type of entity from commercially produced electronic journals. Their lower assessment stems more from their deviations from the traditional scholarly journal pattern. Judging by developments so far, subscriptions to commercially produced electronic journals will be of the same order as to their print equivalents. Putting these two points together suggests that, for specialist topics with a restricted audience, low or zero-subscription electronic journals produced by groups of academics may well prove competitive with commercially published electronic journals, so long as they follow an appropriately scholarly approach.

Acknowledgements

References

Harnad, S. (1994) "Publicly retrieval FTP archives for esoteric science and scholarship: a subversive proposal" In Scholarly Journals at the Crossroads: A Subversive Proposal for Electronic Publishing, edited by Ann Okerson & James O'Donnell, June 1995 (Washington, DC.: Association of Research Libraries) http://cogsci.soton. ac.uk/~harnad/intpub.html

Harter, S. P.,  and Kim, H. (1996) "Electronic journals and scholarly communication: a citation and reference study". American Society for Information Science (ASIS) Midyear Meeting, San Diego, CA, May http ://ezinfo.ucs.indiana.edu/~harter/harter-asis96midtxt.htm

JSTOR (1998) "About JSTOR". http://www.jstor.org 

Kidd, T. (1997) "Are print journals dinosaurs?" Ariadne, issue 12, November, pp 6-7  http://www.ariadne.ac.uk/issue12/main/

Odlyzko, A.M. (1996) "Tragic loss or good riddance? The impending demise of traditional scholarly journals". In Scholarly publishing: the electronic frontier, edited R.P. Peek and G.B. Newby (Cambridge, Mass.: MIT Press) pp. 91-101

Okerson, A.S. and O'Donnell, J., eds (1995) Scholarly Journals at the Crossroads: A Subversive Proposal for Electronic Publishing (An Internet Discussion About Scientific And Scholarly Journals And Their Future) (Washington, D.C.: Association of Research Libraries)

Rowland, F. (1995) "Recent and current electronic journal projects". In Project ELVYN: An Experiment in Electronic Document Delivery, edited by F. Rowland, C. McKnight and J. Meadows, (East Grinstead, UK: Bowker-Saur) pp. 15-36

Shackel, B. (1991) BLEND-9: Overview and Appraisal, British Library Research Paper 82 (London: British Library)

Stewart, L. (1996) "User acceptance of electronic journals: Interviews with chemists at Cornell University". College and Research Libraries, 57(4), pp 339-349

Woodward, H. (1997) Electronic journals in an academic library environment. Serials, 10(1), pp 53-57
 

Appendix 1: Commercial And Free Electronic Journals: User Studies: user questionnaire

Download latest PDF viewer
 

Appendix 2: e-journals accessed by students during pilot study