To evaluate the resource centre, ‘qualitative data’ is needed, as well as the ‘quantitative data’ collected by monitoring.
9.3.1 Collecting qualitative data
Qualitative data, such as users’ opinions, can be gathered in different ways. For example, users could be asked to complete a questionnaire, or they could be interviewed using the questionnaire as a structure for the interviews. Questionnaires could be given to a representative selection of visitors and enquirers, to keep the number of questionnaires down and make them easier to analyse.
Questionnaires sent to enquirers should be accompanied by a copy of the original request and the reply, including details of the materials that were supplied. This is particularly important if the questionnaire is sent some time after the enquiry was made (such as more than six months).
Focus group discussions (small group discussions) could be set up for staff or students of the parent organisation, visitors and more distant users. They could be organised as part of a national or regional meeting attended by users of the resource centre. Users could discuss their opinions of the resource centre and its services, what impact information from the resource centre has had their work, and what subjects and services they require.
Information gathered through questionnaires and focus group discussions can be used both for evaluation and as part of ongoing needs assessment (see Section 1.1).
9.3.2 Using the data
The data collected through monitoring and questionnaires or discussions can be used to answer questions such as:
1. What impact does the resource centre have on users?
Has using the resource centre or the enquiry service helped users to carry out their work or studies? Has information from the resource centre led to any particular action or change in work practices?
This information can be obtained from the monitoring information and comments made by users through questionnaires and discussions.
2. How far does the resource centre meet users needs?
Do users believe that the resource centre is meeting their needs?
It is also useful to consider whether the resource centre was established in response to a demonstrated need. Many resource centres grow from an individual campaign and are based on staff’s view of what is needed, rather than the views of the community that they aim to serve. It is worth checking whether a needs assessment was carried out, and whether the collection and services reflect the needs identified in the assessment.
This information can be obtained from documents about setting up the resource centre, and by asking users how far the resource centre is meeting their needs. Users can also be asked what they expect of the resource centre and what services are available elsewhere. It is useful to note how many users visit the resource centre more than once, as repeat visits help to show how useful the resource centre is. Similarly, it is useful to note how many enquirers make more that one enquiry. A second request on a different topic suggests that the user was satisfied with the response to their first enquiry. A sample of enquirers could be contacted to obtain more detailed feedback about how useful they find the services.
3. What do users save by using the resource centre?
How much time, money and effort would users have to spend to obtain information, if the resource centre did not exist? For example, they might have to travel further, or they might have to visit several different resource centres. If they could not obtain the information they need from elsewhere, this would mean that the resource centre was providing a unique service. Even if the resource centre was not very efficient, being unique could be a key strength.
This information can be obtained through focus group discussions and questionnaires, and through knowledge of what other information services exist, what subject areas they cover and what services they provide (see Section 1.4.2).
4. Are enough people being reached?
How clearly defined is the resource centre’s target audience, and are enough of them being reached? For everyone who comes to the resource centre, there are sure to be many others who do not know that they need information or do not know where to find it. Is the resource centre publicised in places where potential users go, such as training institutions, religious centres or community groups? Is it possible to identify sections of the community who need the services but are not using them?
To answer these questions properly, it is necessary to know the size of the target audience, such as the number of health workers or trainers in the area covered by the resource centre. It is also important to look at efforts to promote the resource centre to members of the target audience who were not previously using it, to see how well the promotional activities have worked.
This information can be obtained from the records of visitors using the resource centre, enquiry services, and number of users before and after promotional activities. It can also be obtained from the opinions and ideas of users for whom promotional activities are targeted, including those that have not used the resource centre.
5. Does the resource centre meet the needs of funding organisations?
This information can be obtained by checking funding organisations’ goals and criteria for supporting the resource centre, and comparing these with the results of monitoring and evaluation.
6. How good are the materials?
How often are the materials used? Are there many materials that are rarely used? Are some subject areas more popular because they are more up-to-date? Are both resource centre staff and users clear about the subject areas covered by the resource centre? Have all the subjects or material formats requested in the needs assessment been regularly used? If they are not, does the collection policy need to be changed, do materials need updating, or is more promotion required?
This information can be obtained from the monitoring information and user questionnaires and discussion.
7. What does it cost users to obtain information?
How much time do users spend learning to use systems such as the classification scheme, catalogue or database? How easy is it for them to use these once they have learned how? How do the advisory services and information skills training provided by staff help users to find information in the resource centre?
This information can be obtained by providing advisory services (see Section 7.4), and through user discussions and questionnaires.
8. How skilled are staff?
Can staff provide information as well as process materials? Are staff friendly and helpful? Are they involved in planning new developments and knowledgeable about what is going on? Do they need more training?
This information can be obtained from user discussions and questionnaires, and discussions with the resource centre advisory committee and resource centre staff about staff’s capabilities and training needs.
9. How well is the resource centre networking?
Are enquiries received from other organisations or individuals, such as public libraries, research organisations, community groups or individual experts? Are enquiries from users referred to other resource centres? Have efforts been made to eliminate duplication by sharing responsibilities, such as collection, processing and storage, with other groups? Is there a file of people or organisations who can provide information and share their expertise? Staff should not simply add names of useful contacts as they hear about them, but they should go out and ask people if they will collaborate with the resource centre.
This information can be obtained from staff records, minutes of resource centre advisory committee meetings and discussions with staff.
10. How useful are the resource centre’s publications?
Are publications such as current awareness bulletins, information packs, newsletters, articles, or resource lists produced? Are they accurate, legible, appropriate to users and efficiently distributed? Do users find them useful and timely? Are they a worthwhile activity in terms of the time and resources available to the resource centre?
This information can be obtained from staff records for preparation and distribution, users’ discussions and questionnaires, and staff comments about the time and effort taken to prepare publications.
11. Are systems for selection, indexing, cataloguing and retrieving information cost-effective?
How much does it cost to process each material in the collection (in terms of both staff time and materials)? How long does it take to process materials (for example, accessioning, cataloguing and classifying, entering records onto the database, and quality controlling)? Are these systems worth the staff time involved, because they speed up the retrieval process, or do they take more time than can be spared?
This information can be obtained by monitoring the time taken to process materials, looking at the records of information searches that have been carried out, how long searches have taken, and how far they have met users’ needs. Opinions expressed through staff and user discussions and questionnaires are also important.
12. How can the resource centre increase its collection and improve its services in the most cost-effective way?
This information can be obtained by listing ways in which the resource centre can increase its collection and improve its services, and then identifying which of these are least expensive in terms of money and staff time. Comments from users can be obtained from the suggestions/comments box, monitoring forms, and questionnaires and discussions.
13. What improvements are the most cost-effective and beneficial?
This can be decided by comparing improvements that are cost-effective with what users most need, and reaching a balance. It makes no sense to offer services that the resource centre cannot afford, but if there is a choice of services that can be offered, the needs of the users should always come first.
9.3.3 Using the results of an evaluation
The purpose of carrying out an evaluation is to help improve the resource centre and its services. The process of evaluation demonstrates what is being done well and should be continued, as well as what needs to change and what additional activities could be undertaken. Poor results are as important as good ones, as they can point to ways to improve a service.
Evaluation results should be used to identify new objectives, and develop new action plans (see Sections 1.2 and 1.3). They may result in changes to how the resource centre is run, what it collects and what services it provides. They may also identify staff training needs, to enable staff to carry out their work efficiently and provide the services required.
The planning cycle
