Issues surrounding the embedding of a web-based course
Julia Phelps1 and Ross Reynolds
Department of Meteorology, University of Reading, Earley Gate, Reading, RG6 6BB, UK
[email: J.Phelps@unl.ac.uk, R.Reynolds@reading.ac.uk]
Abstract: EuroMET has been created to address the education and training needs of professional meteorologists and students in tertiary education throughout Europe and more widely. Two meteorology libraries, each modular in format, have been developed for the WWW. The libraries have already undergone a formative evaluation and have recently been evaluated during their use on teaching/training programmes in all our partner institutions. This summative evaluation aimed to assess the effectiveness of the module libraries as a learning resource and to identify how they can be successfully integrated into a teaching programme. This paper describes the results obtained during this evaluation and presents guidelines for the integration of online teaching material.
Introduction
The education and training of meteorologists varies widely across Europe. Basic meteorology is taught in many universities most often as part of other courses such as geography or environmental science. Comprehensive courses in meteorology are generally taught in specialist departments at both undergraduate and postgraduate level. Due to the relatively small number of universities teaching meteorology at sufficiently advanced levels (approximately 5% of all universities Europe-wide), National Meteorological Services (NMS) usually recruit employees at university degree level and then offer their own training as is necessary or possible. The European NMS' education and training courses which exist at present differ greatly according to the size and organisation of each service. Some NMS run their own training courses, such as in France and the UK, and also offer courses to other countries unable to run their own, e.g. the Netherlands. Other NMS such as in Belgium offer very little formal training. A further important part of NMS training programmes is extension training which is required to keep all employees up to date with the latest developments in meteorological science and technology. At present no distance or open learning courses in meteorology exist within Europe.
Thus it can be seen that within the European meteorological community computer-based learning resources could be extremely useful. They could offer an alternative to those NMS with little or no training programmes of their own and enable NMS’ training centres and universities in this relatively small field to share their educational material. In addition, meteorology is a subject which could benefit from being taught in a computer-aided learning (CAL) environment. Much of the science concerns descriptions of the atmosphere in 3-dimensions with the added complication of time variation. These are usually presented as complex mathematical equations: a CAL package offers the opportunity to visualise these descriptions as animations or models and simulations which can be explored and experimented with.
The European Meteorological Education and Training (EuroMET) project therefore aimed to address these education and training requirements of professional meteorologists and students in tertiary education throughout Europe by establishing a multimedia, networked-based, open and distance learning service. EuroMET arose due to both the emerging interest in CAL within the meteorological community and the decision of the European Commission to foster telematic applications within the 4th programme for research and development.
The EuroMET Consortium
The EuroMET project is an EC-funded initiative with matched funding from the participating National Meteorological Services. The EuroMET consortium was comprised of 22 meteorological partners, i.e. universities and NMS, from 14 European countries and the University of Quebec in Canada. The consortium was divided into two groups for the first 18 months of the project, the developers who were responsible for delivering the module libraries, and the evaluators who were responsible for assessing the modules. The developers were led by Météo France, where the project co-ordinator was also based, and the University of Edinburgh. The evaluation was lead by EUMETSAT (the European Organisation for the Exploitation of Meteorological Satellites) and the University of Reading. The 30 month project commenced in June 1996 and was split into three key stages: the prototyping of minimal content modules, the development and evaluation of full versions of the modules, and finally, the implementation by all partners of the module libraries into their teaching/training programmes and the evaluation of this use.
The Module Libraries
When producing educational material for a group as large as the EuroMET consortium several factors have to be taken into consideration. The course has, for example, to be accessible to everyone regardless of their location and what machine is used to access it, and it should be easy for everyone to have access to the latest version. For these reasons and because of the potential to include communication facilities, the course is web-based.
The subject matter for the libraries was selected in response to a European-wide needs analysis of the National Meteorological Services. The two subjects of Numerical Weather Prediction and Satellite Meteorology were chosen. Both of these libraries are comprised of nine chapters covering different sub-topics, with several modules in each chapter, thus there are around 70 modules in each library. Modules take an average of 30-40 minutes to complete. This modular approach allows teachers at different locations to choose those which are directly relevant to the course they teach.
A common interface was chosen so that each module would look and feel the same. To aid the development process, the style and content of the modules were separated as much as possible. Thus, the developers could compose a module's contents in a simple 'editor' environment, a EuroMET-developed macro would then add the graphical interface and navigation tools. This development process is discussed in much more detail in Duncan et al. (1998). The interface was designed to be as simple as possible so that users could learn to use it as they went along. There is, however, an online help facility which explains all the navigation buttons, symbols and icons used throughout the course. The interface has also been designed to be language independent, an important consideration for a multi-lingual project. Each module is constructed in the same way, opening with a motivation page to introduce the topic and engage the user's interest. The next page lists the learning objectives of the module, any pre-requisite modules the user should study first, any subsequent modules which follow on, and a list of the module's contents. The following pages then contain the teaching material, whilst the final page contains a summary of the material which draws out the salient points of the topic. Users navigate through the modules by clicking on arrows on a button bar at the bottom of the screen to progress page by page, either forwards or backwards. They can also click on the 'module structure' button, which enables the user to go directly to any page of their choice. A 'course structure' button similarly allows the user to select any module.
In support of this learning environment is a glossary and a keyword index, enabling the user to easily find those modules which contain material covering the term selected from the index. Facilities for one-to-one and one-to-many communication are also included, such as email to subject experts, discussion groups and Internet relay chat areas. Due to the large number and variety of users within EuroMET, Draper et al's (1996) statement that "designers of CAL frequently do not design in detail the teaching within which its use will embedded, that teachers make many local adaptations just as they do for all other teaching, and that even if designers did prescribe their use many teachers would still modify that use just as they do that of text-books", is particularly true as the EuroMET developers were not designing the teaching within which it would be used. Rather than ignore this fact, EuroMET is designed to be as flexible as possible with tools to create adapted packages and extra material available from the EuroMET WWW site with full documentation. This tool-kit allows teachers to create their own online course from the EuroMET modules completely tailored for their needs.
The EuroMET module libraries were one of 10 computer aided learning products which were awarded European Academic Software Awards (EASA) in 1998. This biennial award is organised by EKMA, the European Knowledge Media Association, and is a competition for innovative software and IT use in education, designed to stimulate the understanding, development and use of knowledge media within Higher Education across Europe. As well as the award, EuroMET received a special commendation, “The Juror Committee unanimously agreed to make a special commendation for the EuroMET Project. The Committee considered the European dimension of the EuroMET Project was of exceptional quality in supporting the development and use of materials across Europe and in supporting European co-operation.”
Evaluation Methodology
The evaluation of the EuroMET module libraries has included both a formative and a summative evaluation. The formative evaluation was embedded into the development phase of the modules, while the summative evaluation was conducted after all the development work was completed. Both phases of the evaluation have been discussed fully in other papers (Phelps and Reynolds, 1999 and Phelps and Reynolds, 1998) so only a brief outline will be presented here.
Modules were released onto the web in three batches at regular points during their development for evaluation. In this way, the evaluation team could assess the first set of modules while the developers worked on the production of the second set. Enough time was incorporated before the second release date for the developers to act on the evaluators' report, so that both sets of modules could be modified to include the recommendations made. This was then repeated for the second and third module releases. In this way the modifications could be tested by the evaluation team, which resulted in an iterative approach to the development work.
The aim of the formative evaluation was to assess the libraries in terms of their ease of use, pedagogy (including ensuring its scientific integrity), and possible utility in replacing conventional teaching. The EuroMET evaluation strategy was planned and executed by all the evaluation partners with occasional advice from staff at the Institute of Educational Technology, Open University, UK and from the EC-sponsored INUSE (Information Engineering Usability Support Centres) project. The modules were evaluated at each institute by a number of users ranging from novices, i.e. students or new employees, to subject matter experts, such as university lecturers and experienced meteorologists. Around 100 to 200 users evaluated the modules in each phase. The three major evaluation issues mentioned above were addressed in two questionnaires, devised with the help of staff from the Open University. The questionnaires were completed after using each module to ensure feedback on individual module elements as well as on course issues. One questionnaire addressed the broader issues of functionality and ease of use, while the other was directed towards the actual content of the module, such as the scientific material and the applicability of any multi-media elements. The questionnaires consisted of yes/no and Likert scale questions with comment areas in which subjects were encouraged to expand on any of the answers they had given as well as to make suggestions to the developers.
The formative evaluation was extremely useful in producing modules which were suitable for their target users. The fact that the evaluation was embedded into the development work meant that it was relatively easy for the developers to modify the modules according to the recommendations of the evaluators and in turn for these modifications to be tested during the next evaluation phase. There was a marked improvement in the quality of the modules from the first evaluation phase to the third. The evaluation also resulted in a more homogenous feel to the modules. Full results of the formative evaluation are presented in Phelps and Reynolds (1999).
The EuroMET summative evaluation aimed to test the effectiveness of the finished web-based libraries and examine the implementation strategies used to embed the modules into teaching and training programmes. This evaluation period ran for 12 months, finishing at the end of October 1998. During this time all 22 consortium partners as well as interested non-consortium members used the modules on courses at their institutions. With this number of participating institutions integrating the module libraries into their training and lecture courses a large number of students would be exposed to the EuroMET resources. Indeed one of the project commitments was that at least 1000 students should use the modules. Further, as each institution obviously offers different types of training, (i.e. academic and work-based) and courses both in subject matter, ('pure' meteorology, environmental science and so on) and level, there was huge scope for the modules to be used as a learning tool. Uses of the module libraries ranged from simple classroom demonstrations to illustrate a particular concept to full lecture replacement, or use of the modules as an electronic reference through to complete independent study programmes.
It was therefore necessary to employ a range of evaluation methods to investigate the different uses and the effectiveness of the modules. An evaluation web-site was developed not only so that the project partners could access the evaluation tools, but also to enable the lead evaluation team at Reading University to monitor the evaluation activities of the consortium partners. The evaluation site comprised details on possible module implementation strategies, details of different evaluation strategies, information on all the courses being run within the consortium and evaluation results from these courses as they came in. In this way the evaluation site served as a useful resource for the evaluators and for course leaders deciding on how to incorporate the EuroMET modules into their students’ studies. Owing to the number of partners in the consortium, it was decided to collect evaluation data solely from consortium members, even though a number of non-consortium institutions had asked to be able to use the modules with their students. The evaluation methods used are discussed in more detail in Phelps and Reynolds (1998).
The evaluation methodology described above was used to enable EuroMET to establish exactly how students used the modules, which implementation methods were most successful in terms of their study, which methods were less successful and the reasons why this might be so. Although it is extremely difficult to isolate the effects of CAL from other learning resources, (Draper et al, 1996 and Gunn, 1996), it was hoped that by collecting a range of data from a range of learning environments it would be possible to ascertain whether EuroMET provides a useful and effective learning tool and also enable us to draw up some guidelines on how best to integrate it into a course of study in both university and workplace settings.
During the period April – October 1998, 17 institutions registered 54 courses on the evaluation website on which teachers planned to the EuroMET modules to teach some 1318 students. The web-tracking implemented in July 1998 revealed that between 1/7/98 – 30/10/98 the EuroMET module libraries had been accessed from 43 different countries, with an average number of pages accesses of 325 per day, including weekends (Black et al. 1998). In addition, the tracking showed that the libraries had been accessed from a total of 1248 distinct computers in this period giving further indication of the number of users accessing the modules. Questionnaire results were obtained between April and October 1998 for 32 of these courses, which accounted for 471 students.
The teachers' use of the EuroMET libraries fell into three broad categories, namely the replacement of lectures by modules, using the modules as an extra resource alongside lectures and using the modules as the primary resource while students followed a course of independent study. Of the 32 courses for which questionnaire results were obtained, on nine courses the modules were used as a lecture replacement, on 14 courses they were used as additional material and on the remaining nine courses the modules were used for independent study. On 19 of these courses the use of the modules was compulsory and/or during time-tabled sessions and on the remaining 13 use of EuroMET was optional and/or students were expected to work through the modules in their free time.
The teachers who returned questionnaires were very positive about the EuroMET modules, regardless of the implementation strategy they had adopted. They found that the modules were a useful and enjoyable teaching tool. Teachers particularly found those modules with a lot of interactive content to be the most effective as they could be used to great effect to enhance their lecture material. This view was reflected by 91% of the teachers who thought that the modules could sometimes be more effective than their usual teaching methods. The overwhelming majority of teachers did not have any misgivings about using the modules to teach their students (only 4% agreed with the statement "I do not have faith in the modules being able to teach my students well") and all the teachers who responded said that they would use the EuroMET modules to teach their students again. However, four teachers expressed concerns about the efficacy of the modules because they found it difficult to track their students’ progress. One of these teachers overcame this by giving his students exercises to complete at home which he then marked to assess their progress. On another course, although the teacher felt that it was difficult to track his students’ progress, two exams were set during the course, so that while no direct feedback on the modules’ effectiveness was possible, general feedback on the students’ understanding was.
Most of the teachers returning questionnaires found it easy to navigate around the EuroMET website and module libraries and to integrate the modules into their teaching/training programmes. However a significant number did not think that it was easy to find which, if any, of the 139 modules were appropriate for their course. This difficulty in finding modules often made their lesson preparation time consuming, one teacher suggested improving the existing key word index to facilitate this search. Those teachers who expressed difficulties in integrating the modules into their courses found that these problems were mainly due to lack of experience in teaching with this medium. For example, one teacher severely underestimated the time it would take his students to work through the modules in his class,
“Two hours were needed to use the two modules selected during a group session. I was also obliged to come back afterwards for a further hour to explain some concepts in the modules….. so three hours were consumed in place of one and a half hours for a lecture without the EuroMET modules”.
Another important aspect of CAL is the impact, if any, on student-staff contact time and staff time. Can the use of online tutorials reduce the workload of teachers and thus save them time? For example, can a CAL package be used to replace a traditional class and so cut the time teachers need to spend with their students? Can staff save time on lesson preparation by adopting the use of EuroMET into their courses? Figure 1 shows that there appears to be no straightforward answer to these questions, not surprisingly as many of these issues depend very much on how a course has been taught in previous years and the method of module integration each lecturer choses.
Figure 1: Previous computing experience of the three main types of EuroMET module user
However, it would appear from Figure 1 that in general using the modules to teach with does not decrease staff contact time with students. Those cases where teachers did report a decrease (20%) were all examples where the modules were used to replace lectures and students were given no provision in their timetables for working with the modules and so were expected to use them in their own time only. Similarly, integrating the use of the EuroMET modules did not usually result in any reduction in the time spent on lesson preparation. This result is not too surprising given the comments that teachers have made about difficulties that they had in finding appropriate material in the module libraries and the lack of experience that they felt they had in relation to teaching with this type of resource. Therefore the question of whether the introduction of EuroMET into their courses resulted in giving them more free time provoked a mixed response with more lecturers disagreeing (56%) with this statement than agreeing (35%). However as has been pointed out by Davies and Crowther (1995), efficiency defined solely from the educator’s perspective fails to recognise the students’ involvement in the learning process and that efficient teaching is not necessarily efficient learning.
Another issue which raised concern was how the students were learning with modules and how that knowledge was retained. For example one lecturer observed that while most of his students worked through the modules diligently “one remaining problem is the fact that the students took no personal notes during the learning process” and that this therefore meant that the students had “nothing personal to take away at the end”. This suggests that teachers should consider carefully what it is they want students to get out of this type of material and to guide them to take notes and so on if necessary.
Results from the users' questionnaire revealed that access to a computer and to the WWW was not a problem for the students. Similarly most students (91%) agreed that it did not take long to become familiar with the EuroMET graphical interface and that there was sufficient online help provided. As stated in the preceding section, the modules were implemented into teaching and training courses in three distinct ways namely, for independent study, as additional material to accompany a lecture course, and to replace a lecture or series of lectures. Again, in order to examine the issues surrounding the embedding of the EuroMET modules these three groups were analysed separately. The users in all three groups had similar previous computing experience (see Figure 2) and all gave EuroMET a comparable overall rating of “good”. It is also interesting to note that all the users received similar levels of supervision, both academic and technical, whilst using the modules.
Previous computing experience of the EuroMET students according to learning strategy adopted.
The main differences noted between the three user types was in their attitudes to the perceived learning benefits of the modules and the types of support and/or help they would have liked, see Figures 3 and 4. In general the independent learners were more positive about their learning experience than the other two groups and were more inclined to agree that the modules helped them to learn and that they improved their skills/knowledge. These learners were all experienced forecasters aged between under 25 to over 45 years old. They typically were based in the smaller NMS and so had access to limited training resources, which could account for their more enthusiastic response to the online modules.
These students often said that they sometimes found it difficult to motivate themselves when learning on their own and that they would have liked to have been able to meet with other students or to have a workshop day. Indeed, all the students on two courses where there was no personal contact with a tutor or fellow students said that they found their experience with EuroMET 'lonely'. It is therefore not surprising that these users would have most liked to have had classroom sessions or exercises after the modules, a tutor made available to them or an introductory lecture before using the modules. However, even though only 10% of these students said that they always had technical supervision whilst studying, none of them said that they would have liked more technical support.
Graphs comparing learners' attitudes to the perceived learning benefits of the EuroMET modules according to study mode
Approximately a third of the students who used the modules as additional material to support their lecture course were based in NMS and two-thirds in universities. The age profile was slightly younger than for the independent learners with all users being under 45 years of age. On the whole this group thought that the modules were worth the time that they had spent on them and that they had helped them to visualise the concepts and material addressed. Although generally positive about their learning experience with the modules, they were slightly less likely to think that using them had improved their skills or knowledge. There are two possible reasons why this could be so. Firstly, these users were using the modules in addition to attending lectures which often covered the same material. Therefore the modules should have either served as an introduction to, or consolidation of, the material discussed by their teacher and hence were not considered as the primary source of their learning by the students.
However, there is evidence to suggest that this was not the only reason for the smaller perceived gain in learning benefits. Along with more online tests (which all user groups expressed a strong preference for) the most common forms of support which these students would have liked were a tutor and a class or exercises on the material covered by the module after studying it. Considering that these students were using the modules in addition to a traditional lecture course this result is somewhat surprising, however the picture becomes a little clearer when the teachers’ comments are taken into consideration. In the preceding section teachers’ lack of experience in making effective use of this type of resource was discussed, indeed some course teachers commented that the next time they used the modules they wanted to integrate them more fully into their courses. If the EuroMET modules were not very well embedded into the traditional part of the course so that the two components did not complement each other well it is understandable that the students did not feel any benefit from using them and that they required more support from their teachers.
There is also evidence that students on two courses had difficulties learning due to the quality of the individual modules which they studied. Due to time constraints some of the modules had yet to undergo re-development work which had been recommended during the formative evaluation. These modules did not contain any factual errors but were rather poorly written so that students often did not feel confident in their understanding of the material after studying them. Indeed in answer to the question “Would you recommend EuroMET to a friend or colleague?” of the 13% who said that they would not recommend them, the most common reason given was the quality of the individual modules they had studied and in particular the language used in them. For example, one student commented that the “language was complicated” and another said that “there are quite a lot of writing mistakes in the modules, they should be checked carefully”.
The last group of students, those who used the modules as a replacement to lectures, were split approximately evenly between universities and NMS and were generally younger than the other two groups with all the users in the under 25 and 25 – 35 year old groups. These students would have liked more online tests included in the modules and a significant number would also have liked a classroom session after they had used the modules in which to discuss what they had learnt. These results suggest that although some improvement could be made in the integration of the modules, when replacing lectures teachers took more care to ensure that the material in the modules and their classes complemented each other effectively. This is confirmed by the fact that most of these students thought that the modules were worth the time that they spent on them, and were more inclined to agree that the modules helped them to visualise the concepts which they address.
Discussion of Evaluation Results
The evaluation strategy enabled us to learn a great deal about implementing online modules into undergraduate teaching courses and professional meteorologists’ training.
It was found that access to a computer and, more importantly, to the WWW, was not a problem for EuroMET users. This result was not unexpected, a large amount of work carried out in NMS is computer-intensive so employees often have their own machine, while in universities students in science departments also generally have access to reasonable numbers of computers and the Internet. Further, students found the modules easy to use and did not find that they needed a great deal of technical support to assist them in their learning. The EuroMET environment was certainly designed with usability in mind, as is borne out in the formative evaluation (Phelps and Reynolds, 1999), but the students' reported ease of use is undoubtedly aided by the fact that meteorologists are generally fairly computer-literate to start with (Figure 2).
Overall, students from all three groups agreed that the modules were a useful study aid, easy to use, provided good complementary material and moreover that they were good fun. However, a small number of students reported that the modules which they had used were difficult to read and that they had had problems downloading and running certain modules, although only three students thought that there were “too many technical difficulties”. These negative results may largely be due to the problems of some modules still requiring re-engineering work during this demonstration phase.
Both students and teachers were particularly enthusiastic about the interactive parts of the modules and often said that these were the most useful feature of the modules. This comment is exemplified by the popularity of the module "Today's Satellite Image" in which students can submit their own interpretation of the day's satellite image and compare it with the analysis of an expert. Comments such as:
“it’s good to have something to show what happens, why the equations are important in the actual world, some idea of how it works”
"the multimedia and interactive simulations help you understand and visualise",
"animation helps you to visualise the different types of subject, this cannot be achieved in an ordinary lecture course but it works fine in the modules"
"you get pieces of information and you can test the theory in examples"
were common. Observation of and interviews with students provided further evidence that students took greater advantage of those modules with which they could interact and which provided feedback on their understanding. These findings are in agreement with those of other researchers, see for example Watkins et al. (1995) who reported that “specific comments made by 3% of the sample highlighted interactivity as being the main advantage of computer-based education (reinforced during observations)”.
While most teachers said that they found it fairly easy to integrate the modules into their courses, they also found that to do so successfully required them to spend some time looking through the modules' content. Those teachers who said that they found it difficult to integrate the modules generally also said that the main reason why this was so was because they had not spent enough time familiarising themselves with the modules or their contents. For example one teacher said, "before the course I need a serious study of the content of EuroMET's modules", while another commented that he should have done "more pre-lecture preparation of EuroMET material". Students also found the modules to be more effective when they were integrated properly into their courses, for example, students reported studying a module on the same topic as a lecture they had just had was "very useful as the module helped to put the material into context". Further, they found that using these modules in conjunction with other modules which covered new material helped them to build up a more complete picture of the topic.
As well as modules which complemented their face-to-face learning sessions, both university and NMS students valued being given time-tabled slots in which to work through modules rather than having to find their own time in which to study. One university student commented that if instead they had simply been told where the computers for their use were and to study the modules in their own time that:
“It’s probably quite bad, but I don’t think that would have worked as well, mainly because of our laziness”
and that even knowing that they would be examined on material in the modules probably would not be enough motivation:
“there may be a tendency to leave it too long, which would be a bad thing. Having it time-tabled gave you more of a motivation for set goals”.
In the case of the NMS forecasters this is particularly important as it means they are given time when they can concentrate on learning without any distraction, as one forecaster said,
"The modules worked out rather well, I had a place and time to do just purely this job, otherwise, for instance together on duty - NO!".
For students on the other hand, there was evidence to suggest that if their module use was not compulsory or they were not given a specific time-tabled slot they do not use them, for example, one lecturer commented that:
"It seems that students do not use the modules much if they are optional. Of seven students, two or three used about 6-10 modules, two or three used less than 6 and one or two did not use the modules at all".
It has been suggested that students tend to judge learning materials in relation to their usefulness in achieving what the students perceive to be the aims of the course (Joyes 1998). Therefore, if the modules are seen to be unnecessary in terms of passing an end of year of exam then students will not see the need to use them. It is therefore very important that all teachers consider very carefully which materials are appropriate for their students and how best to fit in use on online resources with other teaching methods.
The web tracking has revealed that the EuroMET module libraries have been widely used throughout Europe and beyond, including in North and South America and Australia. EuroMET can also be said to have improved the education and training of meteorologists as it has enabled individual topics and whole courses to have been taught to students and professional meteorologists which otherwise they could not have studied. For example in the Finnish Meteorological Institute one trainer commented “we couldn’t have organised such a massive course in NWP without the modules, we have no lecturers and we couldn’t take so many students simultaneously from their operational work to listen to a lecture”.
The demonstration phase has shown that this type of resource provides an effective learning tool in this subject area, with animated satellite images, case studies of meteorological events, models and simulations of particular atmospheric processes allowing students to visualise concepts in a way which is not possible in any lecture, seminar or tutorial. Both teachers and students stressed that the real advantage the module libraries had over other learning resources was the interactive and animation possibilities they offered. The EuroMET module libraries are thus a very useful teaching resource which can sometimes be more effective than other teaching methods and all teachers who returned questionnaires said that they would use the modules again. Students did report difficulties understanding the material in some of the modules but these problems were restricted only to those modules which still required re-engineering work at the time of this study and underline the importance of the work done during the formative evaluation.
This study has highlighted the importance of using a well-developed integration strategy to ensure not only that the students actually use the modules, but also that they find them useful and effective. There is some evidence which suggests that students do not use the modules if it is left entirely up to them whether they do so or not. Therefore a “bolt on” strategy which simply uses the modules as an additional resource without modifying any other aspect of the teaching does not work well. The best implementation strategies would seem to be those where the use of the modules is compulsory or time-tabled computer sessions are arranged and where the modules and face-to-face sessions build on each other. Additionally those forecasters who were able to use the modules in time slots set aside specifically for this purpose were very appreciative of this fact and stressed that it would have been extremely difficult to study otherwise.
The modules can also be used successfully as a stand-alone resource. Though those students who used the modules for independent study often commented that they found it lonely and sometimes hard to motivate themselves and would ideally have liked to have been able to meet other students or have a workshop day. This is something which should be taken into consideration by someone planning to run such a course.
The case studies and some comments from the questionnaires have highlighted that how students study when they use this type of resource should perhaps be examined further. In particular how students integrate online learning into their general study should be examined, as well as teachers’ expectations of it. As has already been mentioned, teachers showed concern about not being able to track students’ progress and one lecturer was worried that his students did not seem to take any notes while using the material and so had “nothing … to take away” with them. But is note-taking a real measure of the learning which may or may not be taking place? We need to ask what type of learning is occurring and can lecturers do more to ensure that students get the most out of these resources. In an interview one student said:
"I think it is a good tool for visualising problems - if used in conjunction with other learning methods. I personally would never be able to learn equations and solutions from looking at them on the screen and guessing the correct answer, I need to get a piece of paper and work it out for myself".
Why didn't this student get out a piece of paper and work out the equations while they were at the computer, or even look in a book for more information? By not doing so was he really engaging with the subject matter? This study shows therefore that while good practice in the embedding of online material is emerging there are still some questions which need resolving before all students find it a valuable experience.
Acknowledgements
The authors would like to acknowledge the Telematics Division of the European Union and the participating Weather Services who have funded this project. Many thanks also go to all the EuroMET consortium members for all their hard work in producing and assessing the course.
A demonstration of the EuroMET modules can be found at http://www.euromet.met.ed.ac.uk/
Black, R. et al., 1998, Accurately tracking student usage of EuroMET. ALT-C
Davies, M. L. and Crowther, D. E. A., 1995, The benefits of using multimedia in Higher Education: myths and realities. Active Learning, 3.
Draper S. W., Brown M. I., Henderson F. P., and McAteer E., Integrative evaluation: an emerging role for classroom studies of CAL. Computers Educ., 26, 17-32 (1996).
Duncan, C. et al., 1998, Creating web-based courses: separating style and navigation from content. ALT-C
Gunn C. and Maxwell L., CAL in human anatomy. J. Comp. Assist. Learn., 12, 205-215 (1996).
Joyes, G. et al., 1998, Learning outcomes and the role of evaluation in embedding COMPACT. Active Learning, 8, 1-4.
Phelps, J. and Reynolds, R., 1998, Summative evaluation of a web-based course in meteorology. In Innovation in the Evaluation of Learning Technology ed. M. Oliver, University of North London, London.
Phelps, J. and Reynolds, R., 1999, Formative evaluation of a web-based course in meteorology. Computers and Education, 32, 181-193.
Watkins J. et al. (1995) Evaluation of a physics multimedia resource. Computers Educ, 24, 83-88.