Clark stated that, "…media are still advocated for their ability to increase learning when research clearly indicates that such
benefits are not forthcoming. Of course such conclusions are disseminated slowly and must compete with advertising budgets of
the multi-million dollar industry which has a vested interest in selling these machines." (1983)

Although this statement could be considered cynical in its emphasis, the question of whether the benefits of technology in
education are forthcoming are still arguable, almost 15 years after this statement was made. This could be taken as a reflection
on the lack of valid research into this issue. However, the emphasis on computer based technology in education has not abated,
and, in fact, continues to increase. This is particularly so when considering distance education. In 1994 $2.4 billion was
expended on educational technology in kindergarten through 12th grade and $6.0 billion in higher education. Higher education
alone has spent some $20 billion over the past 15 years (Katz et al, 1995).

Jones and Paolucci (1996) estimate that less than 5% of published research is sufficiently empirical, quantitative and valid to
support conclusions with respect to the effectiveness of technology in educational learning outcomes. They argue that the
influence of technology, while substantial, is largely unfounded and serious consequences may result if the sustained acceptance
of technology in educational delivery continues without considering the appropriate application. They questioned the untested
educational quality resulting from relatively unproven paradigms involving technology and the questionable cost benefit
associated with this continuance. Their conclusion was a call for further research concentrating on the application of
appropriate technologies to the learning outcomes of the subject matter to which technology is applied.

This paper continues the theme of this argument. While substantial research exists which identifies components of various
teaching/learning models, few suggestions have been made to identify a framework which would help to guide a unified research
agenda contributing to the formulation of a technology/content matrix. This matrix, by its nature would allow the identification of
relevant technologies to be applied to appropriate content.We identify a matrix of taxonomies, which would provide a
framework for researchers to identify and target of specific work and ultimately contribute to a more complete solution.

Evaluation Framework

What is the most appropriate technology to use? In what educational context is a particular technology most effective? How
effective is the chosen technology? The answers to these questions are critical in establishing the value added to the student's
learning and achievement with the use of educational technology. However, with too many evaluation research studies, these
questions are rarely addressed. And, if they are, the results are often unclear, due to the lack of a clear evaluation framework.

In order to address the above questions, we believe that the use of a system-oriented approach to educational technology
evaluation is critical. Thus, our evaluation framework is based on the use of widely accepted and researched Instructional
Systems Design (ISD) models. Although there are many ISD models: Gagne, Briggs, and Wager (1992); Smith and Ragan
(1993); Kemp, Morrison and Ross (1994); R2D2 (1995); Reiser and Dick (1996); Dick and Carey (1996); Seels and
Glasgow (1997), they all have life-cycle phases that include: analysis, design, development, implementation, and evaluation
(Seels and Glasgow, 1998). Although some of these provide guidelines for selecting media and delivery systems (Kemp,
Morrison, and Ross, 1994), they do not adequately address the technology dimensions and how these may relate to learning
outcomes.

In response to this shortcoming, this study will focus on the dimensions of the delivery system and technology. It will present a
general framework that can hopefully guide researchers in establishing a clearer relationship among instructional objectives, the
choice of technology-based delivery system, and learning outcomes.

Instructional Systems Model

As mentioned above, to evaluate the effectiveness of educational technology, it is critical that a systematic approach be used. In
its most general form, an instructional system can be viewed to consist of three major components: Instructional Objectives
(Input), Delivery System (Process), and Learning Outcomes (Output).

Instructional Objectives

Learning is achieved when a permanent change in thinking, attitude, or behavior is experienced. Therefore, the objective of any
instructional system should be to facilitate this process. It should be noted, however, that learning is an internal process which
can only be done by the student. For many formal learning situations, an instructional system does not happen by serendipity. It
requires significant planning and a sophisticated decision-making process. This process begins by clearly identifying a set of
instructional objectives and goals. All instructional objectives can and should be based on one or more of the following factors:
learning domain, learner profile, task characteristics, and grouping. These dimensions are described below.

Learning Domain. The instructional objectives should correspond to one or more learning domain(s). There are three basic
domains of learning: cognitive, affective, and psychomotor. Taxonomies have been widely used to define learning within each of
these domains. The most popular ones by Bloom (1956) for the cognitive domain, Krathwohl (1964) for the affective domain,
and Harrow (1972) for the psychomotor domain. In practice it is very difficult to separate these, although it is possible to clearly
emphasize one over the others.

Learner Profile. The instructional objectives should be appropriate for the learner's level of ability. Many learners may need
prerequisite skills and knowledge for success with any delivery system. Key information about the learner characteristics can be
used to develop a profile. It is recommended that, as a minimum, such a profile should include information on cognitive style,
aptitude and ability, relevant experience, educational status, level of achievement with subject domain, attitude, interest level,
age, and gender (Seels and Glasgow, 1998).

Task Characteristics. The instructional objectives should be appropriate for the tasks associated with the subject matter that
is to be learned. A clear description of the topic to be learned, and the steps necessary to achieve this, can lead to clear
instructional objectives. Task analysis is frequently employed to achieve this goal. Its purpose is to define the operational
components of a skill or subject matter (Jonassen and Hannum, 1995).

Grouping. The instructional objectives should be appropriate for the grouping arrangement and learning situation. This can be
simply determined by deciding whether the instruction will be with a large number of students, small or just one individual.
Additionally, it is necessary to identify whether the instructional objectives require independent or group study, and the physical
locations of the students.

Many technologies can provide support opportunities for all of the above options. However, research shows that the inherent
properties of some technologies have more affinity for supporting some factors over others (Seels and Glasgow, 1998).

Delivery System

The delivery system is the method used by the instructional system to transfer information and knowledge from the subject
matter expert (human and/or machine) to the learner or vice versa. The choice of the "optimal" delivery system should be made
only after the instructional objectives have been clearly identified and specified. In general, the delivery system can either employ
older and more traditional technologies (print, audiovisual, etc.) or newer technologies (computers, telecommunications, etc.), or
a combination of both (Seels and Glasgow, 1998). Although both the older and the newer technologies continue to coexist as
possible alternatives, the focus of this study is on the newer, electronic technologies. Furthermore, older technologies are
increasingly being incorporated in newer technologies (e.g., video and audio information can be delivered by cassettes or by
multimedia computer software such as CD-ROMs).

Today, most of the delivery systems are highly integrated information technologies, with the computer playing a pivotal role.
They incorporate two major classes of technologies: telecommunications (e.g., teleconferencing, Internet, etc.) and
multimedia/hypermedia computing (e.g., CD-ROM, World Wide Web, etc.). These technologies (whether used together or
separately) are often used for distance learning and can provide individualized, interactive, and multi-sensory learning
experiences. They have the advantage of learner control features, which can allow for interactivity. They can be used to access
information and people either linearly and/or randomly, as well as locally and/or remotely. Moreover, they are capable of
integrating media of many different types from many different sources, all within the control of the system software and/or
learner (Seels and Glasgow, 1998). We believe that these digital delivery systems, when used as components of an instructional
system, have major characteristics and dimensions that effect the educational experience. These characteristics include control,
presence, media, and connectivity.

Control. The level of interactivity is a measure of the amount of control a learner has in the learning process and over the
information source. With many instructional systems, this interaction is complicated by the inclusion of digital technology. That is
to say, with the use of technology systems, one needs to determine the relationship to the traditional instructor-learner dyad
(Branson, 1997). These instructor-technology-learner relationships can be categorized in terms of the following configurations:

    Instructor as Lecturer: The instructor has direct control over the learning activities and the technology and content,
    while the learner does not (e.g., audio-visual technologies, programmed instructional software)
    Instructor as Facilitator: The instructor has direct control over the learning activities and indirect control of the
    technology and content, while the student has direct control of the technology and content (e.g., hypermedia, World
    Wide Web)
    Technology as Mediator: The instructor and learner dyad is mediated by the technology, with the instructor having
    indirect control over the learning activities, and both the instructor and learner having direct control of the technology and
    content (e.g., teleconferencing, computer conferencing)
    Technology as Tutor: The instructor plays little or no role over the learning activities and has no control of the
    technology, the learner has complete control of the learning activities and the technology and content (e.g., intelligent
    tutoring systems, simulations)

Furthermore, in those cases when the learner can directly interact and control the technology and content, one should be able to
identify the locus of this control. That is to say, one needs to determine whether the control over the content resides
predominantly with the instructional technology (system control), with the learner (learner control), or somewhere in between
(both).

Presence. Digital technology systems provide the means for the instructor and learner to come together physically or virtually,
synchronously or asynchronously. These modes can be characterized along two dimensions: space or location and time. The
possible combinations of these two dimensions yield the following possible spatio-temporal configurations (Hedberg, Brown,
and Arrighi, 1997):

    Same-place/Same-time: Instructional and learning activities are synchronous and instructor and learner are co-located.
    Typical technologies available for this configuration may include all computer-assisted and programmed instruction.
    Different-place/Same-time: Instructional and learning activities are synchronous and instructor and learner are remotely
    located. Typical technologies available for this configuration may include MUD's (Multi-User Dungeons), On-line chat
    systems, and many types of computer-conferencing systems.
    Different-place/Different-time: Instructional and learning activities are asynchronous and instructor and learner are
    remotely located. Typical technologies available for this configuration may include electronic mail, Internet Usenet
    (newsgroups), and electronic bulletin boards.
    Same-place/Different-time: Instructional and learning activities are asynchronous and instructor and learner are
    co-located. Typical technologies available for this configuration may include tutoring systems and simulation software.

It is obvious that the choice of any of the above modes of deliveries will require an "appropriate" choice of technology. It is a
commonly held view that some are more appropriate than others. However, the level of technology effectiveness in enhancing
the instructional and learning processes remains to be seen, especially with distance learning (Recker, 1997).

Media. Digital technology systems allow for the dynamic access and processing of a multitude of media types (multimedia).
These multimedia types commonly include text, audio, graphics, and video, and are usually interactive (e.g., CD-ROM, DVD).
Furthermore, when these media are dynamically linked or hyperlinked, they allow for significant learner control over the
information. Finally, increasingly, these media are becoming immersive, where the information can be presented in
three-dimensional space, allowing the learner to interact with a virtual environment (e.g., virtual reality, simulations). In general,
the choice of media is highly dependent on the specific learning context (Kozma, 1991). However, much research is still
needed, especially with hypermedia and virtual reality technologies.

Connectivity. Digital technology systems allow for the interconnection of people and information resources. More specifically,
they can be designed to facilitate and support communication, collaboration, coordination, and cooperation among members of
groups. These technologies are often referred to Computer-Mediated Communication (CMC) and GroupWare. Presently, the
Internet is the network of choice with educators to connect instructors, learners and information on a global basis. Furthermore,
with the popularity of the World Wide Web, this information can be hypermediated, highly unstructured, and readily available.
Consequently, we are currently experiencing an explosion of Web-based instructional systems. The Web has suddenly become
the de-facto, global technology platform for instruction and learning. Although Web-based instruction is the fastest growing area
of educational technology research, we know little about how to effectively design and implement these systems for educational
applications (Romiszowski, 1997).

Learning Outcomes

The assessment of learning outcomes provides the major feedback mechanism within the instructional design process. It is
critical in evaluating the instructional system and its effectiveness. The information that is collected as evidence of learning
achievement will depend on the nature of the competency being measured. Usually, these consist of cognitive tests
(measurements of intellectual skills), performance tests (measurements of capability), and attitudinal test (measurements of
disposition and perspective). Additionally, the instrument and technique used to assess these outcomes will also depend on the
learning domain and objective — written tests for cognitive objectives, portfolios for performance objectives, and interviews for
attitudinal objectives (Seels and Glasgow, 1998).

The use of Bloom's Taxonomy of Educational Objectives (Bloom, 1956), as an example, provides a widely accepted and
researched framework for evaluating cognitive abilities. These include knowledge, comprehension, application, analysis,
synthesis, and evaluation. Furthermore, often, these cognitive skills are classified as lower-order (knowledge, comprehension,
and application) and higher-order (analysis, synthesis, and evaluation). Assessment tests based on Bloom's Taxonomy have
been effectively employed to measure the effectiveness of educational technology on cognitive learning (Paolucci, 1996).

Finally, too many educational technology evaluation studies minimize the cognitive domain (Jones and Paolucci, 1996). In
general, all learning objectives have a cognitive component associated with them, whether these are primarily behavioral or
affective in nature. In general, we think that cognitive tests are particularly important in the assessment of learning and technology
and, therefore, they should receive much more research attention that they are presently given.

Conclusions

In this paper we have attempted to define a framework which brings together the multiple dimensions of integrating technology
with the learning process. Our goal is to establish the relationships amongst the various dimensions of instructional objectives,
delivery system, and learning outcomes. This is done with the aim of identifying the need, and laying a foundation to allow
controlled studies that contribute meaningful inputs to the open question on the effectiveness of technology on learning
outcomes. Only through systematic research and assessment will we identify the appropriate technologies to deliver specific
learning objectives and materials. This framework was developed to provide some guidance in the development of this research
agenda.

This is a first stage in providing meaningful answers to these questions. Jones and Paolucci (1996) identify that less than 5% of
research completed to date may contribute in this respect, which leaves substantial work for the future. In fact, it is this lack of
research, addressing the specific mix of the dimensions discussed that warrant this type of academic discourse.

Additionally, while this framework addresses the teaching learning process and the potential for technology to contribute to the
system, it ignores what may be a major consideration that at some point must be considered in a full multi-dimensional analysis.
That is the issue of cost effectiveness. Given the investment in technology, we must at some level consider the incremental cost
of adding technology to the process and the value added to the learning outcomes by the expenditures made. This may indeed
be the most difficult of all dimensions to assess.

References

    Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals,
    Handbook I: Cognitive domain. New York: Longman & Green.

    Branson, R.K. (1997). The Florida school year 2000 initiative: Redesigning public education. Educational
    Technology, 37(6), 14-23.

    Clark, R. (1983). Reconsidering Research on Learning Media. Review of Education Research, 53, 4.

    Dick, W., & Carey, L. (1996). The systematic design of instruction (4th ed.). New York: HarperCollins
    Publishers.

    Gagne, R.M., Briggs, L.J., & Wager, W.W. (1992). Principles of instructional design (4th ed.). New York:
    Harcourt Brace Jovanovich Publishers.

    Harrow, A.J. (1972). A taxonomy of the psychomotor domain. New York: David McKay Co.

    Hedberg, J., Brown, C., & Arrighi, M., (1997). Interactive multimedia and Web-based learning: Similarities and
    differences. In B.H. Khan (ed.), Web-based Instruction. Englewood Cliffs, NJ: Educational Technology
    Publications.

    Jonassen, D.H., & Hannum, W.H. (1995). Analysis of task analysis procedures. In G.H. Anglin (ed.),
    Instructional technology: Past, present, and future. Englewood, CO: Libraries Unlimited, Inc., 197-209

    Jones, T. & Paolucci, R. (1996). Putting Educational Technology in Perspective: the Question of Learning
    Effectiveness. SITE '97 Conference Proceedings, Orlando, Fl, April 1 _ 5, 1997.

    Katz, E., Tate, R.G. & Weimer, W. (1995) Survey of Telematics for Education and Training: Vol. 2, United
    States. European Association of Distance Teaching Universities, 1995.
    http://194.70.69.3/ethos/survey/vol2/techsch.htm

    Kemp, J.E., Morrison, G.R., & Ross, S.M. (1994). Designing effective instruction. New York: Merrill.

    Kozma, R. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.

    Krathwohl, D.R., Bloom, B.S., & Masia, B.B. (1964). Taxonomy of educational objectives: Handbook II:
    Affective domain. New York: David McKay Co.

    Paolucci, R. (1996). The effects of cognitive style and knowledge structure on performance using a hypermedia
    learning system. Journal of educational multimedia and hypermedia (in press).

    Recker, M. (1997). Appropriate use of educational technologies: A layered approach. Educational Technology
    Review, 7, 9-12.

    Reiser, R.A., & Dick, W. (1996). Instructional planning: A guide for teachers (2nd ed.). Boston: Allyn and Bacon.

    Romiszowski, A.J. (1997). Web-based distance learning and teaching: Revolutionary invention or reaction to
    necessity. In B.H. Khan (ed.), Web-based instruction. Englewood Cliffs, NJ: Educational Technology
    Publications.

    Seels, B. & Glasgow, Z. (1998). Making instructional design decisions. Englewood Cliffs, NJ: Educational
    Technology Publications.

    Smith, P.L., & Ragan, T.J. (1993). Instructional design. New York: Merrill/Macmillan Publishing.

    Willis, J. (1995). A recursive, reflective instructional design model based on constructivist-interpretivist theory.
    Educational Technology, 35(6), 5-23.
 

Dr. Rocco Paolucci is Associate Professor & Chair, Computer Information Science Dept., Cabrini College, 610 King of Prussia Rd., Radnor, PA., 19087. Voice: (610) 902-8332 . Fax: (610) 902-8309. E-mail: paolucci@cabrini.edu

Dr. Trevor H. Jones is Assistant Professor of Information Technology, School of Business and Administration, Duquesne University, Pittsburgh, PA., 15282. Voice: (412) 396-6243. Fax: (412) 396-4764. E-mail: jonest@duq3.cc.duq.edu