In today's information-driven society, the call for resources required to maintain the impetus of technological use receives much
attention. This is no less evident in the realm of education, where significant effort (time and cost) is invested in identifying and
bringing to the arena the latest in technology-based products for application in the instruction/learning process educational
technology.

Educational technology has historically played a central role in the delivery of instruction in the classroom. Over the years,
teachers have used books, televisions, projectors, and many types of lab equipment as tools to help them transfer knowledge to
their students (Hawkins, 1993). In the past decade, computers have been added to teacher's technology toolbox. Like no other
technology, it has been able to capture the imagination of educators. Today, computers can be frequently found in classrooms
and laboratories throughout our schools, colleges, and other educational and training institutions.

In the classroom, the applications of computers have evolved from the provision of drill and practice for remediation, to later
providing structured curriculum and instruction. Today, the computer is often used for knowledge explorations and construction
(Jonassen, 1993). On the technological front, during the past decade, the computer has evolved from being a command-line
instructional machine. Graphical and friendlier user interfaces have made human-computer interaction much easier and more
effective.

Today, computers inexpensively and easily provide highly interactive multimedia information (text, sound, images, and video) to
its users. Furthermore, these interactive multimedia systems may now be programmed to deliver hypermedia multimedia
information stored in networks of nodes connected by links (Dede, 1992). These systems come in a variety of formats
(CD-ROM, authoring software, etc.). However, the most popular hypermedia technology within many educational institutions is
the Internet's World Wide Web. The wide reach of the Internet network (global in scope), coupled with the multimedia
capabilities of the computers it interconnects, have made the World Wide Web a truly international and highly distributed
hypermedia system. Today, the Web is increasingly being used for a multitude of educational applications, most notably
"distance learning".

The above technological capabilities have not come cheap. Recently published analysis of key U.S. Department of Education
national studies reveal that:

    In the 1994-1995 school year, schools spent approximately $3.3 billion on educational technology.
    Today, there are an estimated 5.8 million computers for instructional use, approximately one computer for every nine
    students.
    Between 1989 and 1992 alone, K-12 schools increased their computer inventory by nearly 50%, jumping from 2.4
    million units to 3.5 million units.

It is worthwhile to note that the above numbers are for basic educational institutions only (K-12), and do not include
investments made by higher-educational institutions (Fulton, 1996).

However, after more than a decade of significant technology investments by America's schools (as described above), many
educational administrators and policy-makers are asking the questions, "Do computers and other related technologies make a
difference in learning? Do they improve achievement and performance scores?" The brief answer to these questions, based on
scientific research, seem to be mixed and inconclusive. Many reasons are given as to why this is the case. Some researchers
claim that evaluation of educational technology effectiveness is not easily captured by the use of current standardized tests
(Hawkins 1993, Rockman, 1993). Still others claim that a major reason why educational technology has not had a more
positive effect on learning outcomes is that appropriate staff development has not taken place (Fulton, 1996).

On campuses of all levels today, the cry is for more money to support the investment in technology. The resultant investments
are often made at the expense of other resources. There is an obvious economic trade-off required by the sharing of limited
resources, in this case funding for other educational initiatives. This trade-off, currently weighed towards technology, is often
supported by bodies oblivious to their applications and use in their particular domains. Regardless of the reasons why, as we
approach a second decade of educational technology applications and infusion, it is important to pause and assess the research
focus (or lack of) presently given to educational technology effectiveness by the academic community.

Research Question

In this paper, we attempt to question the research associated with the adoption of technology in the educational environment
based on the question of learning effectiveness, and the degree to which this effectiveness issue has been demonstrated in
technology adoption. In the context of this analysis, we define educational technology research in a broader scope as not only
including the latest hardware, but also the various software products being made available for the classroom, along with the
various research studies undertaken to develop the environments of teaching and learning.

We also look at the use of technology in the context of providing a communication channel or conduit for the dissemination of
the educational process. This is distinguished from the use of technology as a "subject matter" or course content (i.e., technology
education). We are interested in the effectiveness of technology as a substitute, or partial substitute for traditional
teaching/learning methods.

We question the outright adoption and acceptance of technology, and the associated expenditures, based on current research
results in this domain. We present a meta-analysis of the research literature in the domain of educational technology. We classify
the literature according to their research content. Content areas identified are as follows: Technology, Analysis & Applications,
Design, Implementation, Educational Issues, Instructional Process Evaluation, and Learning Performance Evaluation.

Our intent is to identify the weighting and attention given by the research community to the last classification item (Learning
Performance Evaluation). Classifications reflect the visibility and work accomplished on testing, measuring, and comparing
results from differing educational delivery channels which incorporate technology. Furthermore, we identify the emphasis within
these classifications by addressing research concerned with high-level educational outcomes, versus those associated with
specific variables within these high-level models. The study is a review of nine major educational technology journal publications,
spanning the last three-year time period, and includes the classification of almost one-thousand articles.

Our paper includes the results of this meta-analysis indicating the breakdown of publication content into the specified research
categories. We intend to investigate the percentage of work that is directed at measuring the learning effectiveness with
technology as it is currently used or is proposed to be used, and contrast this with the acceptance of technology adoption.
Furthermore, we test the content of that limited amount of research concentrating on learning outcomes measurements by
identifying those subcomponents that address high-level variables versus domain-specific model variables. Lastly, weaknesses in
present technology adoption decisions are discussed, along with a call for an increased emphasis on learning outcomes and/or
development of more effective educational measurements.

It is our intention that the results of this study will stimulate discussion of this issue among all parties involved in education and
educational delivery systems, including all levels of academia as well as educational institutions and private corporations. This
includes administrators as well as private practitioners and policy-makers, since the issues raised pertain not only to the
application of educational technology, but also to the policy issues addressed at the highest levels of decision-making.

Research and Data

Our research was derived from a review of nine major refereed journals addressing technology in education. We attempted to
cover a consecutive three year time span for each journal. Consequently, the dates covered by the review range from 1991
through summer 1996, due to the availability of the articles and journals. This resulted in a review of 932 individual journal
articles. The journals reviewed were:

    Journal of Educational Technology Systems
    Computers and Education
    Journal of Research on Computing in Education
    Computers in the Schools
    Educational Technology
    Educational Technology Research and Development
    International Journal of Instructional Media
    Journal of Educational Computing Research
    Journal of Research on Computing in Education

Each article was reviewed and categorized according to its major content. The categories used were developed from the
framework used for classifying papers by the Association for the Advancement of Computing in Education (AACE) at the
Ed-Media & Ed-Telecom, 1996 Conference. Our intent was to identify the percentage of journal articles that specifically
addressed the impact on learning outcomes of a particular technology or combination of technologies. The categories used for
classifying the articles were:

Technology (TEC): explanations and introduction of new and developing technologies (as distinguished from application
development - see APP below).

Applications (APP): applications of technology including analysis of applications.

Development (DEV): development of educational technology design methods including evaluation of methodologies, models,
guidelines, frameworks and taxonomies.

Implementation (IMP): implementation of educational technology including case studies, organizational, management, and
personnel issues (e.g. gender, minority, age issues).

Pedagogy (PED): discussion of pedagogical, learning and other issues.

Evaluation of instructional process (EIP): quantitative and qualitative assessment of instructional processes and outcomes.

Evaluation of learning and performance outcomes (ELO): quantitative and qualitative assessment of affective, cognitive
and physiological outcomes.

The final category listed above (ELO) was the category of interest. In addition to the broad analysis, we further subdivided the
articles in the category of interest, "evaluation of learning outcomes", into two further subcategories, quantitative and qualitative.
This was done in an attempt to further quantify the empirical research being undertaken, and to identify specifically that
percentage directed at learning outcomes from the use of educational technologies.

Our overall strategy was to gage the research effort being applied to assessing learning outcomes from the application of
technology and make some comparison against the level of expenditures in this area. This was to provide us with a basis for
raising questions and/or answers as to the applicability of the groundswell of support for technology in education. To achieve
this, we conducted a meta-study of the literature to identify what percentage of research, and potentially the types of research,
that support the adoption of technology. We hypothesized that there exists a lack of significant research to substantiate the level
of technology expenditures and adoption.

We observe from the results in Table 1 above that 20% of the research is directed at identifying the learning outcomes derived
from utilizing technology in education. While this may seem like a significant proportion, we compare this with over 50% which
is directed at developing methodologies, applications and installing those same technologies. Additionally, when we further
investigate the research category aimed specifically at learning outcomes (Table 2), we find that approximately 20% of this
research is subjective (qualitative). When this percentage is applied back to the original analysis, we estimate that only 16%
(78.8 * 20.2) of work is of a quantitative and objective nature.

One additional issue to consider in the questions raised here is not fully documented from our review, but is added in terms of
subjective observation; much of the quantitative research is aimed at specific channels of technology and very specific content
domains. For example, empirical studies included measurements of attitudes towards technology in particular disciplines; the
effects of computers on anxiety; the addition of computer-aided instruction (CAI) to traditional delivery methods; measurements
of communications among students using E-mail. While studies such as these have their place in the research arena, and
individually can be considered aspects of the learning process, they add little when questions are raised regarding the
cost/benefits associated with the massive expenditures on educational technology when compared with student achievement. In
summary we conclude that approximately only 16% of total research being published is associated with quantitative
documentation of learning outcomes and achievement, which we deem the effectiveness of the technologies being adopted.
Our observations suggest that approximately 5% of total research is conducted using formal methods such as control groups
with comparative learning outcomes (i.e. experimental).

Conclusion

When we consider the broader issues of the learning effectiveness of technology, we see a large discrepancy between the scale
on which the expenditures are being made and the level at which the research is being conducted. To a large extent, the
expenditures and adoption of technology are discussed at a policy level whereas the research being conducted is at an individual
variable level. Our point here is that the research is not of a sufficiently high (complex) and formal level to support conclusions
being made by technology adopters when learning outcomes are considered.

What are the implications of this? The answer probably lies in the perspective from which education is viewed, and possibly
what position is held regarding the future of education and the direction in which education delivery should take. Perhaps the
first issues to be raised must the purpose and definition of education. Whatever they may be, we believe that the level of student
achievement and learning outcomes are of central importance. Many argue, already, that the quality of U.S. education received
at any level has already declined, not only with respect to historical levels, but also with respect to other nations, worldwide.
Surely then, it is incumbent on the education system to establish that future delivery methods provide some added value without
possibly contributing to any further decline.

Certainly many questions surround the quality and methodologies of current education. The answers to these questions tend to
be in the form of intuitive expressions from individuals, and work which supports those answers tends to be in qualitative form
with little formal research support. We emphasize here the distinction of formal research directed at learning outcomes from
other types of assessment concerning technology adoption. Many individuals, groups and institutions claim quantitative support
for adoption of various technologies. However, these tend to be in the form of superficial assessments indicating little more than
satisfaction for the delivery channel used, and again provide little input to the question of "technology effectiveness".

In response to these issues we maintain that a missing component is the establishment of research agendas utilizing formal
experimental methodologies. We argue that we must be able to demonstrate that quantifiable learning outcomes can be achieved
and sustained through technology adoption. This can only be established through research methodologies. It is reasonable to
expect "educational technology delivery" models, with recognized variables (e.g. delivery channels, content types, etc.) and
quantifiable dependents (learning outcomes).

With this lack of a formal and quantifiable approach to the introduction of technology, few if any measures or guidelines exist
which help in distinguishing which types of technological delivery channels are suitable and effective for the various contents
found at differing levels of education. If indeed, there is no formal support for the various technologies, is it not valid to question
the expenditures of resources on the scale on which we are progressing? Is it not also valid to question the motivation for these
expenditures if the motivation is coming from a source other than educational learning outcomes? Should we compromise
outcomes (however defined) for the sake of other factors and if so, do we have, as educators, some obligation to make this fact
apparent to all interested stakeholders (students, employers, parents, teachers, etc.)?

The questions of technology's' place in education are not new. Apple (1991) has previously stated:

    Whose idea of progress? Progress for what? And fundamentally, who benefits? These questions may seem rather
    weighty ones to be asking about schools and the curricular and teaching practices that now go on in them or are
    being proposed. Yet, we are in the midst of one of those many educational bandwagons that governments,
    industry, and others so like to ride. This wagon is pulled in the direction of a technological workplace, and carries
    with it a heavy load of computers. (Apple, 1991, p. 59)

While Apple goes on to discuss the potential social impact resulting from the adoption of technology, our aim is to question the
foundation on which these initial technology adoption decisions are based. Our work raises questions regarding the quantifiable
support for the adoption of technology in educational delivery when considered from a learning outcomes perspective. Our
preliminary data indicates that this support does not exist to the extent that justifies our expenditures and adoption strategies.
Ultimately, we believe that technology must be appropriately combined with content to provide the most cost effective
combinations for delivery. These combinations will be driven by a number of different factors which themselves have various
combinations; but we can find little evidence that the cost/effectiveness of combinations is being addressed in any formal and
acceptable research agenda.

References

Apple, M. W. (1991). The new technology: Is it part of the solution or part of the problem in education. Computers in the
Schools, 8(1/2/3), 59- 81.

Dede, C. J. (1992). The future of multimedia: bridging to virtual worlds. Educational Technology, 32(5), 54-60.

Fulton, K. (1996). Moving from boxes and wires to 21st century teaching. Technological Horizons in Education Journal,
23(11), 76- 82.

Hawkins, J. (1993). Technology and the organization of schooling. Communications of the ACM, 36(5), 30-34.

Jonassen, D. (1993). Conceptual frontiers in hypermedia environments for learning. Journal of Educational Multimedia and
Hypermedia, 2(4), 331-335.

Rockman, S. (1993, March). Asking the right questions. The American School Board Journal, 29-31.

 
Dr. Rocco Paolucci is Associate Professor & Chair, Computer Information Science Dept., Cabrini College, 610 King of Prussia Rd., Radnor, PA., 19087. Voice: (610) 902-8332 . Fax: (610) 902-8309. E-mail: paolucci@cabrini.edu

Dr. Trevor H. Jones is Assistant Professor of Information Technology, School of Business and Administration, Duquesne University, Pittsburgh, PA., 15282. Voice: (412) 396-6243. Fax: (412) 396-4764. E-mail: jonest@duq3.cc.duq.edu