definition of evaluation by different authors

0000002868 00000 n Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. The risk of relying on narratives to assess impact is that they often lack the evidence required to judge whether the research and impact are linked appropriately. n.d.). To understand the socio-economic value of research and subsequently inform funding decisions. Evaluation is the systematic collection and inter- pretation of evidence leading as a part of process to a judgement of value with a view to action., Evaluation is the application of a standard and a decision-making system to assessment data to produce judgments about the amount and adequacy of the learning that has taken place., 1. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). 2008), developed during the mid-1990s by Buxton and Hanney, working at Brunel University. CERIF (Common European Research Information Format) was developed for this purpose, first released in 1991; a number of projects and systems across Europe such as the ERC Research Information System (Mugabushaka and Papazoglou 2012) are being developed as CERIF-compatible. In the UK, more sophisticated assessments of impact incorporating wider socio-economic benefits were first investigated within the fields of Biomedical and Health Sciences (Grant 2006), an area of research that wanted to be able to justify the significant investment it received. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. 2005; Wooding et al. A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. 5. Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. Authors from Asia, Europe, and Latin America provide a series of in-depth investigations into how concepts of . There is a great deal of interest in collating terms for impact and indicators of impact. Evaluative research is a type of research used to evaluate a product or concept, and collect data to help improve your solution. 2008; CAHS 2009; Spaapen et al. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide, This PDF is available to Subscribers Only. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.. stream From the outset, we note that the understanding of the term impact differs between users and audiences. In this case, a specific definition may be required, for example, in the Research Excellence Framework (REF), Assessment framework and guidance on submissions (REF2014 2011b), which defines impact as, an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Enhancing Impact. In the UK, evidence and research impacts will be assessed for the REF within research disciplines. Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. 0000342980 00000 n Baselines and controls need to be captured alongside change to demonstrate the degree of impact. Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. Although some might find the distinction somewhat marginal or even confusing, this differentiation between outputs, outcomes, and impacts is important, and has been highlighted, not only for the impacts derived from university research (Kelly and McNicol 2011) but also for work done in the charitable sector (Ebrahim and Rangan, 2010; Berg and Mnsson 2011; Kelly and McNicoll 2011). 2009). An evaluation essay or report is a type of argument that provides evidence to justify a writer's opinions about a subject. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. 0000348082 00000 n Donovan (2011) asserts that there should be no disincentive for conducting basic research. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. Impact can be temporary or long-lasting. The Oxford English Dictionary defines impact as a Marked effect or influence, this is clearly a very broad definition. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. In many instances, controls are not feasible as we cannot look at what impact would have occurred if a piece of research had not taken place; however, indications of the picture before and after impact are valuable and worth collecting for impact that can be predicted. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, What is The Concept of Evaluation With its Importance? 0000008591 00000 n Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. 2007). 0000007777 00000 n Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel (Duryea et al. Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact. 0000010499 00000 n In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. 2009), and differentiating between the various major and minor contributions that lead to impact is a significant challenge. They aim to enable the instructors to determine how much the learners have understood what the teacher has taught in the class and how much they can apply the knowledge of what has been taught in the class as well. It is time-intensive to both assimilate and review case studies and we therefore need to ensure that the resources required for this type of evaluation are justified by the knowledge gained. Figure 2 demonstrates the information that systems will need to capture and link. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. By allowing impact to be placed in context, we answer the so what? question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. The term comes from the French word 'valuer', meaning "to find the value of". Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. To achieve compatible systems, a shared language is required. n.d.). 10312. It is acknowledged that one of the outcomes of developing new knowledge through research can be knowledge creep where new data or information becomes accepted and gets absorbed over time. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. The term "assessment" may be defined in multiple ways by different individuals or institutions, perhaps with different goals. x[s)TyjwI BBU*5,}~O#{4>[n?_?]ouO{~oW_~fvZ}sCy"n?wmiY{]9LXn!v^CkWIRp&TJL9o6CjjvWqAQ6:hU.Q-%R_O:k_v3^=79k{8s7?=`|S^BM-_fa@Q`nD_(]/]Y>@+no/>$}oMI2IdMqH,'f'mxlfBM?.WIn4_Jc:K31vl\wLs];k(vo_Teq9w2^&Ca*t;[.ybfYYvcn Oxford University Press is a department of the University of Oxford. By asking academics to consider the impact of the research they undertake and by reviewing and funding them accordingly, the result may be to compromise research by steering it away from the imaginative and creative quest for knowledge. While assessments are often equated with traditional testsespecially the standardized tests developed by testing companies and administered to large populations . One might consider that by funding excellent research, impacts (including those that are unforeseen) will follow, and traditionally, assessment of university research focused on academic quality and productivity. Cb)5. This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. Not only are differences in segmentation algorithm, boundary definition, and tissue contrast a likely cause of the poor correlation , but also the two different software packages used in this study are not comparable from a technical point of view. Studies (Buxton, Hanney and Jones 2004) into the economic gains from biomedical and health sciences determined that different methodologies provide different ways of considering economic benefits. Many times . The case study of the Research Information System of the European Research Council, E-Infrastructures for Research and Innovation: Linking Information Systems to Improve Scientific Knowledge, Proceedings of the 11th International Conference on Current Research Information Systems, (June 69, 2012), pp. Standard approaches actively used in programme evaluation such as surveys, case studies, bibliometrics, econometrics and statistical analyses, content analysis, and expert judgment are each considered by some (Vonortas and Link, 2012) to have shortcomings when used to measure impacts. From 2014, research within UK universities and institutions will be assessed through the REF; this will replace the Research Assessment Exercise, which has been used to assess UK research since the 1980s. Worth refers to extrinsic value to those outside the . Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. For systems to be able to capture a full range of systems, definitions and categories of impact need to be determined that can be incorporated into system development. Reviews and guidance on developing and evidencing impact in particular disciplines include the London School of Economics (LSE) Public Policy Groups impact handbook (LSE n.d.), a review of the social and economic impacts arising from the arts produced by Reeve (Reeves 2002), and a review by Kuruvilla et al. This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). In the majority of cases, a number of types of evidence will be required to provide an overview of impact. In this sense, when reading an opinion piece, you must decide if you agree or disagree with the writer by making an informed judgment. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. 1. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. HEIs overview. What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? This transdisciplinary way of thinking about evaluation provides a constant source of innovative ideas for improving how we evaluate. This is particularly recognized in the development of new government policy where findings can influence policy debate and policy change, without recognition of the contributing research (Davies et al. In putting together evidence for the REF, impact can be attributed to a specific piece of research if it made a distinctive contribution (REF2014 2011a). Definition of Assessment & Evaluation in Education by Different Authors with Its Characteristics, Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have., 2. Scriven (2007:2) synthesised the definition of evaluation which appears in most dictionaries and the professional literature, and defined evaluation as "the process of determining merit, worth, or significance; an evaluation is a product of that process." . (2007:11-12), describes and explains the different types of value claim. Why should this be the case? The Goldsmith report (Cooke and Nadim 2011) recommended making indicators value free, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. According to Hanna- " The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". What are the reasons behind trying to understand and evaluate research impact? At least, this is the function which it should perform for society. Assessment is the collection of relevant information that may be relied on for making decisions., 3. 2006; Nason et al. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. (2008), and Hanney and Gonzlez-Block (2011). (2007), Nason et al. Definitions of Performance Appraisal - By McGregor and Dale Beach . A comparative analysis of these definitions reveal that in defining performance appraisal they were saying the same thing, but in a slightly modified way. While the case study is a useful way of showcasing impact, its limitations must be understood if we are to use this for evaluation purposes. The Author 2013. Different authors have different notions of educational evaluation. working paper). Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. In demonstrating research impact, we can provide accountability upwards to funders and downwards to users on a project and strategic basis (Kelly and McNicoll 2011). In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. Organizations may be interested in reviewing and assessing research impact for one or more of the aforementioned purposes and this will influence the way in which evaluation is approached. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility. This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. SIAMPI is based on the widely held assumption that interactions between researchers and stakeholder are an important pre-requisite to achieving impact (Donovan 2011; Hughes and Martin 2012; Spaapen et al. Developing systems and taxonomies for capturing impact, 7. This raises the questions of whether UK business and industry should not invest in the research that will deliver them impacts and who will fund basic research if not the government? An alternative approach was suggested for the RQF in Australia, where it was proposed that types of impact be compared rather than impact from specific disciplines. There are areas of basic research where the impacts are so far removed from the research or are impractical to demonstrate; in these cases, it might be prudent to accept the limitations of impact assessment, and provide the potential for exclusion in appropriate circumstances. The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). Citations (outside of academia) and documentation can be used as evidence to demonstrate the use research findings in developing new ideas and products for example. The case study approach, recommended by the RQF, was combined with significance and reach as criteria for assessment. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital.