Assessment and process of medical education

sagacity and subroutine of medical educationAssessment plays an or so-valu qualified occasion in the extremity of medical education as it is an pictureive ray which detect quality in schoolchilds fostering to motivate and direct them to what they essential learn(1). Assessment mounts tuition this statement focus on the essential role of judgment as well planned and implemented judging has an big steering effect on in tenoration beca exercise it transfers what is main(prenominal) to learn and motivate students for learning(2). Many people argued that as the class should be the key which motivate learning while judgment should be designed to be sure that learning outcomes feature occurred, So surveyment tool must has clarity of the learning function and must be designed to drive educational intent and maximize learning(3).Constructive alignment is an important influential imagination in which the students construct meaning from related learning activities and t apieceers apply learning environment which support planned learning activities to achieve the intended learning outcomes(4). So constructive alignment makes the precept systems consistent when curriculum, learning activities and appraisement methods argon aligned with intended learning outcomes(5) . Moreover, sagacity may reveal learning outcome which isnt expected but it is recognized as important outcome, so it must be compoundd into the intended learning outcome as emergent outcome(6).Formative judging promotes deeper learning as it provides students with feedback to bring forward them to go to bed their strength and weakness which reinforce students home(a) motivation to learn and improve their fellowship and skills(7). Summative assessment is a final assessment which determine the rank-order students and determine grades(1). Wass et al(7) argued superficial learning which aim mainly on passing the examination and they emphasized on the importance of feedback on stud ents assessment which encourage student comment and deep learning. However, Epstein(8) showed that summational assessment influence learning even in the absence of feedback as students engage what they expect to be turn uped on. Although formative and additive assessment argon stark in contrast, they are both assumeful and distinction between them should be made to detect which assessment is fitted alto embark onher for formative use or have sufficient rigorous for summational use(7). Van der Vleuten and Schuwirth(9) emphasized that formative and summative assessment rat be use with little difference with foc utilise on the break upment of comprehensive assessment programme in which both encourage learning and right decision nearly learners.I will focus my makeup on create verbally assessment as I am involved in assessing written examination of samarium of Radiology scince 5 years. According to Miller pyramid we use written assessment to assess the domain of cognit ion, either factual rec in all of acquaintance knows or application of knowledge and problem solving knows how. We use written assessment in the form of analyses and multiple choice challenges in formative assessment of the residents and in summative assessment of final exam. Our final written exam formed of twain papers of shows, each one formed of four study questions with triple hours duration for each, and third paper of 20 multiple choice questions with one hour duration. When we prepare a written exam we identify the direct of residents training to apply canvass which assess knowledge appropriate to students own.Essay questions are impelling method for assessing cognitive skills as they can assess ability of students to form resultant and measure their attitude and opinions, also they can give students effective feedback on their learning(10,11). But it has the disadvantage of creation clock-consuming test to grade and its test doesnt obliterate a wide domain.Ne wble and Cannon(11) verbalize that essay is either extended response questions which are useful in assessing higher cognitive skills give care analysis, entailment , problem solving, and restricted response questions used for interrogatory knowledge of unhorse level but it has the advantage of being much than substantially as scoring variation can decreased with it. Epstein(8) tell that well structured essay with clear framework can eliminate cueing and introduce more cognitive process with context rich responds. We usually used extended response questions by which we assess students higher level of knowledge, but I specify for improving essay test utility, we must make mix of the two essays types with exploitation clear words on constructing questions like using describe, criticize and compare kind of of contend to direct students to desired attend, as I find some curt structured essay questions in our exam, for example discuss radiological imaging of mammilla mass which I can change it to be compare between ultrasound and mammography for differentiating breast mass.Van der vleuten(12) tell five criteria to assess assessment tool utility which are dependability, hardiness, educational impact, acceptability and cost effectiveness. Reliability measures consistency of the assessment test and it is often described as reliability per hour of testing time as time is a limiting gene during exam, so essays are low reliable than MCQ be face it require extensiveer time to serve up(13). Schuwrith and Van der Vleuten(14) stated that inter-case correlation of different essays in veritable test is low as the essays numbers which can be asked in a certain test is limited.Chase(15) stated that essay scoring is a complex process as it has many variables which are essay fill, writer, rater and early(a) colleague variability with their significant writing effect. The most important type of reliability for rater-type assessment is inter-rater reliab ility, single inter-rater reliability (which mean correlation between two raters) casts from 0.3 to 0.8 as this depend on topic of the essay, the essay length, the rater experience and the level of rater training(16). But Munro(17) et al stated that single inter-rater reliability can be regularly obtained as 0.7 if there is continuous extensive rater training. On agreement with those authors approximately increasing inter-rater reliability we already use replicate markers for assessing essays question and the mean of their stigma is calculated to be the end gain.Essays are poor accusive test for assessing learning outcomes as there is variability in the assessment get ahead through different examiner with variation of perfect answer (18,19). Norman et al(20) stated that providing structured scar of the essay may improve its reliability but it may cause process trivialization. Schuwrith and Van der Vleuten(14) emphasized that using one marker for each essay for all students is more reliable than one marker for all essays of the same student. Davis(18) stated that using double marking for the same question is mandatory to reduce variation incidence between the markers. Beattie and James(21) suggested using checklist in marking essay to reduce subjectivity and improve objectiveness of essay as it provides the examiner with key point of each item and its allocated marks. As mentioned before, double markers are applied in our radiology department for assessing each question but we dont use checklist in marking the essay question, so I think this make our examination less reliable with poor objectivity and we have to use checklist with specific marks on each part of the question.Validity is the ability of assessing method to measure what is purported to(19). The valid method will polish what the students achieved from intended learning objectives of the course, so increasing the test item is essential for more valid test, therefore the validity of the ess ays is limited(6). Brown(22) advises using large numbers of inadequate essays to improve its reliability and validity and to reduce sampling errors. However, Davis(18) argued that as this may cause more time consuming to mark. As we begin to apply a test blueprint to determine the main content of the test which must have high content validity to cover intended learning objectives, we have to use larger number of shorter essays to be eight to ten short essays instead of four long essays correspond to test blueprint. Van der Vleuten(23) stated that assessment methods should have content validity which must be designed and mapped on a blueprint.Modified essay questions was initially produced by the Royal College of General Practitioners in London and are widely used now(11). Davis(18) stated the importance of using context rich scenario which will direct the students to answer with precise data and ontogeny exam veryity. Schuwirth and Van der Vleuten(14) showed that written case-si mulation essay appeared to be more valid as its questions focus on history taking, diagnosis, probe and examination findings which are virtually related to real practice. However Swanson et al (24) argued that as these essays arent suitable for assessing problem solving questions. Newble and Cannon(11) showed that certain skills is needed for constructing circumscribed essay questions to turn away giving idea close answers of a previous question or punshing the student on question constructing error. Also, Schuwirth and Van der Vleuten(13) emphasized that considerable structure of essay question is necessary but over-structuring may lead to limited increase in its reliability, As we use essays in both formative and summative assessment we have to use modified essays instead of traditional essays especially in resident formative assessment as we returned it to students with its model answer for discussing during the tutorial, as this will encourage student critical thinking and reflection, but also we must take training about constructing modified essay questions to avoid poor form which may cause assessment error.Schuwirth and Van der Vleuten(13) advised using essays in limited occasions when objective tests are not suitable, Objective written tests like short answer question, matching exercise and multiple choice questions ( MCQ) have the advantage of being economic, rapidly scoring, high reliable and evaluate the student in large content(25). on that point are two major format of MCQ which are True/ sham format and single best answer. True/False format can cover a broad aggregate of the topics and are easily marked but they mainly measure definitions and simple facts(26). fact and Swanson(27) explained why using True/False format is markedly reduced as it is not only difficult to construct but it mainly used to assess recalling of isolated fact to avoid ambiguous items, also they cant detect if the student who identify in good order the false stateme nt knows the right answer or not. Another disadvantage of authoritative/false format is their high probability of guessing(28). To overcome guessing, nix marking was achieved in which there is deducing marks for the wrong answer, but these may produce negative psychometric results(25). We sometimes use true/false format instead of single best answer, as we think it covers a broad items in the curriculum and can measure complex outcomes but we dont apply negative marking for MCQ correction as we think that is stressful to the students, also I have baffling memory about using negative marking when I was medical student at 2nd year I got 19/50 in physiology MCQ test and this caused to me poor willingness to MCQ risk. When, I read carefully a previous exam of True/False format, unfortunately I find some lucid questions which may cause a critical failure for these questions. So I think we must limit using these types only for assessing definitions and facts identifications and apply other types of objective tests to avoid the guessing probability of true/false format. This is in agreement with Schuwrith et al(13) who stated that True/false questions are only suitable when the question purpose is to evaluate if the student is able to determine the correctness of hypothesis.MCQ are able to evaluate broad range of learning outcomes within a short time and limited human intervention, also they have low guessing probability with free question of ambiguity(29). In the tutorial of decembrer 2010, there is a debate about effect of MCQ guessing on test reliability, but I learn from the discussion an interesting concept which emphasized that guessing doesnt change test reliability as good student is a good guesser.For constructing good MCQ items it is essential to have a good idea about the content, study the objective of the assessment and apply high quality form for items writing(27). MCQ consist of stem and several options, stem is formed of sentence or question and m ay be accompany by diagrams or tables, while the correct option is defined as keyed response and the wrong options are called distracters(29,30).Case and Swanson(27) stated that MCQ must be well structured to be simple, easily understood with using plausible distracters, also grammatical errors especially using negative and inaccurate words like never, sometimes, frequently and usually should be avoided as they may lead to examinees confusions(31). Lowe(32) stated that the useful distracters should demonstrate a misconception between the students about the right option, so writing many plausible distracters is a difficult part for MCQ building with more time consuming. The flaws of writing distracters which include using more than correct answer, using all of the above or none of the above, or making the right option is the longest one should be avoided(33). MCQ reliability increase with removing non plausible distraction(34,35). Although we choose MCQ from question banks or MCQ books to reduce the examination preparation time , unfortunately I find many drawbacks in our last MCQ exam, firstly one question contains double negatives, also in another question I find it was easily to eliminate some distracters, while other questions contain inaccurate words which are sometimes and always. So I think we must take care during choosing MCQ distracters which should appear to the students as a valid answer while it is incorrect, also we must avoid apparent incorrect or plain distracters. So, we need to take training courses for MCQ preparation and writing MCQ stems and distracters to avoid MCQ flaws and constructing good items.Collins(30) showed that MCQ have the disadvantage of being test knowledge recognition rather than constructing answer. Mcaleer(31) argued that as MCQ are an objective test which doesnt allow students the chance for giving additional information and doesnt apply examiner to put judgment on student answer quality. I agree with Mcaleer(31) as we use MCQ as an objective test to assess understanding knowledge of a broad range of learning objectives within a short time.Reliability is refered to reproducibility of the assessment score and it is expressed as a coefficient which range from 1 for perfect reliability and 0 for no reliability. MCQ are widely used due its high reliability which is attributed to its ability to assess broad amount of knowledge by providing large number of items which address areas of context specificity within a short time(7,30). Downing(36) stated that written test especially MCQ has high innate consistency reliability as the test score would be near the same if exam is repeated at later time. Van der vleuten and Schuwirth(9) showed that the predominant factor which affect reliability is domain as competence depend on context specificity. While McCoubrie(25) argued that and he stated that the assumption of MCQ as a reliable test is weak as they are only reliable because they maintain a time efficien t test with wild sampling of topics. Van der vleuten and Schuwirth(9) stated that the reliability of MCQ test in one hour is 0.62 which is increased to 0.93 for four hours test due to using more items number. Wass etal(37) stated that for important exam in which stakes are high a high reliability of 0.8 or more is essential to determine pass-fail decision but for formative assessment lower reliability can be accepted. Our final MCQ exam contain 20 questions with examination time of one hour, s has low reliability due to polished number of items within short time which miss many objectives of our curriculum, So I think we have to increase the question numbers to cover more knowledge of context specificity and consequentially increase the test time to improve the test reliability.A criticism of MCQ validity as it measures the factual knowledge and doesnt integrate skills, attitude and communication skills(25). Downing and Yudkowsky(38) emphasized that knowledge is the single best do main which determine expertise, so MCQ is a valid competence method which assess cognitive knowledge. Collins(30) stated that MCQ have a high validity if it represents a wide sample of content that serve the objective learning outcomes. However, Shumway and chasten(1) critic that as MCQ asses discrete superficial knowledge not deep understanding as they designed to detect what students know or dont know.Blooms taxonomy of educational objectives is a hierarchy of knowledge for different cognitive level which are knowledge, comprehension, application, analysis, synthesis and evaluation(39). While educators simplified Blooms taxonomy into three levels which are knowledge recalling, comprehension and analysis, and problem solving(11). Case and Swanson(27) and Mcaleer(31) showed that well-structured MCQ can assess the taxonomic higher cognitive process like interpretation, application and analysis rather than assessing recalling of facts. Peitzman et al(40) argued that as they stated us ing higher-order MCQ doesnt improve MCQ validity but it makes them more real and acceptable to students and examiner. Also, Frederiksen(41) stated it is difficult to construct MCQ with rich context as item writers tend to escape from topics which cant be easily asked. In agreement with Case and Swanson(27) and Mcaleer(31), we try to choose MCQ level with different cognitive level, and when I revise our MCQ tests I find some questions which can assess recalling of knowledge(Q*) and other assess problem solving(Q**) for the same topic, example of this isQ*what is the effective measure which reduce radiation of CT chest?a-120 mAb-150 mAc-200 mAd-250 mAQ**what of the future(a) will reduce dose of radiation for CT chest?a-reducing mA from 250 to 150b- reducing KVp from 160 to 120c-reducing the pitch to be 1 instead of 2d-reducing scanning time to be 1 instead of 2Blueprint is an important powerful tool for integrated curriculum as it maintain assessing all its intended learning objectiv es.(42). Our faculty assessment centre members work in progress and they make many orientation about blueprint construction and its importance, also they asked all departments to finish their blueprint, but until now we evaluate our exams retrograde according to our ILOs, but unfortunately in some written exam we install that the items dont cover all topics of the curriculum and missed many ILOs, also in other written exam we find a focus on certain system rather than other systems which may produce bias of examination results as the questions sample doesnt represent a big domain of knowledge. So, I think we are urgently in need to use test blueprint which cover the learning objectives and assessing methods to identify the key topics which must be tested according to our objectives and determine the questions numbers according to their corresponding weight in the context. This is in agreement with Downing and Haladyna(43) who stated that blueprint reduce two validity threats which are under-sampling bias of the curriculum and constructing irrelevant items.Consequential validity is referred to the real impact of an assessment method on learning which appropriately drive students learning(25). Wass et al(7) stated that consequential validity refers to the educational consequence of the test as it produce the desired educational outcomes, which means that students should study subject rather than studying the test. Although consequential validity is an important process, it is ignored by many examiners(44). I think our written exam has significant educational impact on how our students study, as from my experience students study what they need to pass rather than studying the whole integrated information. To improve this, we have to use different forms of written assessment which must cover the important content of the curriculum, and it should be mixed with continuous formative assessment and feedback to steer our students to determine what they study and how t hey learn. This is in agreement with Van der Vleuten(12) who stated that assessment can drive learning through four ways assessment content and structure, the question which asked and the frequency of repeated examination.Newble and cannon(11) advice using computerized optical mark reader to score and analyze MCQ tests as the computer programmed has the advantage of applying statistical data of the test which include reliability coefficient, standard deviation and test item analysis . In our exam we use a hand marking sheet of answers to correct MCQ. But recently our faculty bought a new computer machine for correcting the MCQ test, so we need to learn how to use it for interpreting the test information as these may help us to improve next exam.Shumway and Harden(1) emphasized that practicability of an assessment method depends on resources, expertise availability and their costs. Resource intensiveness is determined by cost of constructing and correcting the test items(45). Cost in cludes beginning and continuing resources which are needed for test implantation(1). Essay questions appear to be easily constructed items but specific answer key is needed which may cause more time-consuming for preparation(18). MCQ appear to be easy to grade especially with using computer machine but for good structured items more time is needed for construction(30). Shumway and Harden(1) stated that it is important to consider the relation between the assessment method cost and its benefit. Van der Vleuten(12) critics that as he considered investment in an assessment methods is an investment in teaching and learning process. I think we must take care about the criteria of each method and balanced them against each other as the outcome may change according to the assessment context specificity. Also, In agreement with Van der Vleuten, I think we must use different assessment tools especially for summative assessment for high stakes exam to obtain more reliable and valid assessmen t.Schuwirth et al(45) explained that students can answer correct MCQ by detecting the right answer but they arent able to answer it in the absence of MCQ options. Graber et al(46) explained the problematic effect of MCQ cueing which may cause diagnostic errors especially if diagnostic reasoning is assessed. Schuwirth et al(14) advise using extended matching items and short-answer question as they can reduce the cueing effect.Extended matching questions (EMQs) are good authentic test as they use real clinical scenario which need sufficient clinical knowledge and can test a wide range of topics for knowledge application and problem-solving ability like diagnosis, investigation and management(47). Beullens et al(48) emphasized that EMQs are able to assess extended learning and minimize recognition effect rather than memorizing facts which is needed for MCQ solving. McAleer(31) critics that as EMQs with its many different items and long list of suitable answers are difficult to construc t. However, Schuwrith and Van der Vleuten(13) advice using EMQs as they are good reliable test with short time scoring. We dont have experience in EMQs, but after knowing its importance and its significant role for improving written assessment reliability, I think before applying this form we need training of how construct these questions and how practice them to avoid bad representation of some items.Short answer questions is an important assessing tool because they are objectively scored test as they need clear sets of answer with little guessing incidence(3). McAleer(31) critics that as he stated, although short answer questions are easy constructed item, it is used only to measure recalling of information as they cant measure complex learning outcomes like synthesis and information analysis. Epstein(8) stated that short answer questions can be used for summative and formative assessment but its reliability depend on mainly training the students how they answer these items. We do nt apply short answer questions in our exam, but I think we can use in certain situation when we want to cover broad area of content and be sure that the students are able to supply an answer rather than choosing it from many options.Score determines the number of correct answers of an assessment but it doesnt represent the quality of students performance(49). Norcini(50) stated standard move is the process by which pass mark of exam is determined to distinguish competent from non-competent students as it allows for variation according to the level of test difficulty.There are two types of standard mountain relative (Norm-refrenced) and absolute (criterion-refrenced) standard, in relative standard setting fixed number of students will pass the exam irrespective to their level of competence as it is related to catch performance and fixed percentage of success(50). In our faculty we use relative standard setting to select students with highest score for admission to postgraduate cou rse when fixed number is determined. In the tutorial of , I gain a new information which is supposed from one of our peer who advice using relative standard setting for choosing lower achiever in formative assessment who need extra-training.Absolute standard setting is more suitable for competence test as accurate standard should be determined below which the prognosis wouldnt be fit for particular purpose(7). Absolute standard setting may be test-centered method or examinee-centered method, in test-centered method (like Angoff method) the examiner evaluate every item to hypothetically determine how the candidate will get in each item(51). While in examinee-centered method (like contrast group method), panelists decide the pass score by detecting it on the score outdo which should be most fit to the exam purpose(52). In our faculty we dont use any forms of standard setting as we use 60% as an type setting for pass/fail decision for all test types, But as we recently apply assessi ng centre in our faculty, I think we must use standard setting in our assessment, in my opinion I prefer applying modified Angoff method as an example of absolute standard setting as it is widely used in medical assessment and it can be used for many assessment types. This is in agreement of Smee and Black(53) who stated that modified Angoff method reduce the difficulties of traditional Angoff method ,for examlple the difficulty of detecting hypothetical borderline candidates in Angoff method which is facilitated by supplying the examiners with real test scores of previous assessment of the candidates.Norcini etal(50) stated that absolute standard setting is applied either as conjunctive or compensatory standard, In conjunctive standard the candidate must exceed each item separately to pass the total test, while in compensatory standard the test scoring permit the candidate to remunerated poor performance in one item by high performance in another item. In our written assessment we use compensatory method in which the standard is achieved according to total test performance, but now I think we can use conjunctive method in assessing essay paper by which the candidates must pass each essay separately as this will improve their studying to pass in each item.Case and Swanson(27) stated that many medical schools provide their faculty with item analysis of their test before test results are announced by which a useful information about the quality of each item separately and the whole test quality are obtained. Items analysis will be valuable when it maintains effective feedback to test writers as this will improve their skills in further test construction, also it would be helpful in discarding poor items and detecting certain areas of the content which may need more clarity(30). Item difficulty is detected from the proportion of students who answered each item correctly, Items are considered difficult if 50% of students or less answered them correctly and low di fficulty if 85% or above of students answer the item, while moderate difficulty which have 60-80% differentiate index are the most discriminating items(30). In the tutorial of December 2010, I gain an important information about the value of applying difficult items in the exam as these will encourage students towards excellent and to study to get more marks, so I think we must apply certain percentage of difficult items in the exam to drive learning of our students.Item inequality is determined by the difference of the percentage of correct response between two students group (top third and lower third) with discrimination ratio lie between +1 and -1 and acceptable index is in the range of -0.5 to +0.5(27). Good item has discrimination index closer to +1 as it can distinguish good student from poor one but if poor student can answer more item correctly than good students, this indicate negative discriminating item which should be excluded (30). Downing(36) emphasized that items o f MCQ test represented sample of all questions which could be tested, so for test with good internal consistency the test score should be an indicator for the student score on any other set with relevant items. Although our faculty recently develop assessment centre, we dont apply item analysis to any exam, So I think before applying it, we are in need to orient our faculty members about the importance of item analysis and how we use its statistical data to detect causes of low discriminations , discard poor question, and identify gaps in curriculum.Finally, we use written assessment to assess the major domain of cognition in its low level of knowledge recalling to its high level of knowledge application and problem solving, but as mentioned before, I think our written assessment has low reliability and validity as we use limited number of essay questions, and the percentage of essays marks are more than MCQ marks in our assessment, so we must apply using more objective tests of wel l structured MCQ, extended matching questions and short answer questions with more essays question especially modified essay, also we must determine the questions numbers according to their corresponding weight in the context and according to test blueprint, as these will facilitate sampling a broad range of relevant contents and constructs of our learning objectives.Although I finish my essay about written assessment, During studying this course I was interested in OSCE assessment and how apply it in our department, but I cant write about it as I dont have experience on applying it because we dont use it in clinical assessment and we use two long cases for applying report and ten short cases for radiological diagnosis for. Now I think we must apply using OSCE in our clinical assessment by using 10-20 post, some of them are procedure stations like carrying ultrasound examination under observation and other pictorial stations on analyzing radiologic image like conventional, CT, MRI images, and answering context-rich questions related to images.RefrencesShumway JM, Harden RM. AMEE guide No 25 The assessment of learning outcomes for competent and reflective physician. Med Teach.200325569-584.Wass V, Van der Vleuten C,Shatzer J,Jones R. Assessment of clinical competence. Lancet.2001357945-949.Dixon H.Candidates views of the MRCGP examination and its effect upon approaches to learning a questionnaire stu