Project Report – The Design, Implementation, Evaluation and Implications of a Blended Inst

The George Washington UniversityIn Support of the Requirements of EDUC234 – Computers in Education and Human DevelopmentAdam BoothProject Report – The Design, Implementation, Evaluation and Implications of a BlendedInstructional StrategyLesson Title: Excel 2002 (Office XP): Intermediate – Conditional FormattingOverview:This project was undertaken not only in support of the requirements of this project, but as an actual business study to determine the effectiveness of using Marriott’s eLearning tool, MarrLearn, as a way to impart foundational knowledge, coupled with a post-eLearning training laboratory which aimed at using an experienced trainer as a learning guide and Subject Matter Expert (SME). The results of the project demonstrated that blending educational technologies with instructor-led formats not only improved learning transfer, but yielded other benefits including decreased workload on the instructor, content uniformity, the incorporation of a variety of learning formats to suit diverse learning styles, and gave the instructor the ability to concentrate the laboratory work on the practical application of the subject.Lesson:There were two components to this one-hour Lesson:Part 1: Conditional Formatting: Online self-paced – 20 minutesPart 2:Conditional Formatting Lab: Instructor-led instruction with practical laboratory exercises - 40 minutes (lunch and learn)Part 1: The users worked through a self-paced, self-guided learning module that taught the foundational knowledge of conditional formatting. This knowledge and the small practical application available from within the tool, was used as the requisite knowledge needed to successfully complete part 2 of the course:Part 2: The learners worked with the IT instructor, Jessica Poawui, on a real Microsoft Excel spreadsheet used by the learners on a daily basis. This was an activity-based method and the users needed to draw upon the concepts they hade just learned to solve the problem:Activity:“You are a financial reporting consultant. Your supervisor would like you to email a spreadsheet of today’s Accounts Payable report highlighting the cells that contain the requirements below. Use your knowledge of conditional formatting to complete this task. The learners have 40 minutes to complete the task. The instructor will work through the task on the LCD projector with the learners providing guidance, feedback, troubleshooting and expertise.1.“All late invoices need to have the invoice number highlighted so I can easily find them.”2.“All paid invoices need to be labeled “Paid” in the Status drop-down list, and the invoice numbershould then be colored green so I know that the work has been done. “Target Audience, Entry Behaviors and Learner CharacteristicsThe actual audience as specified in the Project Plan did change from 5 IT managers with report-writing responsibilities to 6 accounting administrators due to a late change in the reporting process. This group works with Microsoft Excel on a regular basis, however, conditional formatting is not a feature of the tool that they used regularly. Entry behaviors were not measured by any kind of criterion-referenced test, only by a brief phone-based survey aimed at determining each user’s Knowledge, Skills and Attitudes (KSAs) before arriving for the class. It was determined that the 6 users in question were intermediate users of Microsoft Excel with no formal training in the tool or the conditional formatting subject matter. The learners came from the same group, have similar educational backgrounds (college educated), but have varying levels of computer proficiency. This was noted as one potential pitfall to success in theonline environment; most users were, however, comfortable with Microsoft products and online navigation. The project plan stated that the learners would be required to complete the eLearning course and assessment before coming to class, however, due to time constraints it was decided to complete the eLearning and facilitator pieces in the same session.Class Objectives:The aim of this class will be for the learners to create a reporting spreadsheet based on conditional formatting rules, which once published, will enable its readers to more easily distil and understand the complex metrics used to convey the health of the technical support operations of the business.Performance Objectives“By the end of this 60-minute module, by passing a practical exercise assessment utilizing and eLearning module and an Excel spreadsheet, you will demonstrate the ability in a production environment to:e Conditional Formatting to change the appearance of cell data based on pre-defined rules2.Create Conditional Formatting rules3.Troubleshoot corrupt cells and conditional formatting formulasTools: Hardware and SoftwareExcel Spreadsheet:This spreadsheet is an actual report used by the learners in the production environment, which is extracted as a download from a PeopleSoft Accounts Payable system. It consisted of one worksheet with key accounts payable financial data. Any invoice numbers that were paid were to display in green once the user changed the Status cell to “Paid” In addition, if a due date on the payment terms have been passed, the invoice number cell would turn red, thereby, prompting follow-up from the manager upon receipt of the report. Spreadsheets are useful as an instructional tool, although the computer performs the numerical computations, students must grasp the underlying concept of mathematical formulas (Finnegan and Sinatra, 1991).It was deemed appropriate to use a real example in order to increase the chances of successful transference of the concepts into the production environment, while demonstrating to managers a tangible level of proficiency, rather than provide a score from an assessment. Using a real report also reinforced a key adult learning concept of relevancy and the fusion constructivist-derived prior experience with the content. According to Dinmore (1997), “such a synthesis has the dual outcome of validating students’ experiences, and confirming in the learners’ eyes, the validity of theory because they can reconcile it with their own experience.” This was an important integration into the instructional design both as a way to increase learning retention and transference, but also as a way to increase buy-in from the learners into the blended learning mode being trialed.eLearning Module: MarrLearn – Conditional FormattingThe 15-minute online course was developed using Macromedia’s RoboDemo simulation building software, which enables the learner in this case to be able to experience and apply theoretical knowledge of MS Excel in a controlled environment. This conditional formatting module is one of 15 modules that make up and intermediate Microsoft Excel Course. It is planned to offer at least 5 other Microsoft Lunch and Learn courses using eLearning modules from this course.The tool is interactive, enabling the user to enter data into an interface that replicates the actual production environment of the tool (Excel 2002) in which they will be working after the class. The learner is able to control the environment, the pace of the lesson, and is able to move forward or back through the lesson at his or own pace. There is audio narration which explains the concepts to the learner. This is a true learner-centric training tool in terms of functionality, but the content and sequencing of materials represents a directed learning methodology, which is appropriate for its intended use as skill-building, pre-requisite development tool. The tool was very successful at imparting the foundational knowledgerequired for the practical Excel exercise for this class. With the aid of the facilitator who was on had to answer questions about the tool’s functionality as the user worked through the module, the response from the learners to the tool and training format became overwhelmingly in favor of using this tool as a first step in learning IT concepts.Teaching and Learning StrategiesThe instructional strategy employed in this lesson was designed to maximize the possibility that the learners could reach production-level proficiency in conditional formatting within a tight timeframe of one hour. Rather than teach the concepts as the learners worked through the practical exercise, it was deemed more beneficial to have the learners work through the eLearning with the instructor as an SME, and then reinforce the concepts with a real-world practical exercise. Given the duration was only one hour, the eLearning was integrated to ensure that the concepts could be imparted within a strict timeframe, thus ensuring that the instructor had more time to concentrate on the practicum. The spreadsheet was a real sheet given that “the more complete and accurate the worked examples presented during instruction, the more learning is accomplished quickly, and the more effectively learning transfers to work performance” (O’Neil, 2003).We also wanted to study the efficacy of moving from an instructor-centric learning environment to a learner-controlled environment. In this learning mode, the instructor was free to act as a subject matter expert when the students, who worked through the module at a different pace, encountered problems or questions. This was deemed to be a more effective use of the instructor’s time as she was able to devote time to individual students without disrupting the learning environment, instructor mentoring is vital to the learning process, allowing learners to ask questions when a particular concept eludes them (Price, 2001).Results and OutcomesThe results of the activity were overwhelmingly positive in terms of knowledge retention and transference to the production spreadsheet. In modern IT-driven organizations, it has become essential to tailor Donald Kirkpatrick’s four-level training evaluation system, reaction, learning, behavior, and result, to also incorporate measurement systems that are congruent with the needs of end-user IT training (Mahapatra, 2005). For the evaluation of the effectiveness of this training program, assessments and questionnaires were developed to incorporate features from Mahapatra’s End-user Training Evaluation Model. The essential difference between this model and Kirkpatrick’s is that given that IT-driven educational technology has become an integral part of organizational learning, there needs to be a measure if how well IT is integrated, used and accepted in the end-user training process (Mahapatra, 2005).This model evaluates the following features:1.The technology being used in the instruction, as the subject, or to support the instructionalstrategy.2.The reaction to the learning event (similar to Kirkpatrick’s Level 1)3.Skill acquisition4.Skill transferanizational effect (Mahapatra, 2005)A post-training survey was created to assess all of the five evaluation levels stated above. The survey was completed online using a WebSurveyor survey tool and send via email to the participants. As a questionnaire, this survey was qualitative in nature, yet provided an excellent measure of the learners’ perceptions of the courses.Level 1: Technology.The majority of the learners believed that the blended learning approach was the correct way to go, but given that they were not informed of the new format before class, there was some initial hesitation to the eLearning introductory course. However, once the users became comfortable with the tool, they believed that working through this course actively, rather than listening to an instructor and then applying the concepts, actually assisted in higher levels of retention and transference later in the class and back at each learner’s workstation.Through the survey, the learners also stated that they enjoyed using a real working copy of the Excel report as a learning tool. This, according to the learners, was the best way to ensure that the knowledge could be retained and applied beyond the classroom.Some negative feedback about the technology included an initial decrease in motivation to learn the Excel portion because of nervousness resulting from a lack of confidence that they would be able to master the eLearning tool to get the pre-requisite knowledge. Although the users were very good Excel users, they felt overwhelmed by ‘learning’ two entirely different software systems in the same session.Level 2: ReactionCaptured in the survey, reactions to the learning event were also highly positive. The students were all highly motivated to learn the concepts as they applied directly to their positions, and there was an expectation that these skills would help them become more productive. In addition, the blended format was accepted as an excellent way to learn theory, although, at least one of the learners would have liked the opportunity to complete the module before coming to class in order to use the short class time to leverage the expertise of the instructor.The following components were also surveyed (Mahapatra, 2005):1.Relevance of the course to the trainee’s job2.Satisfaction with the course content3.Quality of instruction4.Quality of eLearning tool navigation5.Overall Satisfaction with the training experienceLevel 3: Skill AcquisitionThe survey revealed that the learners felt that they all were able to learn and apply the skills taught in the class. Feedback from the instructor notes noted that there were six learners in the class and six different levels of proficiency by the end of the class as observed by the instructor. It is impossible to determine, without conducting a pre-test to level-set the group, whether or not there is a correlation between each learner’s ability in Excel before the class, or another factor such as learning styles, or even feeling connected with the instructor, the content, or the format.Skill acquisition from a quantitative standpoint was measured by a 5-question online assessment at the end of the initial eLearning section. All six of the learners scored 100% on this assessment. In addition to this assessment, skill acquisition was measured by the successful completion of the task attempted during the laboratory. Given that the learners worked through the tasks with the aid of the instructor and peers to correct mistakes (and it was not self-paced), it is difficult to measure actual performance; however, from a post-training review of each learner’s spreadsheet submitted for review, it was apparent that each of them was able to grasp and apply the concepts fairly easily.Level 4: Skill TransferThis is the most crucial measure of the success of a training program, given that an increase in performance, which by default is driven by the rate of skills transfer to production, is the reason that corporate training is conducted. The instructional goal is usually derived from the gap in current to desired states, and measuring the rate of skills transfer is one way to tell whether or not the desired state has been reached in a production environment. To gauge the effectiveness of the training program, we asked them to copy the trainer on the email that contained the next report to be submitted to each learner’s manager. Each user was able to download the spreadsheet from the PeopleSoft application and apply the conditional formatting rules to the appropriate cells. This meant that from our perspective there was a 100% rate of skill transfer to the production environment.Level 5: Organizational effectAlthough it would have been a more robust measure had a comparison between the time it took to do the work in class between the time in the production environment, it was not really necessary to measure a rise in efficiency as it was a brand new skill, and one of several Excel skills needed to complete the sheet correctly. Efficiency and organizational effect could be measured in other ways, however, including flow-through efficiencies i.e. the decrease in time it takes for the manager reviewing the sheet to locate the information contained in the formatted cells against the old process.Qualitatively, the learners felt that there was a marked increase in the quality of the sheet, which would be positive organizational effects moving forward, which could save the organization money moving forward.Return on Investment (ROI) from the user’s perspective can be easily calculated based on the following assumptions:1: Each Manager reduced the time it took to find information and approve the sheet from 5 minutes to 4 minutes. At $50 per hour, the total savings in $ per manager per sheet (50 sheets per year) is the following:1.Old Process: (50/60) x 5 = $4.21 per sheet2.New Process: (50/60) x 4= $3.33 per sheet.3.Saving per sheet: $0.88ROI4.6(managers) x $0.88 = $5.28 savings per week5.$5.28 x 50 = $264.00 savings per fiscal year6.Cost of class: $40 per student7.Total cost of class: $240.00ROI = 45 weeksAlthough this will take 45 weeks for the learners to pay off, this calculation assumes that the level of efficiency will stay the same despite an assumed rise in actual ability as the users gain more experience.Conclusions and Lessons LearnedThe instructor and the students agreed that a blended format is most certainly a viable option for lab-based training where the class time is restricted to around one hour. Based on feedback it would have been more effective to have had the learners take the eLearning portion before coming to class if appropriate FAQ communications accompanied the email link to the eLearning courseware. This would have provided an additional 20 minutes of lab time that the instructor could have utilized to teach either more advanced skills, or conducted a Q&A session to bolster the students’ knowledge of the tools and associated business processes.The eLearning tool, MarrLearn, is increasingly being integrated into Marriott’s training philosophy, and this was an excellent test case in which to test a blended learning solution. One of the excellent features of the eLearning suite in general, is that being developed using scratch pages in MacroMedia RoboDemo (now Captivate), the learners are able to interact on some level with the software, thus leading to increased levels of knowledge retention. In addition, the revised suite of learning tools has a more multimedia aspect combining, demos, guided and unguided practical exercises, audio narration, transcripts, tips and tricks, intelligent error correction and a host of other features that enable learners with different learning styles to interact with and control the tool and sequencing, yet arrive at the same learning outcomes. The decision has now been taken within the learning group to create specialized curriculum paths for the Lunch and Learn series. Given that there are more than 300 IT courses within this library, with each course having as many as 15 modules, the opportunities to leverage the content to expand the range of instructor-led offerings is enormous. This could be an excellent source of revenue moving forward for the department.In terms of the development of the lesson plan, the instructor is a highly experienced Microsoft Office trainer, so I was able to leverage her talents as an SME to create a lesson plan that was consistent with the eLearning piece, yet was able to seamlessly integrate those concepts with the learner goals and information on the worksheet. It took around 3 hours to develop the plan, which mostly consisted of moving through the eLearning module and summarizing the content. This summary was then applied to the worksheet needs and a ‘superimposed’ set of learning items was developed. Taking learner needs also into consideration, a final set of learning objectives was developed. This was a fairly easy plan to create and implement, however, had the learners completed the online learning before class, it could have been even more effective.ReferencesDinmore, I. (1997). Interdisciplinary and integrative learning: An imperative for adult education. Education. Spring 1997.Vol.117, Iss. 3; pg. 452, 16 pgs. Retrieved March 26, 2005. From/pqdweb?index=0&did=11496792&SrchMode=1&sid=3&Fmt= 4&VInst=PROD&VType=PQD&RQT=309&VName=PQD&TS=1114379775&clientId=31812#fulltext.Finnegan, R. & Sinatra, R. (1991). Interactive Computer-Assisted Instruction with Adults. Journal of Reading. Oct 1991. Vol. 35, Iss. 2; p. 108. Retrieved April 23, 2005. From,/pqdweb?index=0&did=1873929&SrchMode=1&sid=5&Fmt=6 &VInst=PROD&VType=PQD&RQT=309&VName=PQD&TS=1114380010&clientId=31812.Mahapatra, R, (2005) Evaluating End-user Training Programs. Retrieved April 23, 2005. From, /pqdweb?index=13&did=792884651&SrchMode=1&sid=3&Fmt =6&VInst=PROD&VType=PQD&RQT=309&VName=PQD&TS=1114374175&clientId=31812 O’Neil, H.F. (2003). What Works in Distance Learning. University of Southern California/CREEST. Retrieved April 22, 2005. From,/index.cfm?fuseaction=rcdetails&tab=2&libid=568&bc=true&keywords=&filterid=3 7&applyto=libTitle,libAuthor,contentText.Price, D. & von Schlag, P. (2001). Bringing Classroom Curriculum Up to E-Speed. American Society of Training and Development. Retrieved April 22, 2005. From,/2001/sep2001/price.html.。