Log in
Search

Evaluating e-learning implementation

  • Public
Started by Keir Whipp 04 Jun 2015 12:41pm () Replies (14)

HI 
I'd like to ask for any references (academic articles especially) on the evaluation of e-learning implementations in secondary schools.
Any models or theories on how to evaluate an e-learning innovation  much appreciated.

Kind regards

Keir

Replies

  • Hamish McLean (View all users posts) 04 Jun 2015 12:58pm ()

    Hi Keir,

     

    Wouldn't this be any results you have for your classes?  What I mean to say is that any results that you have, will be evidence...  This is because eLearning is just 'learning'.  Almost everything we do is on a digital medium, so its just learning.

  • Keir Whipp (View all users posts) 04 Jun 2015 5:02pm ()

    Thanks, Hamish

    You're absolutely right.  We evaluate the effectiveness of our teaching by looking at the student learning outcomes.  I should have made myself clearer.  I want to take a more general view and look at models or theories on how to evaluate the implementation of innovation in a secondary school.  An evaluation of the way a major change was implemented - in this case the introduction of BYOD and e-learning pedagogy.  How it was done; what worked, what didn't work, what could be done differently.

    I'm interested to know what has been written about this specific to e-learning.  Particularly stuff written by academics, researchers, leaders and teachers.

    Cheers

    Keir

  • Hamish McLean (View all users posts) 05 Jun 2015 8:46am ()

    Tessa's given you some good content.

    I'm still a little hesitant at looking at eLearning as different, and specific outcomes of using technology as opposed to not... perhaps I'm being facetious, not trying to be...  but we didn't have research why using a pen was good.  

    We use tools because it makes collaboration easier and more effective.  It makes sharing and reflecting more powerful.    There are loads of articles on eLearning, but I really hate that term.  It supposes that learning has changed and that we use technology only during an eLearning focus for that subject... .   Learning hasn't changed... its just that teachers/schools/BOT's are beginning to realise that 'they' need change their pedagogy to prepare students for a different future.  A future that has a vastly different job market... where creativity is the only thing that mass production and robots can't replicate... 

    The biggest learning and change for teachers comes when they see what other schools are doing.  Research only show so much, and doesn't give perspective for what your specific school could or should change.  

  • Tessa Gray (View all users posts) 05 Jun 2015 9:43am ()

    Hamish, you're on to a good debate here. Not surprising other community members have also discussed this here, How do you know if ICT is making a difference - and does it matter ... 

    The late, great Vince Ham used to say 'it's very hard to track or monitor the effectiveness of ICTs/e-tools when there are so many variables - #1 being effective teaching pedagogy.' Isolating the effectiveness of e-learning would probably mean analysing the difference between control groups on a single task. IE: Mindmappng - some students can do this using paper, coloured pencils etc, others can use mindmapping tools like https://bubbl.us/. Could we then collect data to show one tool was more powerful than the other and that the learning outcomes were better/higher/more advanced? What were we looking for anyway? If it was collaboration, then the web based collaborative tool would have the advantage surely?

    In terms of tracking specific learning outcomes it can get tricky. Some people use rubrics or matrixes to track ICT capabilities. Jury's still out for me on that, which is why I created this page: Alternatives to ICT skills checklists.

    What do you reckon? 

     

     

  • Keir Whipp (View all users posts) 05 Jun 2015 9:52am ()

    Hi Hamish,

    You’re getting me thinking more deeply about this, which is great.  Again, I’m full of agreement with you.  Although I do think we might be at cross-purposes because I do not so much want to evaluate the implementation of e-learning at my school, rather, I want to know about evaluation of change implementation more generally.  

    However, I do wonder about the pen.  I'm sure there was a lot discussion around the time of its initial implementation as a pedagogical tool.  I do believe the ink pen took on a symbol of status and students would graduate to its use (in classrooms where such technology was permitted).  And as for learning changing, well, I'm not a neuroscientist, but from what I've been reading the digital world is having an alarming effect on the wiring of the brain, and as a result, digital-age learners learn differently.  There is a lot of research going on in this area (see Dryden, G. and Vos, J. (2009), Medina, J. (2008), Small, G. and Vorgon, G. (2008)). 

    So if we understand that learning is changing, perhaps it is our best-practise pedagogy that doesn’t need to change.  All that may need to change, as I think you would agree, is that we include the use of digital tools in our best-practise.

    I do agree we need to change the minds of non-digital leaders and non-digital BOTs.  And I do understand that e-learning IS learning, but at this stage I think we still have to differentiate with terminology because there is a HUGE digital divide in teaching amongst professionals, for example, the digital teachers vs. the non-digital teachers (and the non-digital teachers vastly outnumber the digital). 

    So, I think it is important we keep making this distinction until the balance changes; otherwise it’ll be too easy to hide behind the term ‘learning’ when we really want those teachers to be developing digital-based learning.  

    Regards

    Keir

     

  • Hamish McLean (View all users posts) 05 Jun 2015 10:20am ()

    Hey Keir,

     

    Yes that is a good point.  I had a post last year sometime commenting on a teacher that wanted a framework to assess ICT.  I posted a similar comment as I have today, because I was afraid that teachers would teach 'to' this to be at a 'standard'.  I do see that unless we have this as a start, teacher just wont change.  But the more important thing I believe, is why they need to change... They need to see the benefit of what changing their pedagogy means for the learner.  Otherwise, they are just in a teachers retirement home collecting a pay check.  (hope that doesn't offend anyone... devil haha)

    unfortunately like you say there is a massive divide between teachers that use digital mediums and those that don't.  

    "However, I do wonder about the pen.  I'm sure there was a lot discussion around the time of its initial implementation as a pedagogical tool."

    Funnily I totally agree.  I did some research about this for a presentation about 5 years ago, this was to my BOT as they didn't want 'digital learning'.  

    - The ball point pen was introduced to the U.S. market.  10,000 were sold at the launch at Gimbel's department store in New York on October 29th 1945.  It sold out on the first day.  Remind you of anything??   iPad??

    digital-age learners learn differently.  Yes they totally do.  I read an article about how digital learning is bad for the brain because they learner doesn't use handwriting which effects a part of the brain that ... umm does something, i forget what it said.  But the article wasn't taking into account using fingers and touch screens to manipulate.  using augmentative reality to learn.  It was just speaking about typing vs' hand writing.  

    Like Tessa's comment:

    "tools like https://bubbl.us/Could we then collect data to show one tool was more powerful than the other and that the learning outcomes were better/higher/more advanced? What were we looking for anyway? If it was collaboration, then the web based collaborative tool would have the advantage surely?"

    Its another point to think about.  In this instant I believe what ever tool the student think will be more effective for them, depending on their passions and agency.  Some may want to be more artistic about how they present it...  others may want to be able to edit it very easily and make changes, so a digital map will be better for them...  

  • Keir Whipp (View all users posts) 05 Jun 2015 11:06am ()

    Yes.  The 'why' is so important.  I really like the idea of teachers having to understand their students better as students become more 'alien' - (refugees from a digital world?)  Of course, when the profession becomes saturated with a generation of digital-age teachers, things will change again, but as you say, we need to be responsive to the needs of these students today, and not have to 'wait for the pension' so to speak.

    I think evaluating tools (which is a different research to mine) has its use, but again,as you point out, there will be no one tool for the job (or one size to fit all).  Like a craftsman, a student can select a tool they prefer, which meets their needs, competency and demands of the task they want to achieve.

  • Tessa Gray (View all users posts) 05 Jun 2015 10:15am ()

    Hi Keir, while you're looking for more general view and look at models or theories on how to evaluate the implementation of innovation in a secondary school, here's a few more links that might help:

    'Leadership in digital technology: The challenge of decision making' (A Weijermars, 2012)

    BLURB: Masters Thesis – Master of Educational Leadership and Management, Unitec Institute of Technology

    This research investigated the perceptions of decision-making in the use of digital technologies by three secondary schools, leaders and teachers. In the use of digital technology teachers in these schools understood it to mean how data was captured, stored, manipulated, produced and distributed digitally as mass media. In the context of education digital technological devices offered a host of opportunities in the teachers’ repertoire of instructional tools. With so many innovative web-based ICT resources and DT devices being used in secondary schools, subject specific departments were literally free to pick and choose from a range of digital resources they saw fit for class instruction that enabled students to be captivated and engaged in their learning. For leaders charged with sanctioning budget requests for both ICT and digital technology resources a level of expertise, knowledge and jurisdiction in how these resources supported classroom instruction needed to be examined.

    The following may be applicable to secondary too:

    Effective Digital Learning Pedagogies & Environments

    BLURB: This is a site that provides summaries and resources related to Manaiakalani research and evaluation conducted by the Woolf Fisher Research Centre in partnership with the Manaiakalani schools and the Manaiakalani Education Trust . Useful to LCN Leaders, Teachers and Facilitators.

    Manaiakalani project is know for increasing students' access to digital technologies through BYOD (Bring/Buy your own device).

    Disrupting the boundaries of teaching and learning: How digital devices became a resource for transformative change in a time of crisis (Full research report) Te Toi Tupu, CORE Education, Greater Christchurch Schools Network, MoE, Louise Taylor, Tara Fagan, Merryn Dunmill

    BLURB: As a result of the Christchurch earthquakes, schools were facing closures, mergers and infrastructural changes.

    The report outlines how the project came about, who was involved, how the devices were integrated into schools, and the learning and change that occurred for teachers, students and the community as a result.

    Educational leaders have some credible research projects that relate directly to leadership and ICT. These can be found in this tag search.

    There's more in this original post here: Expanding our digital tools.

  • Tessa Gray (View all users posts) 04 Jun 2015 2:41pm ()

    Hi Keir, try these e-learning readings and research from Enabling e-Learning in TKI. http://elearning.tki.org.nz/Research-and-readings which has some references to e-learning and improved outcomes for students IE >>>

    IE: Digital readers at age 15: An overview of the PISA 2009 Electronic Reading Assessment
    The Programme for International Student Assessment (PISA) is an international study that assesses how well countries are preparing their 15-year-old students to meet real-life opportunities and challenges. In 2009, the main PISA study offered countries the option of assessing some of the participating students in reading using a computer based assessment — the Electronic Reading Assessment (ERA). The average (mean) digital reading literacy score for New Zealand 15-year-olds was significantly higher than the print reading literacy score. Compared with other high-performing countries or economies participating in the ERA, New Zealand had a relatively large proportion of students who demonstrated very advanced digital reading skills as well as a relatively high proportion of students who demonstrated poor skills.
    Author: Sarah Kirkham
    Published: June 2011

    and

    PISA 2009 results: Students on line: Digital technologies and performance (volume VI)
    The Programme for International Student Assessment (PISA) is an international study. Students from 19 countries were tested on their ability to navigate and evaluate information on line. Key curriculum areas focused on in the report are reading, mathematics, and science. Findings show that, in every participating country, girls outperform boys in reading skills, boys outperform girls in mathematics, and in science gender differences tend to be small. Countries of similar prosperity can produce very different educational results. 
    Author: published on the responsibility of the secretary-general of the OECD
    Published: June 2011 

    as well as

    There is, a growing body of evidence to support the view that digital technologies have the potential to improve student outcomes and to enrich, if not transform, the learning experience of children. This report makes links to research demonstrating the increasing evidence of the positive impact of digital technologies in areas such as motivation, engagement, efficacy, and interaction.
    Author: Ministry of Education
    Published: July 2012

    You might also be interested in similar threaded conversation (but not necessarily specific to secondary contexts) >>>

    How do you know if ICT is making a difference - and does it matter ... with references to The Impact of Digital Technology on Learning" via DERN (the Digital Education Research Network).

    Hope that helps smiley

  • Keir Whipp (View all users posts) 04 Jun 2015 5:11pm ()

    Thank you, Tessa!  A brilliant help :)  Plenty of learning for me.

    Cheers

    Keir

  • Greg Carroll (View all users posts) 05 Jun 2015 11:11am ()

    ..... just responding to the (digital) pen thing.

    This is fascinating about the affordances of a digital pen, even over-and-above a pen and paper.  Watch from about 1hr in to the end for the best bits.

    cheers

    Greg

  • Hamish McLean (View all users posts) 05 Jun 2015 11:31am ()

    Thanks Greg,

    Yeah interesting.  Not necessarily for this information, but certainly for documentaries, there are facts and figures that can be presented to prove points.  

    I watched a doco called 'forks over knives'.  The main idea was that plant based protein is better than animal based protein.  The doco used data from Germany in 1940 opposed to 1946.  The use of animal protein was decreases during this period due to Germany having to export 90% of its meat as part of its surrender during WWII.  They then compared the frequency of cancer diagnosis.  Was a massive decrease.  They then said it was this decrease in consumption of animal protein that was the reason that cancer diagnosis decreased.  However they didn't comment on anything else.  Just the animal protein.  So this wasn't fair testing at all. But for the doco it helped prove a point.

    I find this especially true for those that want to find excuses for not using digial tools... Haters gonna hate!  :)

    ptt.jpg

     

     

  • Tessa Gray (View all users posts) 17 May 2017 9:26am ()

    This is an older thread that's still relevant to us today where educators are asking, (and so they should) Where's the research to support digital technologies can have a positive impact on learning? 

    New research on Digital Technology in Schools released May 16th, 2017

    Eight-in-ten principals report that digital technologies are having a positive impact on student achievement, according to the 2016/17 ‘Digital Technologies in Schools’ survey, prepared by Research New Zealand for the 20/20 Trust. The survey was shorter and simpler than the 2014 survey and it was again open to all New Zealand schools. A total of 464 schools completed the survey.

    Summary: Digital technologies are having a positive impact on student achievement (4MB, pdf) – includes the 2017 report card, enabling schools to compare their position against the national average.

    Full report: Digital Technologies in Schools 2016-17 (3MB, pdf)

    Really exciting to read,

    Positively, more than three-quarters of principals reported that teachers in their school view integration of digital technologies in their teaching as being relevant, and more than half of principals felt that their school’s teachers have a good knowledge about what to do to achieve good integration of digital technologies in the teaching and learning process.

    This aligns to the research that implementing technologies effectively in the classroom really comes down to teacher beliefs

     

    Does this research match up with what's happening in your school/kura?

Join this group to contribute to discussions.