b85169f4e7
- PhD defense slides (defense.key, Nov 2018) → phd_defense/ - Master's defense on MOOC peer evaluation (Dec 2014) - ENGI 600 data-driven program repair (Apr 2015) - COMP 600 data-driven program completion (Fall 2015, Spring 2016) - COMP 600 Program Splicing presentation + feedback + response (Spring 2018) - Program Splicing slides in .key and .pdf formats (Spring 2018) Each file has a .md transcription with academic frontmatter. Skipped www2015.pdf (duplicate of existing www15.zip) and syncthing conflict copy.
103 lines
3.1 KiB
Markdown
103 lines
3.1 KiB
Markdown
---
|
||
category: academic
|
||
type: academic
|
||
person: Yanxin Lu
|
||
date: 2014-12
|
||
source: master_defense_2014.pptx
|
||
---
|
||
|
||
# Master's Thesis Defense: Improving Peer Evaluation Quality in MOOCs
|
||
|
||
Yanxin Lu, December 2014. 40 slides.
|
||
|
||
## Slide 2: Title
|
||
Improving Peer Evaluation Quality in MOOCs — Yanxin Lu, December 2014
|
||
|
||
## Slide 3–4: Summary
|
||
- Motivations and Problems
|
||
- Experiment
|
||
- Statistical Analysis
|
||
- Results
|
||
- Conclusion
|
||
|
||
## Slide 5: What is MOOC?
|
||
|
||
## Slide 6: Intro to Interactive Programming in Python
|
||
- Coursera course, 120,000 enrolled, 7,500 completed
|
||
|
||
## Slide 7–8: Example Assignments
|
||
- Stopwatch
|
||
- Memory game
|
||
|
||
## Slide 9: Grading Rubric for Stopwatch
|
||
- 1 pt: Program successfully opens a frame with the stopwatch stopped
|
||
- 2 pts: Program correctly draws number of successful stops at whole second vs total stops
|
||
|
||
## Slide 10: Peer Grading
|
||
- Example scores: 1, 9, 9, 9, 10 → Score = 9
|
||
|
||
## Slide 11: Quality is Highly Variable
|
||
- Lack of effort
|
||
- Small bugs require more effort
|
||
|
||
## Slide 12: Solution
|
||
A web application where students can:
|
||
- Look at other peer evaluations
|
||
- Grade other peer evaluations
|
||
|
||
## Slide 13: Findings
|
||
- Grading evaluation has the strongest effect
|
||
- The knowledge that one's own peer evaluation will be examined does not
|
||
- Strong effect on peer evaluation quality simply because students know they are being studied
|
||
|
||
## Slide 15: Experiment Summary
|
||
- Sign up → Stopwatch → Memory
|
||
|
||
## Slide 16: Sign up
|
||
- Web consent form, three groups, prize
|
||
- Nothing about specific study goals or what was being measured
|
||
- 3,015 students
|
||
|
||
## Slide 17: Three Groups
|
||
- G1: Full treatment, grading + viewing
|
||
- G2: Only viewing
|
||
- G3: Control group
|
||
- Size ratio G1:G2:G3 = 8:1:1
|
||
|
||
## Slides 18–24: Experiment Phases
|
||
- Submission Phase: Submit programs before deadline
|
||
- Evaluation Phase: 1 self evaluation + 5 peer evaluations per rubric item (score + optional comment)
|
||
- Grading Evaluation Phase (G1): Web app, per evaluation × rubric item → Good/Neutral/Bad
|
||
- Viewing Phase (G1, G2): See number of good/neutral/bad ratings and their own evaluation
|
||
|
||
## Slide 25: Statistics
|
||
- Most evaluations are graded three times
|
||
|
||
## Slide 27: Goal
|
||
- Whether G1 does better grading compared to G2, G3 or both
|
||
- Measuring quality: correct scores, comment length
|
||
- Reject a set of null hypotheses
|
||
|
||
## Slide 28: Bootstrapping
|
||
- Simulation-based method using resampling with replacement
|
||
- Statistically significant: p-value <= 0.05
|
||
|
||
## Slide 30: Terms
|
||
- Good programs: correct (machine grader verified)
|
||
- Bad programs: incorrect
|
||
- Bad job: incorrect grade OR no comment
|
||
- Really bad job: incorrect grade AND no comment
|
||
|
||
## Slides 31–38: Results
|
||
Hypothesis tests on comment length, "bad job" fraction, and "really bad job" fraction across groups on good and bad programs.
|
||
|
||
## Slide 39: Findings
|
||
- Grading evaluation has the strongest positive effect
|
||
- The knowledge that one's own peer evaluation will be examined does not
|
||
- Strong Hawthorne effect: improvement simply from knowing they are being studied
|
||
|
||
## Slide 40: Conclusion
|
||
- A web application for peer evaluation assessment
|
||
- Study has positive effect on quality of peer evaluations
|
||
- Implications beyond peer evaluations
|