[comment]: # (Compile this presentation with the command below) [comment]: # (mdslides index.md && mv index/index.html .) [comment]: # (The list of themes is at https://revealjs.com/themes/) [comment]: # (The list of code themes is at https://highlightjs.org/) [comment]: # (Pass optional settings to reveal.js:) [comment]: # (markdown: { smartypants: true }) [comment]: # (Other settings are documented at https://revealjs.com/config/) #### Reconciling DORA Metrics for Educational Projects on GitHub ---------- Kevin Buffardi and Aviral Kumar Srivastava _{California State University, Chico} <img src="qr-dora.png" alt="QR code for this presentation" width="25%"> </img> _{Back to [LearnByFailure.com](https://learnbyfailure.com/)}

#### Motivation * **D**ev**O**ps **R**esearch and **A**ssessment (DORA) metrics * Industry-adopted indicators of software delivery * Software engineering courses increasingly use professional tools like GitHub * Student projects differ from industry projects * Instructor supervision of 10+ different projects at a time * Different products, each with distinct tech stack * **Can DORA metrics be meaningfully automated for educational GitHub projects?**

#### What Are DORA Metrics? * **Deployment Frequency**: how often teams ship changes * **Lead Time for Changes**: time from code change to production * **Change Failure Rate**: proportion of deployments causing production incidents * **Mean Time to Recovery**: time needed to restore service after failure

#### Research Questions * **RQ1:** How can DORA metrics be measured in educational GitHub projects? * **RQ2:** Are DORA metrics associated with other measurements of teamwork from software engineering education research?

#### Study Context * 10 semesters of software engineering course at Chico State (n=465) * Course introduces version control, containerization, CI/CD, issue tracking, code review, MVC architecture, testing, anti-patterns * Semester-long Agile team projects (94 teams of ~5) * Teams build novel "Tech Startup" applications from scratch * GitHub used for project boards, version control, issues, pull requests, code reviews, and comments

#### RQ1: Operationalizing DORA: Deployment Frequency <img src="df.png" width="60%" alt="Deployment Frequency is number of deployments over a period of time" /> * DF = Deployments / time

#### RQ1: Operationalizing DORA: Lead Time for Changes <img src="lt.png" width="60%" alt="Lead Time is the commit-to-deployment elapsed time" /> * LT = time_deployment - time_commit

#### RQ1: Operationalizing DORA: Change Failure Rate <img src="cfr.png" width="60%" alt="Change Failure Rate is the rate of deployments that cause faults in production" /> * CFR = deployment_fault / deployment_total

#### RQ1: Operationalizing DORA: Mean Time to Recovery <img src="mttr.png" width="60%" alt="Mean Time to Recovery is the average time to restore service after production fault" /> * MTTR = Average of (time_recovery-time_failure)

#### RQ1: Operationalizing DORA: Barriers * **Deployment** is inconsistent between projects * Proxy: Merge of Pull Request (w/ feature branches) * Convention for `deploy` GitHub Workflow * Future: Require [GitHub Releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) * No native, direct signal of **Production faults** on GitHub * Future: conventions for labeling issues

#### Proxy Results For 79 teams with usable pull-request data: | Metric | Mean | Standard Deviation | |---|---:|---:| | Deployment Frequency | 1.99 PRs/week | 1.25 | | Lead Time | 1.37 days | 1.64 |

#### RQ2: Teamwork Measures Compared with DORA Proxies The study compared DF and LT proxies with: * **Hoover Index**: inequality in artifact contributions * **Cohesiveness**: self-reported team movement toward a unified goal * **Median Consistency**: regular participation by the median contributor

#### Descriptive Team Metrics | Team Metric | Mean | Standard Deviation | |---|---:|---:| | Deployment Frequency | 1.99 | 1.25 | | Lead Time for Changes | 1.37 | 1.64 | | Hoover Index | 0.32 | 0.12 | | Cohesiveness | 3.76 | 0.41 | | Median Consistency | 0.50 | 0.16 |

#### Predicting Equality or Cohesiveness * DF and LT <u>did not</u> significantly predict contribution equality * DF and LT <u>did not</u> significantly predict self-reported cohesiveness

#### DF Predicted Median Consistency Teams with more frequent deploymentment also showed more consistent participation by the median contributor. * DF was a significant positive predictor of Median Consistency * (F(2,76)=7.96, p<.001, Adjusted R^2 =.15) <small>More frequent integration may reflect steadier week-to-week participation across the team.</small>

#### Practical Framework for Courses To make DORA metrics more reliable in educational GitHub projects: 1. **Require deployment workflows** 2. **Establish consistent pull request practices** 3. **Require production failure issues**

#### Tip 1: Require a Deployment Workflow * Require every team to use GitHub Actions * Standardize the deployment workflow name, such as `deploy` * Successful `deploy` workflow as signal for deployment * [GitHub Releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) may be most reliable cross-project signal

#### Tip 2: Establish consistent pull request practices * Require pull requests for changes to `main` * Require formal code review approval * Prefer merge or rebase over squash merges * Link pull requests to issues using keywords such as `fix`, `close`, or `resolve` * Vice versa for identifying PR that caused fault

#### Tip 3: Require Production Failure Issues * Require students to create an issue when a production incident occurs * Apply a [standardized label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels) such as `production failure`

#### Potential Benefits #### For instructors * Real-time monitoring across many teams * Earlier identification of struggling teams * Better feedback grounded in project activity #### For students * Exposure to industry-aligned delivery metrics * Reflection on team workflow and delivery habits * Practice using GitHub metadata as actionable feedback

#### Future Work * Adopt standardized deployment and incident practices in future courses * Build a centralized real-time DORA dashboard * Explore DORA alongside peer evaluations, instructor assessments, and contribution counts * Study whether DORA metrics support formative feedback and project interventions

#### Reconciling DORA Metrics for Educational Projects on GitHub <small>Kevin Buffardi and Aviral Kumar Srivastava</small> <small>This work was supported in part by the National Science Foundation, award 2337271.</small> <small>Project data and materials: <a href="https://github.com/kbuffardi/DORA-CSEET-2026" target="_blank">github.com/kbuffardi/DORA-CSEET-2026</a></small> <img src="qr-dora.png" alt="QR code for this presentation" width="25%"> </img> _{Back to [LearnByFailure.com](https://learnbyfailure.com/)}