Denae Ford , profile picture
Uploaded byDenae Ford
PDF, PPTX530 views

Beyond the Code Itself: How Programmers Really Look at Pull Requests

The document explores how programmers review pull requests, focusing on their visual attention and decision-making processes through eye-tracking studies. It aims to identify discrepancies between perceived and actual areas of focus, as well as strategies employed to manage personal identity signals during reviews. Participants included 42 individuals familiar with GitHub, and the study analyzed various contributing factors such as code, social, and technical signals influencing their decisions.

Embed presentation

Download as PDF, PPTX
Beyond the Code Itself: How Programmers Really Look at Pull Requests Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, Chris ParninDenaeFordRobin North Carolina State University + Eindhoven University Of Technology
PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull Request
PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is Merged
PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is MergedDiscuss + Review
CODE CONTRIBUTIONS BASED ON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
CODE CONTRIBUTIONS BASED ON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
ABBY’S GITHUB PROFILE PAGE 4
ABBY’S GITHUB PROFILE PAGE 4
5ABBY’S GITHUB PROFILE PAGE
6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 7 RQ2 RQ1 RQ3 RESEARCH QUESTIONS
8METHODOLOGY PROTOCOL Pre Experiment Survey Have you ever accepted or rejected or submitted a pull request? * Please rate your programming experience on a scale from 1-10. * How much experience do you have performing integration tasks? * Do you have a GitHub account? *
9 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session 9 METHODOLOGY
10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
11 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request 11 Reasonable Code Snippet: Unreasonable Code Snippet: METHODOLOGY
12 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Post Experiment Survey Review Profile Page + Pull Request What elements of the displayed profile or pull request did you consider when making your decision? * What strategies do you use to decide to share this content? Does it vary across each community? * METHODOLOGY
13 PARTICIPANTS METHODOLOGY
Recruited 42 Participants 13 PARTICIPANTS METHODOLOGY
Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
All familiar with submitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
All familiar with submitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY Gender Quantity Age Range Country of Origin Minority Not Minority Men 30 22-33 24 India, 1 Nepal, 1 USA 2 23 Women 12 23-29 2 China, 5 India, 1 Iran, 3 USA 1 10
How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS
How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience
How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses
How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses Platform Specific Strategies
Social Signals Code Signals Technical Signals
(AI) (DN) (FF) (RE) (HM) (CA) (RS) (BC) (AC) (UD) (PT) (SD) (RP) (CD) (TM) Social Signals Code Signals Technical Signals
RESULTS
TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Code Signals
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject 5 out of 20 participants spent 48% to 62% of their time fixating on technical signals and an above average time on social signals (17% to 31%) Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject — = no decision More details in paper: http://bit.ly/BeyondCode
FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
45% reported using information related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
45% reported using information related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest 1 participant reported using the submitter’s profile image
RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy
RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)
RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26) “such familiarity gives a feeling of trust.” (S15)
DISCUSSION + IMPLICATIONS 23 All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
DISCUSSION + IMPLICATIONS 23 GitHub profiles make social + human capital explicit. How does giving users control over their identity influence participation? All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
Summary Paper: http://bit.ly/BeyondCode DenaeFordRobin (BC) (AC) (UD) (PT) (SD) (RP) (CD) (TM) dford3@ncsu.edu We observe that both social and technical aspects are being taken into consideration when deciding upon pull request acceptance. Moreover, many more social aspects are being considered during the experiment than reported during the post-experiment survey Future work will study how the execution of concealing or amplifying these signals affect developers across the identity spectrum and development experiences at scale. Artifact: http://bit.ly/BeyondCodeArtifacts

More Related Content

PDF
Technology - It's position and impact on Surveying 2020 Report - May 2020
byAnthony Walker
 
PPT
The impact of attribute preferences on adoption timing of hotel distribution ...
byInternational Federation for Information Technologies in Travel and Tourism (IFITT)
 
PPTX
Towards Automated Supports for Code Reviews using Reviewer Recommendation and...
byMasud Rahman
 
PDF
Final Algos
byAnirudh Mallem
 
PDF
Supporting Newcomers in Software Development Projects
bySebastiano Panichella
 
PDF
How to Ask for Technical Help? Evidence-based Guidelines for Writing Question...
byFabio Calefato
 
PPTX
CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Te...
byMasud Rahman
 
PDF
Open Source At Facebook and Beyond
byAll Things Open
 
Technology - It's position and impact on Surveying 2020 Report - May 2020
byAnthony Walker
 
The impact of attribute preferences on adoption timing of hotel distribution ...
byInternational Federation for Information Technologies in Travel and Tourism (IFITT)
 
Towards Automated Supports for Code Reviews using Reviewer Recommendation and...
byMasud Rahman
 
Final Algos
byAnirudh Mallem
 
Supporting Newcomers in Software Development Projects
bySebastiano Panichella
 
How to Ask for Technical Help? Evidence-based Guidelines for Writing Question...
byFabio Calefato
 
CORRECT: Code Reviewer Recommendation in GitHub Based on Cross-Project and Te...
byMasud Rahman
 
Open Source At Facebook and Beyond
byAll Things Open
 

Similar to Beyond the Code Itself: How Programmers Really Look at Pull Requests

PDF
Dealing with contributor overload - FOSS Backstage
byHolden Karau
 
PPTX
Automatic Identification of Informative Code in Stack Overflow Posts
byPreetha Chatterjee
 
PPT
We builit it
bywebuiltit
 
PDF
Opening up Open Source
byDerek Buitenhuis
 
PPTX
Growing Developer Community One Pull Request At a Time
byAlex Bulankou
 
PPTX
Leveraging the Crowd: Supporting Newcomers to Build an OSS Community
byMarco Aurelio Gerosa
 
PDF
Is Empirical Enough?
byPer Runeson
 
PDF
淺談 Startup 公司的軟體開發流程 v2
byWen-Tien Chang
 
PDF
Agile startup company management and operation
byJiang Zhu
 
PDF
Agile for digital library projects
byNaomi Dushay
 
PPT
When Users Becom Collaborators: Towards Continuous and Context-Aware User Input
byHans-Joerg Happel
 
PDF
How to Run a Successful Open Source Java EE Project
byphilipdurbin
 
PDF
[3.6] Beyond Data Sharing - Pieter van Gorp [3TU.Datacentrum Symposium 2014, ...
by3TU.Datacentrum
 
PPT
FSOSS 2011 Beyond MAN Pages
byBeth Agnew
 
PDF
WordCamp New York - About Mozilla
byPaul Kim
 
PDF
Xen Project Contributor Training - Part 1 introduction v1.0
byThe Linux Foundation
 
PPTX
How Developers’ Collaborations Identified from Different Sources Tell us Abou...
bySebastiano Panichella
 
PDF
Code Reviews in Python - PyZh
byCesar Cardenas Desales
 
PDF
Code Review for Teams Too Busy to Review Code - Atlassian Summit 2010
byAtlassian
 
PPTX
Automatically Identifying the Quality of Developer Chats for Post Hoc Use
byPreetha Chatterjee
 
Dealing with contributor overload - FOSS Backstage
byHolden Karau
 
Automatic Identification of Informative Code in Stack Overflow Posts
byPreetha Chatterjee
 
We builit it
bywebuiltit
 
Opening up Open Source
byDerek Buitenhuis
 
Growing Developer Community One Pull Request At a Time
byAlex Bulankou
 
Leveraging the Crowd: Supporting Newcomers to Build an OSS Community
byMarco Aurelio Gerosa
 
Is Empirical Enough?
byPer Runeson
 
淺談 Startup 公司的軟體開發流程 v2
byWen-Tien Chang
 
Agile startup company management and operation
byJiang Zhu
 
Agile for digital library projects
byNaomi Dushay
 
When Users Becom Collaborators: Towards Continuous and Context-Aware User Input
byHans-Joerg Happel
 
How to Run a Successful Open Source Java EE Project
byphilipdurbin
 
[3.6] Beyond Data Sharing - Pieter van Gorp [3TU.Datacentrum Symposium 2014, ...
by3TU.Datacentrum
 
FSOSS 2011 Beyond MAN Pages
byBeth Agnew
 
WordCamp New York - About Mozilla
byPaul Kim
 
Xen Project Contributor Training - Part 1 introduction v1.0
byThe Linux Foundation
 
How Developers’ Collaborations Identified from Different Sources Tell us Abou...
bySebastiano Panichella
 
Code Reviews in Python - PyZh
byCesar Cardenas Desales
 
Code Review for Teams Too Busy to Review Code - Atlassian Summit 2010
byAtlassian
 
Automatically Identifying the Quality of Developer Chats for Post Hoc Use
byPreetha Chatterjee
 

More from Denae Ford

PDF
MSR Human And Well Being: The Next Generation of Software Developers
byDenae Ford
 
PDF
How Remote Work Can Foster a More Inclusive Environment for Transgender Devel...
byDenae Ford
 
PDF
“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Eng...
byDenae Ford
 
PDF
Eating Your Donuts and Basking in Your Glow
byDenae Ford
 
PDF
Someone Like Me: How Does Peer Parity Influence Participation of Women on Sta...
byDenae Ford
 
PDF
Paradise Unplugged: Identifying Barriers for Female Participation on Stack Ov...
byDenae Ford
 
PDF
The Tech-Talk Balance: What Technical Interviewers Expect from Technical Cand...
byDenae Ford
 
PPTX
EMIP15 Workshop Presentation
byDenae Ford
 
PPTX
Exploring the Causes of Frustration for Software Developers
byDenae Ford
 
MSR Human And Well Being: The Next Generation of Software Developers
byDenae Ford
 
How Remote Work Can Foster a More Inclusive Environment for Transgender Devel...
byDenae Ford
 
“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Eng...
byDenae Ford
 
Eating Your Donuts and Basking in Your Glow
byDenae Ford
 
Someone Like Me: How Does Peer Parity Influence Participation of Women on Sta...
byDenae Ford
 
Paradise Unplugged: Identifying Barriers for Female Participation on Stack Ov...
byDenae Ford
 
The Tech-Talk Balance: What Technical Interviewers Expect from Technical Cand...
byDenae Ford
 
EMIP15 Workshop Presentation
byDenae Ford
 
Exploring the Causes of Frustration for Software Developers
byDenae Ford
 

Recently uploaded

PDF
Highway Construction and Pavement design.pdf
byVinayVitekari
 
PPT
6.1POWER BLOCK NEW.ppt indian railway three
bykhuranarahul599
 
PDF
Workshop practice theory course (Unit-1) By Varun Pratap Singh.pdf
byVarun Pratap Singh
 
PPTX
hydrology and unit hydrograph presents.pptx
byp26335566
 
PDF
Existing Building Survey - Edi Supriyanto & Partners.pdf
byProjectEngineer4
 
PPT
5.4ODC.ppt indian railways bharat sarkar
bykhuranarahul599
 
PDF
Fluid Mechanics & Hydraulic Digital Material.pdf
bysparunkumar1
 
PDF
HYDRAULIC TURBINES with question and answer.pdf
bysparunkumar1
 
PDF
application of matrix in computer science
byankitberamath
 
PDF
A Guide to Marine Air Compressors on Ships
byMahmoud Moghtaderi
 
PPTX
State Space Model of DC-DC Boost Converter
byPorselvi T
 
PPTX
Modularity In Lyra - A new studio's experiences building their prototype on U...
byiamscottcstevens
 
PDF
HYDRAULIC TURBINES with question and answer .pdf
bysparunkumar1
 
PPTX
thermal_design_project_work_internship(updated)-1.pptx
byshahveer210504
 
PDF
Global Research and Innovation Conference
byicsift49
 
PDF
ANPARA THERMAL POWER STATION[1] sangam.pdf
bysangamkumar993613
 
PDF
Event #3_ Build a Gemini Bot, Together with GitHub_private.pdf
by3526ming
 
PPTX
Joseph Hill Honors Thesis Presentation 2024
byJoeHill71
 
PDF
Decarbonizing Healthcare: Exploring the Potential for Electrification with IESVE
byIES VE
 
PPTX
33 KV SUBSTATION.pptx FROM LAYOUT TO PROTECTION
byPradipChanda5
 
Highway Construction and Pavement design.pdf
byVinayVitekari
 
6.1POWER BLOCK NEW.ppt indian railway three
bykhuranarahul599
 
Workshop practice theory course (Unit-1) By Varun Pratap Singh.pdf
byVarun Pratap Singh
 
hydrology and unit hydrograph presents.pptx
byp26335566
 
Existing Building Survey - Edi Supriyanto & Partners.pdf
byProjectEngineer4
 
5.4ODC.ppt indian railways bharat sarkar
bykhuranarahul599
 
Fluid Mechanics & Hydraulic Digital Material.pdf
bysparunkumar1
 
HYDRAULIC TURBINES with question and answer.pdf
bysparunkumar1
 
application of matrix in computer science
byankitberamath
 
A Guide to Marine Air Compressors on Ships
byMahmoud Moghtaderi
 
State Space Model of DC-DC Boost Converter
byPorselvi T
 
Modularity In Lyra - A new studio's experiences building their prototype on U...
byiamscottcstevens
 
HYDRAULIC TURBINES with question and answer .pdf
bysparunkumar1
 
thermal_design_project_work_internship(updated)-1.pptx
byshahveer210504
 
Global Research and Innovation Conference
byicsift49
 
ANPARA THERMAL POWER STATION[1] sangam.pdf
bysangamkumar993613
 
Event #3_ Build a Gemini Bot, Together with GitHub_private.pdf
by3526ming
 
Joseph Hill Honors Thesis Presentation 2024
byJoeHill71
 
Decarbonizing Healthcare: Exploring the Potential for Electrification with IESVE
byIES VE
 
33 KV SUBSTATION.pptx FROM LAYOUT TO PROTECTION
byPradipChanda5
 

Beyond the Code Itself: How Programmers Really Look at Pull Requests

  • 1.
    Beyond the CodeItself: How Programmers Really Look at Pull Requests Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, Chris ParninDenaeFordRobin North Carolina State University + Eindhoven University Of Technology
  • 2.
    PULL REQUEST: SUBMITTINGCODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull Request
  • 3.
    PULL REQUEST: SUBMITTINGCODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is Merged
  • 4.
    PULL REQUEST: SUBMITTINGCODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is MergedDiscuss + Review
  • 5.
    CODE CONTRIBUTIONS BASEDON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
  • 6.
    CODE CONTRIBUTIONS BASEDON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
  • 7.
  • 8.
  • 9.
  • 10.
    6BUILDING ON PRIORWORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  • 11.
    6BUILDING ON PRIORWORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  • 12.
    6BUILDING ON PRIORWORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  • 13.
    6BUILDING ON PRIORWORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  • 14.
    How do programmersreview pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 7 RQ2 RQ1 RQ3 RESEARCH QUESTIONS
  • 15.
    8METHODOLOGY PROTOCOL Pre Experiment Survey Have youever accepted or rejected or submitted a pull request? * Please rate your programming experience on a scale from 1-10. * How much experience do you have performing integration tasks? * Do you have a GitHub account? *
  • 16.
    9 PROTOCOL Pre Experiment Survey Tic TacToe Training Session 9 METHODOLOGY
  • 17.
    10 PROTOCOL Pre Experiment Survey Tic TacToe Training Session Review Profile Page + Pull Request METHODOLOGY
  • 18.
    10 PROTOCOL Pre Experiment Survey Tic TacToe Training Session Review Profile Page + Pull Request METHODOLOGY
  • 19.
    10 PROTOCOL Pre Experiment Survey Tic TacToe Training Session Review Profile Page + Pull Request METHODOLOGY
  • 20.
    11 PROTOCOL Pre Experiment Survey Tic TacToe Training Session Review Profile Page + Pull Request 11 Reasonable Code Snippet: Unreasonable Code Snippet: METHODOLOGY
  • 21.
    12 PROTOCOL Pre Experiment Survey Tic TacToe Training Session Post Experiment Survey Review Profile Page + Pull Request What elements of the displayed profile or pull request did you consider when making your decision? * What strategies do you use to decide to share this content? Does it vary across each community? * METHODOLOGY
  • 22.
  • 23.
  • 24.
    Recruited 42 ParticipantsAdvanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
  • 25.
    All familiar withsubmitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
  • 26.
    All familiar withsubmitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY Gender Quantity Age Range Country of Origin Minority Not Minority Men 30 22-33 24 India, 1 Nepal, 1 USA 2 23 Women 12 23-29 2 China, 5 India, 1 Iran, 3 USA 1 10
  • 27.
    How do programmersreview pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS
  • 28.
    How do programmersreview pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience
  • 29.
    How do programmersreview pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses
  • 30.
    How do programmersreview pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses Platform Specific Strategies
  • 31.
  • 32.
  • 33.
  • 34.
    TABLE II: ParticipantFixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
  • 35.
    TABLE II: ParticipantFixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
  • 36.
    TABLE II: ParticipantFixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
  • 37.
    TABLE II: ParticipantFixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
  • 38.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject
  • 39.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Code Signals
  • 40.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals
  • 41.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
  • 42.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject 5 out of 20 participants spent 48% to 62% of their time fixating on technical signals and an above average time on social signals (17% to 31%) Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
  • 43.
    Participants M1 M2 M3M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject — = no decision More details in paper: http://bit.ly/BeyondCode
  • 44.
    FIXATION + EXPERIENCE= RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  • 45.
    FIXATION + EXPERIENCE= RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  • 46.
    FIXATION + EXPERIENCE= RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  • 47.
    RQ2: WHERE DOPROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  • 48.
    73% reported usingthe code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  • 49.
    45% reported usinginformation related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  • 50.
    45% reported usinginformation related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest 1 participant reported using the submitter’s profile image
  • 51.
    RQ3: WHAT STRATEGIESDO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy
  • 52.
    RQ3: WHAT STRATEGIESDO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)
  • 53.
    RQ3: WHAT STRATEGIESDO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26) “such familiarity gives a feeling of trust.” (S15)
  • 54.
    DISCUSSION + IMPLICATIONS23 All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
  • 55.
    DISCUSSION + IMPLICATIONS23 GitHub profiles make social + human capital explicit. How does giving users control over their identity influence participation? All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
  • 56.
    Summary Paper: http://bit.ly/BeyondCode DenaeFordRobin (BC) (AC) (UD) (PT) (SD) (RP) (CD) (TM) dford3@ncsu.edu Weobserve that both social and technical aspects are being taken into consideration when deciding upon pull request acceptance. Moreover, many more social aspects are being considered during the experiment than reported during the post-experiment survey Future work will study how the execution of concealing or amplifying these signals affect developers across the identity spectrum and development experiences at scale. Artifact: http://bit.ly/BeyondCodeArtifacts