The University of Adelaide, profile picture
Uploaded byThe University of Adelaide
PDF, PPTX1,380 views

Ph.D. Thesis Defense: Studying Reviewer Selection and Involvement in Modern Code Review Processes

The document discusses the challenges and processes of modern code reviews, emphasizing the shift from traditional formal structures to lightweight collaborative tools in contemporary software development. It highlights the importance of effective reviewer selection and involvement in maintaining software quality while identifying issues such as lax reviewing practices that can lead to poor code quality. The thesis argues for an improved understanding of modern code review processes as they are increasingly adopted worldwide.

Embed presentation

Download as PDF, PPTX
1 / 126
2 / 126
3 / 126
4 / 126
5 / 126
6 / 126
7 / 126
8 / 126
9 / 126
10 / 126
11 / 126
12 / 126
13 / 126
14 / 126
15 / 126
16 / 126
17 / 126
18 / 126
19 / 126
20 / 126
21 / 126
22 / 126
23 / 126
24 / 126
25 / 126
26 / 126
27 / 126
28 / 126
29 / 126
30 / 126
31 / 126
32 / 126
33 / 126
34 / 126
35 / 126
36 / 126
37 / 126
38 / 126
39 / 126
40 / 126
41 / 126
42 / 126
43 / 126
44 / 126
45 / 126
46 / 126
47 / 126
48 / 126
49 / 126
50 / 126
51 / 126
52 / 126
53 / 126
54 / 126
55 / 126
56 / 126
57 / 126
58 / 126
59 / 126
60 / 126
61 / 126
62 / 126
63 / 126
64 / 126
65 / 126
66 / 126
67 / 126
68 / 126
69 / 126
70 / 126
71 / 126
72 / 126
73 / 126
74 / 126
75 / 126
76 / 126
77 / 126
78 / 126
79 / 126
80 / 126
81 / 126
82 / 126
83 / 126
84 / 126
85 / 126
86 / 126
87 / 126
88 / 126
89 / 126
90 / 126
91 / 126
92 / 126
93 / 126
94 / 126
95 / 126
96 / 126
97 / 126
98 / 126
99 / 126
100 / 126
101 / 126
102 / 126
103 / 126
104 / 126
105 / 126
106 / 126
107 / 126
108 / 126
109 / 126
110 / 126
111 / 126
112 / 126
113 / 126
114 / 126
115 / 126
116 / 126
117 / 126
118 / 126
119 / 126
120 / 126
121 / 126
122 / 126
123 / 126
124 / 126
125 / 126
126 / 126
Studying Reviewer Selection and Involvement in Modern Code Review Processes — Ph.D. Thesis Defense — Patanamon (Pick) Thongtanunam patanamon-t@is.naist.jp @pamon http://patanamon.com
2 Presentation Outline Modern Code Review Processes Empirical Studies Problem Statement Wrap-upThesis Structure
Software code review is a well-established 
 software quality practice 3
Software code review is a well-established 
 software quality practice 3
Software code review is a well-established 
 software quality practice Poor Software Quality (e.g., Defects, Errors) 3
Software code review is a well-established 
 software quality practice Poor Software Quality (e.g., Defects, Errors) Poor Design Quality (e.g., Spaghetti code) 3
Traditionally, code review processes are formal and rigidly structured 4
Traditionally, code review processes are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team 4
Traditionally, code review processes are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team Performing 
 in-person meetings Time is explicitly allocated and roles are well defined 4
Traditionally, code review processes are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team Performing 
 in-person meetings Time is explicitly allocated and roles are well defined Fixing the problems that are collected during 
 the meeting 4
In modern software organizations, 
 formal code review is hard to adopt Modern software development practices Collaboration between developers is asynchronous Teams are globally distributed 5
In modern software organizations, 
 formal code review is hard to adopt Modern software development practices Collaboration between developers is asynchronous Teams are globally distributed Modern Code Review (MCR): A tool-based code review process 5
Gerrit Code Review Rietveld Modern Code Review is widely adopted in industrial and open source projects 6
Modern Code Review is coordinated using lightweight collaborative tools and processes 7
Modern Code Review is coordinated using lightweight collaborative tools and processes Upload a patch A code review tool (e.g. Gerrit) An author 7
Modern Code Review is coordinated using lightweight collaborative tools and processes Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
Modern Code Review is coordinated using lightweight collaborative tools and processes Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
Modern Code Review is coordinated using lightweight collaborative tools and processes Address the feedback Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
Modern Code Review is coordinated using lightweight collaborative tools and processes Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author 7
Thesis Statement: The lightweight MCR process 
 lacks mechanisms for preventing 
 lax code reviewing practices, which can allow poor quality code to slip through to official software releases 8
A deeper understanding of MCR processes and their impact on software quality is needed as MCR continues to be adopted by projects worldwide 9
Suboptimal reviewing practices 
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author 10
Suboptimal reviewing practices 
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author No criteria of reviewer selection 11
Suboptimal reviewing practices 
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Insufficient reviewer involvement 12
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Reviewer Involvement 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Reviewer Involvement 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 13
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 14
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 14
15 Module B Module A Minor authors Major author Minor author Low code authoring expertise is associated 
 with poor software quality [Bird et al., 2011], [Rahman and Devanbu,2011] Selecting a developer to whom tasks should be delegated is important for software quality
16 Reviewing expertise is important in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer
16 Reviewing expertise is important in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect
16 Reviewing expertise is important in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution
16 Reviewing expertise is important in Modern Code Review processes Module A Qt Review #27977 Author Update A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution v2
17 Reviewing expertise is important in Modern Code Review processes Module A Author A code review tool (e.g. Gerrit) Qt Review #35360
17 Reviewing expertise is important in Modern Code Review processes Module A Author A code review tool (e.g. Gerrit) Providing updates to the code change Note that running qutlook will probably crash. I will push a patch set to fix the issues Reviewer v2 Qt Review #35360
18 Yet, little is known whether the reviewing expertise can reduce the risk of having defects Module A Minor authors Major reviewer Major reviewer
19 Approximating expertise of developers using code authoring and reviewing data Minor Reviewer Major Reviewer Major Author Minor Author Code authoring expertise Code reviewing expertise The more code changes that a developer has authored or reviewed, the more expertise of that activity that developer has
19 Approximating expertise of developers using code authoring and reviewing data Minor Reviewer Major Reviewer Major Author Minor Author Code authoring expertise Code reviewing expertise
Code reviewing expertise Code authoring expertise VS Compare the association with defect-proneness Minor Reviewer Major Reviewer Major Author Minor Author Analyzing the association between 
 expertise and defect-proneness 20
Percentageofbothminor authors&reviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenSta Havan 020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly Open Hav nStack avana OpenStack Icehouse Defective components Clean components Higher rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Reviewing expertise reverses the association between authoring expertise and defect-proneness 21
Percentageofbothminor authors&reviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenSta Havan 020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly Open Hav nStack avana OpenStack Icehouse Defective components Clean components Higher rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Percentageofminorauthorsbut majorreviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenStack Havana OpenStack Icehouse Lower rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Reviewing expertise reverses the association between authoring expertise and defect-proneness 21
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 22
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects 22
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects Studying Reviewer Selection and Involvement in Modern Code Review Processes 23
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects Studying Reviewer Selection and Involvement in Modern Code Review Processes 23
An author Manually selecting reviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) 24
An author Manually selecting reviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) Who should review this patch? 24
An author Manually selecting reviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) Hi, can you please add appropriate reviewers for this change? Android Review #18767 Add a developer 24
Reviewer selection can slow down the code review process Studied Projects Sample Reviews 357 378 363 363 #Patch w/ reviewer selection problem 10% 30% 5% 4% Patches where the discussions are related to reviewer selection 25
Reviewer selection can slow down the code review process Studied Projects Sample Reviews 357 378 363 363 #Patch w/ reviewer selection problem 10% 30% 5% 4% Time that is spent on MCR process (Days) Reviews with code−reviewer assignment proble Reviewingtime(Days) Android OpenStack 0102030 de−reviewer assignment problem Reviews without code− OpenStack Qt Patches w/ reviewer selection problem Patches w/o reviewer selection problem Patches w/ the reviewer selection problem Patches w/o the reviewer selection problem 020406080 Android Qt OpenStack LibreOffice Studied systems Timethatisspenton MCRprocess(Days) 21 Days Longer 13 Days Longer9 Days Longer 6 Days Longer 25
R1 R2 Changed Files - src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach Appropriate reviewers are those who have reviewed the most similar file paths 26
R1 R2 Changed Files - src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach Appropriate reviewers are those who have reviewed the most similar file paths 26
R1 R2 Changed Files - src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach “Pick should review R3” 26
Longest Common Prefix Longest Common Suffix Longest Common Substring Longest Common Subsequence RevFinder uses 4 string comparison techniques 27
Longest Common Prefix Longest Common Suffix Longest Common Substring Longest Common Subsequence Reviewer recommendation Combine (Borda Count) Voting Ranks RevFinder uses 4 string comparison techniques 27
Longest Common Prefix Longest Common Suffix Longest Common Substring Longest Common Subsequence Reviewer recommendation Combine (Borda Count) Voting Ranks Bubble-up correct reviewers RevFinder uses 4 string comparison techniques 27
RevFinder can accurately recommend reviewers 28
RevFinder can accurately recommend reviewers Accuracy Top-10Accuracy 0% 30% 60% 90% 10% 28% 41% 29% 74% 69% 87%86% RevFinder Baseline Approach 28
RevFinder can accurately recommend reviewers Accuracy Top-10Accuracy 0% 30% 60% 90% 10% 28% 41% 29% 74% 69% 87%86% RevFinder Baseline Approach Ranking Performance MeanReciprocalRank 0 0.2 0.4 0.6 0.07 0.22 0.3 0.25 0.4 0.31 0.55 0.6 RevFinder Baseline Approach 28
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects 29
Studying Reviewer Selection and Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch 29
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 30
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 30
A lack of code review increases the risk of having defects Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (E.x. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 31
A lack of code review increases the risk of having defects Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (E.x. Gerrit) Examine the patch An author Integrate a clean patch into software repositories The number of patches without reviewer involvement is associated with the risk of having defects [McIntosh et al., 2015] 31
A low level of reviewer involvement may introduce in Modern Code Review process Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 32
A low level of reviewer involvement may introduce in Modern Code Review process Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 32
Investigating the difference between code review practices of defective and clean files 33
Review Practice A Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 33
Defect-free (Clean) i.e., files that do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 33
Defect-free (Clean) i.e., files that do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 33
Review Intensity e.g., #Review Iterations, Discussion Length Review Participation e.g., #Reviewers, 
 Review Agreement Reviewing Time e.g., Review Length, Code Reading Speed Characterizing reviewing activity using 11 metrics grouped along 3 dimensions 34
Defect-free (Clean) i.e., files that do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 35
Defect-free (Clean) i.e., files that do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 35
Defective i.e., files that have defects Studying defective files along two perspectives 36
Defective i.e., files that have defects Studying defective files along two perspectives 36
Risky Files Files that have historically been defective Past Defective i.e., files that have defects Studying defective files along two perspectives 36
Risky Files Files that have historically been defective Past Future Defective i.e., files that have defects Future-Defective Files Files that will eventually have defects Studying defective files along two perspectives 36
Risky Files Files that have historically been defective Past Future Defective i.e., files that have defects Future-Defective Files Files that will eventually have defects Studying defective files along two perspectives 36
Risky Files Files that have historically been defective Past Future Defective i.e., files that have defectsFuture-Defective Files Files that will eventually have defects Conjecture: Reviews of Future- Defective will be • less intense, • with less team participation, • completed with a shorter time than reviews of clean files Studying defective files along two perspectives 36
#Reviewers #Reviews of Clean files #Reviewers #Reviews of Future-Defective files VS Using a statistical test to determine the difference between the distributions of code review activity Comparing code review activity that has been applied to future-defective and clean files 37
FindingsConjecture Review Intensity Review Participation Reviewing Time Less intense Less team participation Completed within a shorter time Reviewing activity in future-defective files are less rigorous 38
FindingsConjecture Shorter discussion Fewer reviewers Receive slow initial feedback Review Intensity Review Participation Reviewing Time Less intense Less team participation Completed within a shorter time Reviewing activity in future-defective files are less rigorous 38
Defective i.e., files that have defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Conjecture: Reviews of Future- Defective will be • less intense, • with less team participation, • completed with a shorter time than reviews of clean files Studying defective files along two perspectives 39
Defective i.e., files that have defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
Defective i.e., files that have defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
Defective i.e., files that have defects Past Risky Files Files that have historically been defective Conjecture: Reviews of risky files should be • more intense, • with more team participation, • reviewed for a longer time to reduce the risk of having defects in the future FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
FindingsConjecture Review Intensity Review Participation Reviewing Time More intense More team participation Take longer time to complete Reviewing activities in risky files are not careful 40
FindingsConjecture Shorter discussion Fewer reviewers Receive Slow Feedback Review Intensity Review Participation Reviewing Time More intense More team participation Take longer time to complete Reviewing activities in risky files are not careful 40
Defective i.e., files that have defects Past FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Risky Files Files that have historically been defective Conjecture: Reviews of risky files should be • more intense, • with more team participation, • reviewed for a longer time to reduce the risk of having defects in the future Studying defective files along two perspectives 41
Defective i.e., files that have defects Past FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Risky Files Files that have historically been defective Developers are not as careful when they review risky files Studying defective files along two perspectives 41
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 42
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 42
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 43
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 43
A code review tool (e.g. Gerrit) Poor reviewer involvement has an impact on software quality Integrate a clean patch into software repositories 44
A code review tool (e.g. Gerrit) Investigating the characteristics of patches that suffer from poor reviewer involvement Upload a patch This patch is likely to suffer from poor reviewer involvement Select reviewers An author 45
Studying poor reviewer involvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time 46
Studying poor reviewer involvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation 46
Studying poor reviewer involvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation No review discussion 46
Studying poor reviewer involvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation No review discussion Slow initial feedback 46
Properties e.g., Size, Description length of code changes History e.g., Past tendency, #days since the last modification Environment e.g., Review workload Past Involvement e.g., Past involvement of an author and reviewers Measuring characteristics of patches using 20 metrics grouped along 4 dimensions 47
48 Measure patch characteristics STEP 1 Training statistical models to explain the relationship History EnvironmentProperties Past Involvement
49 Measure patch characteristics STEP 1 Train a prediction model STEP 2 x Assess the risk of receiving poor reviewer involvement Relax the linear assumption of the relationship Non-linear logistic regression model Probability x x x x Training statistical models to explain the relationship
50 Measure patch characteristics STEP 1 Train a prediction model STEP 2 STEP 3 Assess the fitness of the model Discrimination Ability (AUC) Stability (AUC Optimism) Model performance Training statistical models to explain the relationship
51 Measure patch characteristics STEP 1 Train a prediction model STEP 2 STEP 3 Assess the fitness of the model Interpret model STEP 4 Explanatory power (Wald χ2 statistics) Plot the relationship with the risk of receiving poor reviewer involvement Observe code ownership variables Increasing Relationship Inverse Relationship or Training statistical models to explain the relationship
Our statistical models are accurate and stable enough for interpretation No reviewer participation No review discussion Slow initial feedback 0.72 0.70 0.74 AUC Optimism AUC 0.002 0.001 0.001 0.70 0.72 0.78 AUC Optimism AUC 0.002 0.001 0.001 0.66 0.61 0.61 AUC Optimism AUC 0.004 0.002 0.001 52
History Past activity tendency Oddsofreceivingpoor reviewerinvolvement Properties Description length Patches whose prior changes received poor reviewer involvement, and that are not descriptive are likely to receive poor reviewer involvement History and properties of patches share a strong relationship with poor reviewer involvement Oddsofreceivingpoor reviewerinvolvement 53
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 54
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes Past tendency and a patch description are strong indicators of poor reviewer involvement 54
55 • (S1) Revisiting Code Ownership and its Relationship with Software Quality in the Scope of Modern Code Review. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Ha- jimu Iida. 
 In Proceedings of the 38th International Conference on Software Engineering (ICSE), 2016, pp. 1039–1050. • (S2) Who Should Review My Code? 
 Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, Kenichi Matsumoto. 
 In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2015, pp. 141–150. • (S3) Investigating Code Review Practices in Defective Files. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 
 In Proceedings of the 12th International Conference on Mining Software Repositories (MSR), 2015, pp. 168–179.
 This paper won the IEEE Kansai Excellent Student Paper Award 2015 • (S4) Review Participation in Modern Code Review. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 
 Springer Journal of Empirical Software Engineering (EMSE), 39 pages. Under review. Related Publications
Thesis Structure Modern Code Review Process Chapter 2 Code Review Process Part I The Modern Code Review Process Chapter 3 Related Research Part II Reviewer Selection Part III Reviewer Involvement Empirical studies Chapter 4: The Impact of Reviewer Selection on Software Quality Chapter 6: The Impact of Review Involvement on Software Quality Chapter 5: Selecting Appropriate Reviewers Chapter 7: Identifying Patches with Poor Review Involvement Chapter8 ConclusionandFutureWork Chapter1 Introduction 56
57 Wrap-up
Gerrit Code Review Rietveld Modern Code Review is widely adopted in industrial and open source projects
Thesis Statement: The lightweight MCR process 
 lacks mechanisms for preventing 
 lax code reviewing practices, which can allow poor quality code to slip through to official software releases
A deeper understanding of MCR processes and their impact on software quality is needed as MCR continues to be adopted by projects worldwide
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects The low level of reviewer involvement can lead to poor software quality RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015
Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects The low level of reviewer involvement can lead to poor software quality RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015 Past tendency and a patch description are strong indicators of poor reviewer involvement EMSE (Under Major Revision)

More Related Content

PDF
Improving Code Review Effectiveness Through Reviewer Recommendations
byThe University of Adelaide
 
PDF
Revisiting Code Ownership and Its Relationship with Software Quality in the S...
byThe University of Adelaide
 
PDF
Using HPC Resources to Exploit Big Data for Code Review Analytics
byThe University of Adelaide
 
PDF
Who Should Review My Code?
byThe University of Adelaide
 
PDF
Review Participation in Modern Code Review: An Empirical Study of the Android...
byThe University of Adelaide
 
PDF
Investigating Code Review Practices in Defective Files
byThe University of Adelaide
 
PDF
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
bySung Kim
 
PPTX
Source code comprehension on evolving software
bySung Kim
 
Improving Code Review Effectiveness Through Reviewer Recommendations
byThe University of Adelaide
 
Revisiting Code Ownership and Its Relationship with Software Quality in the S...
byThe University of Adelaide
 
Using HPC Resources to Exploit Big Data for Code Review Analytics
byThe University of Adelaide
 
Who Should Review My Code?
byThe University of Adelaide
 
Review Participation in Modern Code Review: An Empirical Study of the Android...
byThe University of Adelaide
 
Investigating Code Review Practices in Defective Files
byThe University of Adelaide
 
Partitioning Composite Code Changes to Facilitate Code Review (MSR2015)
bySung Kim
 
Source code comprehension on evolving software
bySung Kim
 

What's hot

PPT
Is Text Search an Effective Approach for Fault Localization: A Practitioners ...
byDebdoot Mukherjee
 
PDF
Python - code quality and production monitoring
byDavid Melamed
 
PDF
Euro python 2015 writing quality code
byradek_j
 
PDF
Static Analysis of Your OSS Project with Coverity
bySamsung Open Source Group
 
PPTX
Top 5 Code Coverage Tools in DevOps
byscmGalaxy Inc
 
PPTX
Statistical debuging for programs written in dynamic programming language ruby
byAdeel Akhter
 
PDF
Presentation slides: "How to get 100% code coverage"
byRapita Systems Ltd
 
PDF
Software Engineering Culture - Improve Code Quality
byDmytro Patserkovskyi
 
PDF
[India Merge World Tour] Coverity
byPerforce
 
PPT
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
byEngineering Software Lab
 
PDF
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
byFeng Zhang
 
PDF
Code coverage
byVijayan Reddy
 
PDF
Parasoft fda software compliance part2
byEngineering Software Lab
 
PPTX
Python code profiling - Jackson Isaac
byJackson Isaac
 
PPTX
Mining Code Examples with Descriptive Text from Software Artifacts
byPreetha Chatterjee
 
PDF
Understanding, measuring and improving code quality in JavaScript
byMark Daggett
 
PDF
Early Detection of Collaboration Conflicts & Risks in Software Development
byRoopesh Jhurani
 
PPTX
Revisiting Assert Use in GitHub Projects
byPavneet Singh Kochhar
 
PPTX
How To Improve Quality With Static Code Analysis
byPerforce
 
ODP
Java code coverage with JCov. Implementation details and use cases.
byAlexandre (Shura) Iline
 
Is Text Search an Effective Approach for Fault Localization: A Practitioners ...
byDebdoot Mukherjee
 
Python - code quality and production monitoring
byDavid Melamed
 
Euro python 2015 writing quality code
byradek_j
 
Static Analysis of Your OSS Project with Coverity
bySamsung Open Source Group
 
Top 5 Code Coverage Tools in DevOps
byscmGalaxy Inc
 
Statistical debuging for programs written in dynamic programming language ruby
byAdeel Akhter
 
Presentation slides: "How to get 100% code coverage"
byRapita Systems Ltd
 
Software Engineering Culture - Improve Code Quality
byDmytro Patserkovskyi
 
[India Merge World Tour] Coverity
byPerforce
 
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
byEngineering Software Lab
 
Cross-project Defect Prediction Using A Connectivity-based Unsupervised Class...
byFeng Zhang
 
Code coverage
byVijayan Reddy
 
Parasoft fda software compliance part2
byEngineering Software Lab
 
Python code profiling - Jackson Isaac
byJackson Isaac
 
Mining Code Examples with Descriptive Text from Software Artifacts
byPreetha Chatterjee
 
Understanding, measuring and improving code quality in JavaScript
byMark Daggett
 
Early Detection of Collaboration Conflicts & Risks in Software Development
byRoopesh Jhurani
 
Revisiting Assert Use in GitHub Projects
byPavneet Singh Kochhar
 
How To Improve Quality With Static Code Analysis
byPerforce
 
Java code coverage with JCov. Implementation details and use cases.
byAlexandre (Shura) Iline
 

Similar to Ph.D. Thesis Defense: Studying Reviewer Selection and Involvement in Modern Code Review Processes

PPTX
Code_Review_Presentation_v22222_LLM.pptx
bySofienBoutaib
 
PDF
On to code review lessons learned at microsoft
byMichaela Greiler
 
PDF
ICSME 2016: Search-Based Peer Reviewers Recommendation in Modern Code Review
byAli Ouni
 
PDF
Code Review Tool Evaluation
byKate Semizhon
 
PPTX
Code Reviews
byphildenoncourt
 
PDF
An Empirical Investigation of Relevant Changes and Automation Needs in Modern...
bySebastiano Panichella
 
PPTX
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
bySAIL_QU
 
PDF
Code Reviews Inside Out
byInes Jelovac
 
PDF
Code reviews
byRaúl Araya Tauler
 
PPTX
Capability Building for Cyber Defense: Software Walk through and Screening
byMaven Logix
 
PDF
Automated Code Reviews with AI and ML - DevOps Next
byPerfecto by Perforce
 
PDF
Code-Review-Principles-Process-and-Tools (1)
byAditya Bhuyan
 
PPTX
The Best Kept Secrets of Code Review | SmartBear Webinar
bySmartBear
 
PPTX
Unit3 software review control software
byReetesh Gupta
 
PPT
Code Review
byRavi Raj
 
PDF
code_review_checklist_6_actions_to_improve_the_quality_of_your_reviews.pdf
bysarah david
 
PPT
Code Review
byrantav
 
PDF
software testing 5.1.pdf
byHarshPrajapati670262
 
PDF
Code Reviews @ Quatico
byJan Wloka
 
PDF
Code review best practice
byOren Digmi
 
Code_Review_Presentation_v22222_LLM.pptx
bySofienBoutaib
 
On to code review lessons learned at microsoft
byMichaela Greiler
 
ICSME 2016: Search-Based Peer Reviewers Recommendation in Modern Code Review
byAli Ouni
 
Code Review Tool Evaluation
byKate Semizhon
 
Code Reviews
byphildenoncourt
 
An Empirical Investigation of Relevant Changes and Automation Needs in Modern...
bySebastiano Panichella
 
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
bySAIL_QU
 
Code Reviews Inside Out
byInes Jelovac
 
Code reviews
byRaúl Araya Tauler
 
Capability Building for Cyber Defense: Software Walk through and Screening
byMaven Logix
 
Automated Code Reviews with AI and ML - DevOps Next
byPerfecto by Perforce
 
Code-Review-Principles-Process-and-Tools (1)
byAditya Bhuyan
 
The Best Kept Secrets of Code Review | SmartBear Webinar
bySmartBear
 
Unit3 software review control software
byReetesh Gupta
 
Code Review
byRavi Raj
 
code_review_checklist_6_actions_to_improve_the_quality_of_your_reviews.pdf
bysarah david
 
Code Review
byrantav
 
software testing 5.1.pdf
byHarshPrajapati670262
 
Code Reviews @ Quatico
byJan Wloka
 
Code review best practice
byOren Digmi
 

Recently uploaded

PPTX
68b5c214ea502_CraftVerse-ppt-Template.pptx
byAdarshThakare3
 
PPTX
Sample Equity research report for Lafarge.pptx
bysamuelkingsofficial2
 
PPTX
Peptide Biomaterials for Skeletal Regeneration: From Molecular Design to Func...
byNaibedyaChattopadhya
 
PPTX
Presentation schizo (1).pptxguiikkkkkkki
byrashibarthwal
 
PPTX
Spelling.pptx...........................
byroyalamericanschools
 
PPTX
Mutliplying and Dividing Fractions Presentation.pp
bykikakashi0
 
PDF
OLLD25 - The Edges of University Living Labs session
bySam Rye
 
PDF
English Learning Presentation (1).pdf....
byhalaalqrni111
 
PPTX
Unit 03CIVIL SOCIETY AND DEMOCRACY IN PAKISTAN.pptx
bymuskan54863
 
PPTX
AI in Tourism: transforming visitor experience & operations
byAidan Crawford
 
PPTX
Powerpoint Classroom Action Research 4 yn.
byLiyana
 
PPTX
tongue and its prosthodontic considerations
byFebaMariya2
 
PPTX
The-Architecture-of-Counter-Ideology.pptx
bynotabolfazlpd
 
PDF
Google_CodeCamp_workshop_FrontendGDG.pdf
bymoharnabgogoi
 
PPTX
Bob Stewart James 3 11 16 2025.pptx
byFamilyWorshipCenterD
 
PPTX
2025-11-16 Moses 07 (shared slides).pptx
byDale Wells
 
PDF
PresentationEnglish language learning presentation file
by446814956
 
PDF
Question Generation LLM Gemini - Conference presentation - FIE 2025
bypmmorenom01
 
PDF
King Khaled university .. Presentation .
by446816119
 
PPTX
A PowerPoint some information taj-mahal.pptx
byvabb1
 
68b5c214ea502_CraftVerse-ppt-Template.pptx
byAdarshThakare3
 
Sample Equity research report for Lafarge.pptx
bysamuelkingsofficial2
 
Peptide Biomaterials for Skeletal Regeneration: From Molecular Design to Func...
byNaibedyaChattopadhya
 
Presentation schizo (1).pptxguiikkkkkkki
byrashibarthwal
 
Spelling.pptx...........................
byroyalamericanschools
 
Mutliplying and Dividing Fractions Presentation.pp
bykikakashi0
 
OLLD25 - The Edges of University Living Labs session
bySam Rye
 
English Learning Presentation (1).pdf....
byhalaalqrni111
 
Unit 03CIVIL SOCIETY AND DEMOCRACY IN PAKISTAN.pptx
bymuskan54863
 
AI in Tourism: transforming visitor experience & operations
byAidan Crawford
 
Powerpoint Classroom Action Research 4 yn.
byLiyana
 
tongue and its prosthodontic considerations
byFebaMariya2
 
The-Architecture-of-Counter-Ideology.pptx
bynotabolfazlpd
 
Google_CodeCamp_workshop_FrontendGDG.pdf
bymoharnabgogoi
 
Bob Stewart James 3 11 16 2025.pptx
byFamilyWorshipCenterD
 
2025-11-16 Moses 07 (shared slides).pptx
byDale Wells
 
PresentationEnglish language learning presentation file
by446814956
 
Question Generation LLM Gemini - Conference presentation - FIE 2025
bypmmorenom01
 
King Khaled university .. Presentation .
by446816119
 
A PowerPoint some information taj-mahal.pptx
byvabb1
 

Ph.D. Thesis Defense: Studying Reviewer Selection and Involvement in Modern Code Review Processes

  • 1.
    Studying Reviewer Selection andInvolvement in Modern Code Review Processes — Ph.D. Thesis Defense — Patanamon (Pick) Thongtanunam patanamon-t@is.naist.jp @pamon http://patanamon.com
  • 2.
    2 Presentation Outline Modern Code ReviewProcesses Empirical Studies Problem Statement Wrap-upThesis Structure
  • 3.
    Software code reviewis a well-established 
 software quality practice 3
  • 4.
    Software code reviewis a well-established 
 software quality practice 3
  • 5.
    Software code reviewis a well-established 
 software quality practice Poor Software Quality (e.g., Defects, Errors) 3
  • 6.
    Software code reviewis a well-established 
 software quality practice Poor Software Quality (e.g., Defects, Errors) Poor Design Quality (e.g., Spaghetti code) 3
  • 7.
    Traditionally, code reviewprocesses are formal and rigidly structured 4
  • 8.
    Traditionally, code reviewprocesses are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team 4
  • 9.
    Traditionally, code reviewprocesses are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team Performing 
 in-person meetings Time is explicitly allocated and roles are well defined 4
  • 10.
    Traditionally, code reviewprocesses are formal and rigidly structured Preparing documentations E.g. Designing checklists, forming an inspection team Performing 
 in-person meetings Time is explicitly allocated and roles are well defined Fixing the problems that are collected during 
 the meeting 4
  • 11.
    In modern softwareorganizations, 
 formal code review is hard to adopt Modern software development practices Collaboration between developers is asynchronous Teams are globally distributed 5
  • 12.
    In modern softwareorganizations, 
 formal code review is hard to adopt Modern software development practices Collaboration between developers is asynchronous Teams are globally distributed Modern Code Review (MCR): A tool-based code review process 5
  • 13.
    Gerrit Code Review Rietveld Modern CodeReview is widely adopted in industrial and open source projects 6
  • 14.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes 7
  • 15.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes Upload a patch A code review tool (e.g. Gerrit) An author 7
  • 16.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
  • 17.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
  • 18.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes Address the feedback Select reviewers Upload a patch A code review tool (e.g. Gerrit) Examine the patch An author 7
  • 19.
    Modern Code Reviewis coordinated using lightweight collaborative tools and processes Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author 7
  • 20.
    Thesis Statement: The lightweightMCR process 
 lacks mechanisms for preventing 
 lax code reviewing practices, which can allow poor quality code to slip through to official software releases 8
  • 21.
    A deeper understandingof MCR processes and their impact on software quality is needed as MCR continues to be adopted by projects worldwide 9
  • 22.
    Suboptimal reviewing practices
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author 10
  • 23.
    Suboptimal reviewing practices
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author No criteria of reviewer selection 11
  • 24.
    Suboptimal reviewing practices
 may take several forms Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Insufficient reviewer involvement 12
  • 25.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Reviewer Involvement 13
  • 26.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Reviewer Involvement 13
  • 27.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement 13
  • 28.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] 13
  • 29.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] 13
  • 30.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] 13
  • 31.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 13
  • 32.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 14
  • 33.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 14
  • 34.
    15 Module B Module A Minor authors Major authorMinor author Low code authoring expertise is associated 
 with poor software quality [Bird et al., 2011], [Rahman and Devanbu,2011] Selecting a developer to whom tasks should be delegated is important for software quality
  • 35.
    16 Reviewing expertise isimportant in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer
  • 36.
    16 Reviewing expertise isimportant in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect
  • 37.
    16 Reviewing expertise isimportant in Modern Code Review processes Module A Qt Review #27977 Author A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution
  • 38.
    16 Reviewing expertise isimportant in Modern Code Review processes Module A Qt Review #27977 Author Update A code review tool (e.g. Gerrit) Reviewer Shouldn't console.log() call the toString() method (where appropriate) on objects? Identifying a defect Reviewer I think it’s better to do var s = "{}" console.log(s) Suggesting a solution v2
  • 39.
    17 Reviewing expertise isimportant in Modern Code Review processes Module A Author A code review tool (e.g. Gerrit) Qt Review #35360
  • 40.
    17 Reviewing expertise isimportant in Modern Code Review processes Module A Author A code review tool (e.g. Gerrit) Providing updates to the code change Note that running qutlook will probably crash. I will push a patch set to fix the issues Reviewer v2 Qt Review #35360
  • 41.
    18 Yet, little isknown whether the reviewing expertise can reduce the risk of having defects Module A Minor authors Major reviewer Major reviewer
  • 42.
    19 Approximating expertise ofdevelopers using code authoring and reviewing data Minor Reviewer Major Reviewer Major Author Minor Author Code authoring expertise Code reviewing expertise The more code changes that a developer has authored or reviewed, the more expertise of that activity that developer has
  • 43.
    19 Approximating expertise ofdevelopers using code authoring and reviewing data Minor Reviewer Major Reviewer Major Author Minor Author Code authoring expertise Code reviewing expertise
  • 44.
    Code reviewing expertise Code authoring expertise VS Comparethe association with defect-proneness Minor Reviewer Major Reviewer Major Author Minor Author Analyzing the association between 
 expertise and defect-proneness 20
  • 45.
    Percentageofbothminor authors&reviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenSta Havan 020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly Open Hav nStack avana OpenStack Icehouse Defective components Cleancomponents Higher rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Reviewing expertise reverses the association between authoring expertise and defect-proneness 21
  • 46.
    Percentageofbothminor authors&reviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenSta Havan 020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly Open Hav nStack avana OpenStack Icehouse Defective components Cleancomponents Higher rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Percentageofminorauthorsbut majorreviewers(%)020406080100 Proportionofdevelopers(%) Qt 5.0020406080100 Proportionofdevelopers(%) Qt 5.0 Qt 5.1 OpenStack Folsom OpenStack Grizzly OpenStack Havana OpenStack Icehouse Lower rates in defective components NoviceExpert ExpertNovice Reviewing Expertise Authoring Expertise Median Median Reviewing expertise reverses the association between authoring expertise and defect-proneness 21
  • 47.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] 22
  • 48.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects 22
  • 49.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects Studying Reviewer Selection and Involvement in Modern Code Review Processes 23
  • 50.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects Studying Reviewer Selection and Involvement in Modern Code Review Processes 23
  • 51.
    An author Manually selectingreviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) 24
  • 52.
    An author Manually selectingreviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) Who should review this patch? 24
  • 53.
    An author Manually selectingreviewers who have reviewing expertise can be a tedious task Upload a patch A code review tool (e.g. Gerrit) Hi, can you please add appropriate reviewers for this change? Android Review #18767 Add a developer 24
  • 54.
    Reviewer selection can slowdown the code review process Studied Projects Sample Reviews 357 378 363 363 #Patch w/ reviewer selection problem 10% 30% 5% 4% Patches where the discussions are related to reviewer selection 25
  • 55.
    Reviewer selection can slowdown the code review process Studied Projects Sample Reviews 357 378 363 363 #Patch w/ reviewer selection problem 10% 30% 5% 4% Time that is spent on MCR process (Days) Reviews with code−reviewer assignment proble Reviewingtime(Days) Android OpenStack 0102030 de−reviewer assignment problem Reviews without code− OpenStack Qt Patches w/ reviewer selection problem Patches w/o reviewer selection problem Patches w/ the reviewer selection problem Patches w/o the reviewer selection problem 020406080 Android Qt OpenStack LibreOffice Studied systems Timethatisspenton MCRprocess(Days) 21 Days Longer 13 Days Longer9 Days Longer 6 Days Longer 25
  • 56.
    R1 R2 Changed Files -src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach Appropriate reviewers are those who have reviewed the most similar file paths 26
  • 57.
    R1 R2 Changed Files -src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach Appropriate reviewers are those who have reviewed the most similar file paths 26
  • 58.
    R1 R2 Changed Files -src/video/camera.java - src/video/player.java Reviewed by Shane Changed Files - src/devices/screen.java Reviewed by Pick R3 (New) Changed Files - src/video/recorder.java Reviewed by ? Review History RevFinder: A file location-based reviewer recommendation approach “Pick should review R3” 26
  • 59.
    Longest Common Prefix LongestCommon Suffix Longest Common Substring Longest Common Subsequence RevFinder uses 4 string comparison techniques 27
  • 60.
    Longest Common Prefix LongestCommon Suffix Longest Common Substring Longest Common Subsequence Reviewer recommendation Combine (Borda Count) Voting Ranks RevFinder uses 4 string comparison techniques 27
  • 61.
    Longest Common Prefix LongestCommon Suffix Longest Common Substring Longest Common Subsequence Reviewer recommendation Combine (Borda Count) Voting Ranks Bubble-up correct reviewers RevFinder uses 4 string comparison techniques 27
  • 62.
    RevFinder can accuratelyrecommend reviewers 28
  • 63.
    RevFinder can accuratelyrecommend reviewers Accuracy Top-10Accuracy 0% 30% 60% 90% 10% 28% 41% 29% 74% 69% 87%86% RevFinder Baseline Approach 28
  • 64.
    RevFinder can accuratelyrecommend reviewers Accuracy Top-10Accuracy 0% 30% 60% 90% 10% 28% 41% 29% 74% 69% 87%86% RevFinder Baseline Approach Ranking Performance MeanReciprocalRank 0 0.2 0.4 0.6 0.07 0.22 0.3 0.25 0.4 0.31 0.55 0.6 RevFinder Baseline Approach 28
  • 65.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects 29
  • 66.
    Studying Reviewer Selectionand Involvement in Modern Code Review Processes Reviewer Selection Software quality Tool Support Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch 29
  • 67.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 30
  • 68.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 30
  • 69.
    A lack ofcode review increases the risk of having defects Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (E.x. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 31
  • 70.
    A lack ofcode review increases the risk of having defects Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (E.x. Gerrit) Examine the patch An author Integrate a clean patch into software repositories The number of patches without reviewer involvement is associated with the risk of having defects [McIntosh et al., 2015] 31
  • 71.
    A low levelof reviewer involvement may introduce in Modern Code Review process Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 32
  • 72.
    A low levelof reviewer involvement may introduce in Modern Code Review process Address the feedback Select reviewers Upload a patch Integrate a clean patch into software repositories A code review tool (e.g. Gerrit) Examine the patch An author Integrate a clean patch into software repositories 32
  • 73.
    Investigating the differencebetween code review practices of defective and clean files 33
  • 74.
    Review Practice A Defective i.e.,files that have defects Investigating the difference between code review practices of defective and clean files 33
  • 75.
    Defect-free (Clean) i.e., filesthat do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 33
  • 76.
    Defect-free (Clean) i.e., filesthat do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 33
  • 77.
    Review Intensity e.g., #ReviewIterations, Discussion Length Review Participation e.g., #Reviewers, 
 Review Agreement Reviewing Time e.g., Review Length, Code Reading Speed Characterizing reviewing activity using 11 metrics grouped along 3 dimensions 34
  • 78.
    Defect-free (Clean) i.e., filesthat do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 35
  • 79.
    Defect-free (Clean) i.e., filesthat do not have defects Review Practice BReview Practice A VS Defective i.e., files that have defects Investigating the difference between code review practices of defective and clean files 35
  • 80.
    Defective i.e., files thathave defects Studying defective files along two perspectives 36
  • 81.
    Defective i.e., files thathave defects Studying defective files along two perspectives 36
  • 82.
    Risky Files Files thathave historically been defective Past Defective i.e., files that have defects Studying defective files along two perspectives 36
  • 83.
    Risky Files Files thathave historically been defective Past Future Defective i.e., files that have defects Future-Defective Files Files that will eventually have defects Studying defective files along two perspectives 36
  • 84.
    Risky Files Files thathave historically been defective Past Future Defective i.e., files that have defects Future-Defective Files Files that will eventually have defects Studying defective files along two perspectives 36
  • 85.
    Risky Files Files thathave historically been defective Past Future Defective i.e., files that have defectsFuture-Defective Files Files that will eventually have defects Conjecture: Reviews of Future- Defective will be • less intense, • with less team participation, • completed with a shorter time than reviews of clean files Studying defective files along two perspectives 36
  • 86.
    #Reviewers #Reviews of Cleanfiles #Reviewers #Reviews of Future-Defective files VS Using a statistical test to determine the difference between the distributions of code review activity Comparing code review activity that has been applied to future-defective and clean files 37
  • 87.
    FindingsConjecture Review Intensity Review Participation ReviewingTime Less intense Less team participation Completed within a shorter time Reviewing activity in future-defective files are less rigorous 38
  • 88.
    FindingsConjecture Shorter discussion Fewer reviewers Receive slow initial feedback ReviewIntensity Review Participation Reviewing Time Less intense Less team participation Completed within a shorter time Reviewing activity in future-defective files are less rigorous 38
  • 89.
    Defective i.e., files thathave defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Conjecture: Reviews of Future- Defective will be • less intense, • with less team participation, • completed with a shorter time than reviews of clean files Studying defective files along two perspectives 39
  • 90.
    Defective i.e., files thathave defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
  • 91.
    Defective i.e., files thathave defects Past Risky Files Files that have historically been defective FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
  • 92.
    Defective i.e., files thathave defects Past Risky Files Files that have historically been defective Conjecture: Reviews of risky files should be • more intense, • with more team participation, • reviewed for a longer time to reduce the risk of having defects in the future FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Studying defective files along two perspectives 39
  • 93.
    FindingsConjecture Review Intensity Review Participation ReviewingTime More intense More team participation Take longer time to complete Reviewing activities in risky files are not careful 40
  • 94.
    FindingsConjecture Shorter discussion Fewer reviewers Receive Slow Feedback Review Intensity ReviewParticipation Reviewing Time More intense More team participation Take longer time to complete Reviewing activities in risky files are not careful 40
  • 95.
    Defective i.e., files thathave defects Past FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Risky Files Files that have historically been defective Conjecture: Reviews of risky files should be • more intense, • with more team participation, • reviewed for a longer time to reduce the risk of having defects in the future Studying defective files along two perspectives 41
  • 96.
    Defective i.e., files thathave defects Past FutureFuture-Defective Files Files that will eventually have defects Reviews of future-defective files tend to be less rigorous than reviews of clean files Risky Files Files that have historically been defective Developers are not as careful when they review risky files Studying defective files along two perspectives 41
  • 97.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch Studying Reviewer Selection and Involvement in Modern Code Review Processes 42
  • 98.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 42
  • 99.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 43
  • 100.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 43
  • 101.
    A code reviewtool (e.g. Gerrit) Poor reviewer involvement has an impact on software quality Integrate a clean patch into software repositories 44
  • 102.
    A code reviewtool (e.g. Gerrit) Investigating the characteristics of patches that suffer from poor reviewer involvement Upload a patch This patch is likely to suffer from poor reviewer involvement Select reviewers An author 45
  • 103.
    Studying poor reviewerinvolvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time 46
  • 104.
    Studying poor reviewerinvolvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation 46
  • 105.
    Studying poor reviewerinvolvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation No review discussion 46
  • 106.
    Studying poor reviewerinvolvement along three perspectives Reviewer involvement in defective files (S3) Review Intensity Review Participation Reviewing Time No reviewer participation No review discussion Slow initial feedback 46
  • 107.
    Properties e.g., Size, Description lengthof code changes History e.g., Past tendency, #days since the last modification Environment e.g., Review workload Past Involvement e.g., Past involvement of an author and reviewers Measuring characteristics of patches using 20 metrics grouped along 4 dimensions 47
  • 108.
    48 Measure patch characteristics STEP 1 Trainingstatistical models to explain the relationship History EnvironmentProperties Past Involvement
  • 109.
    49 Measure patch characteristics STEP 1 Traina prediction model STEP 2 x Assess the risk of receiving poor reviewer involvement Relax the linear assumption of the relationship Non-linear logistic regression model Probability x x x x Training statistical models to explain the relationship
  • 110.
    50 Measure patch characteristics STEP 1 Traina prediction model STEP 2 STEP 3 Assess the fitness of the model Discrimination Ability (AUC) Stability (AUC Optimism) Model performance Training statistical models to explain the relationship
  • 111.
    51 Measure patch characteristics STEP 1 Traina prediction model STEP 2 STEP 3 Assess the fitness of the model Interpret model STEP 4 Explanatory power (Wald χ2 statistics) Plot the relationship with the risk of receiving poor reviewer involvement Observe code ownership variables Increasing Relationship Inverse Relationship or Training statistical models to explain the relationship
  • 112.
    Our statistical modelsare accurate and stable enough for interpretation No reviewer participation No review discussion Slow initial feedback 0.72 0.70 0.74 AUC Optimism AUC 0.002 0.001 0.001 0.70 0.72 0.78 AUC Optimism AUC 0.002 0.001 0.001 0.66 0.61 0.61 AUC Optimism AUC 0.004 0.002 0.001 52
  • 113.
    History Past activity tendency Oddsofreceivingpoor reviewerinvolvement Properties Descriptionlength Patches whose prior changes received poor reviewer involvement, and that are not descriptive are likely to receive poor reviewer involvement History and properties of patches share a strong relationship with poor reviewer involvement Oddsofreceivingpoor reviewerinvolvement 53
  • 114.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes 54
  • 115.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch The low level of reviewer involvement can lead to poor software quality Studying Reviewer Selection and Involvement in Modern Code Review Processes Past tendency and a patch description are strong indicators of poor reviewer involvement 54
  • 116.
    55 • (S1) RevisitingCode Ownership and its Relationship with Software Quality in the Scope of Modern Code Review. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Ha- jimu Iida. 
 In Proceedings of the 38th International Conference on Software Engineering (ICSE), 2016, pp. 1039–1050. • (S2) Who Should Review My Code? 
 Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, Kenichi Matsumoto. 
 In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2015, pp. 141–150. • (S3) Investigating Code Review Practices in Defective Files. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 
 In Proceedings of the 12th International Conference on Mining Software Repositories (MSR), 2015, pp. 168–179.
 This paper won the IEEE Kansai Excellent Student Paper Award 2015 • (S4) Review Participation in Modern Code Review. 
 Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 
 Springer Journal of Empirical Software Engineering (EMSE), 39 pages. Under review. Related Publications
  • 117.
    Thesis Structure Modern CodeReview Process Chapter 2 Code Review Process Part I The Modern Code Review Process Chapter 3 Related Research Part II Reviewer Selection Part III Reviewer Involvement Empirical studies Chapter 4: The Impact of Reviewer Selection on Software Quality Chapter 6: The Impact of Review Involvement on Software Quality Chapter 5: Selecting Appropriate Reviewers Chapter 7: Identifying Patches with Poor Review Involvement Chapter8 ConclusionandFutureWork Chapter1 Introduction 56
  • 118.
  • 119.
    Gerrit Code Review Rietveld Modern CodeReview is widely adopted in industrial and open source projects
  • 120.
    Thesis Statement: The lightweightMCR process 
 lacks mechanisms for preventing 
 lax code reviewing practices, which can allow poor quality code to slip through to official software releases
  • 121.
    A deeper understandingof MCR processes and their impact on software quality is needed as MCR continues to be adopted by projects worldwide
  • 122.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes
  • 123.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects
  • 124.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015
  • 125.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects The low level of reviewer involvement can lead to poor software quality RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015
  • 126.
    Reviewer Selection Software quality ToolSupport Reviewer Involvement S1: The Impact of Reviewer Selection 
 on Software Quality
 [ICSE 2016] S2: Selecting Appropriate Reviewers [SANER 2015] S3: The Impact of Reviewer Involvement on Software Quality
 [MSR 2015] S4: Identifying Patches with Poor Reviewer Involvement
 [EMSE under major revision] Studying Reviewer Selection and Involvement in Modern Code Review Processes The more reviewing experts the module has, the less likely that the module will have defects The low level of reviewer involvement can lead to poor software quality RevFinder can accurately recommend an appropriate reviewer for a new patch SANER 2015 Past tendency and a patch description are strong indicators of poor reviewer involvement EMSE (Under Major Revision)