Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Richard Wettel, profile picture
Uploaded byRichard Wettel
8,109 views

Software Systems as Cities: a Controlled Experiment

This document describes a controlled experiment to evaluate the Software Systems as Cities visualization tool CodeCity. The experiment involves participants completing program comprehension and design quality assessment tasks on medium and large software systems using either CodeCity or traditional tools like Eclipse. The main research questions are whether CodeCity increases task correctness and reduces time compared to traditional tools, regardless of system size. Key variables that are measured include task correctness, completion time, tool used, system size, participant experience level and background.

Technology◦
Downloaded 64 times
Software Systems as Cities: A Controlled Experiment Richard Wettel, Michele Lanza Romain Robbes REVEAL @ Faculty of Informatics PLEIAD @ DCC University of Lugano University of Chile Switzerland Chile
Software Systems as Cities
City Metaphor VISSOFT 2007
City Metaphor class building package district VISSOFT 2007
City Metaphor class building package district VISSOFT 2007
City Metaphor class building package district nesting level color VISSOFT 2007
City Metaphor number of methods (NOM) height number of attributes (NOA) base size number of lines of code (LOC) color class building package district nesting level color VISSOFT 2007
Program Comprehension ArgoUML LOC 136,325 ICPC 2007
Program Comprehension
FacadeMDRImpl NOA 3 skyscraper NOM 349 LOC 3,413
CPPParser NOA 85 office building NOM 204 LOC 9,111
JavaTokenTypes NOA 173 parking lot NOM 0 LOC 0
house PropPanelEvent NOA 2 NOM 3 LOC 37
Program Comprehension ICPC 2007
Design Quality Assessment disharmony map ArgoUML classes brain class 8 god class 30 god + brain 6 data class 17 unaffected 1,715 SoftVis 2008
System Evolution Analysis time traveling WCRE 2008
System Evolution Analysis time ArgoUML time traveling 8 major releases 6 years WCRE 2008
System Evolution Analysis time ArgoUML time traveling 8 major releases 6 years WCRE 2008
http://codecity.inf.usi.ch implemented in Smalltalk ICSE 2008 tool demo
Is it useful ?
A Controlled Experiment
Design
technical report 2010 State of the art?
Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Provide a not-so-short tutorial of the experimental tool to the participants. 4 Avoid, whenever possible, giving the tutorial right before the experiment. 5 Use the tutorial to cover both the research behind the approach and the tool. 6 Find a set of relevant tasks. 7 Choose real object systems that are relevant for the tasks. 8 Include more than one object system in the design. 9 Provide the same data to all participants. 10 Limit the amount of time allowed for solving each task. 11 Provide all the details needed to make the experiment replicable. 12 Report results on individual tasks. 13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated. 14 Take into account the possible wide range of experience level of the participants.
Design desiderata 1 Avoid comparing using a technique against not using it. 2 Involve participants from the industry. 3 Provide a not-so-short tutorial of the experimental tool to the participants. 4 Avoid, whenever possible, giving the tutorial right before the experiment. 5 Use the tutorial to cover both the research behind the approach and the tool. 6 Find a set of relevant tasks. 7 Choose real object systems that are relevant for the tasks. 8 Include more than one object system in the design. 9 Provide the same data to all participants. 10 Limit the amount of time allowed for solving each task. 11 Provide all the details needed to make the experiment replicable. 12 Report results on individual tasks. 13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated. 14 Take into account the possible wide range of experience level of the participants.
Finding a baseline 1. program comprehension 2. design quality assessment 3. system evolution analysis
Finding a baseline 1. program comprehension 2. design quality assessment 3. system evolution analysis
Finding a baseline 1. program comprehension 2. design quality assessment 3. system evolution analysis
Finding a baseline 1. program comprehension 2. design quality assessment
Tasks
Tasks program comprehension 6 A1 Identity the convention used in the system to organize unit tests. A2.1& What is the spread of term T in the name of the classes, their attributes and A2.2 methods? A3 Evaluate the change impact of class C, in terms of intensity and dispersion. A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code per A4.2 method.
Tasks program comprehension 6 A1 Identity the convention used in the system to organize unit tests. A2.1& What is the spread of term T in the name of the classes, their attributes and A2.2 methods? A3 Evaluate the change impact of class C, in terms of intensity and dispersion. A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code per A4.2 method. B1.1 Identify the package with the highest percentage of god classes. B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-level B2.1 design problem. B2.2 Write an overview of the class-level design problems in the system. design quality assessment 4
Tasks program comprehension 6 5 A1 Identity the convention used in the system to organize unit tests. A2.1& What is the spread of term T in the name of the classes, their attributes and A2.2 methods? A3 Evaluate the change impact of class C, in terms of intensity and dispersion. A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code per A4.2 method. B1.1 Identify the package with the highest percentage of god classes. B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-level B2.1 design problem. B2.2 Write an overview of the class-level design problems in the system. design quality assessment 4
Tasks quantitative 9 8 A1 Identity the convention used in the system to organize unit tests. A2.1& What is the spread of term T in the name of the classes, their attributes and A2.2 methods? A3 Evaluate the change impact of class C, in terms of intensity and dispersion. A4.1 Find the three classes with the highest number of methods. Find the three classes with the highest average number of lines of code per A4.2 method. B1.1 Identify the package with the highest percentage of god classes. B1.2 Identify the god class with the largest number of methods. Identify the dominant (affecting the highest number of classes) class-level B2.1 design problem. B2.2 Write an overview of the class-level design problems in the system. qualitative 1
Main research questions
Main research questions 1 Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
Main research questions 1 Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size? 2 Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?
Variables of the experiment
Variables of the experiment correctness dependent completion time CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced controlled academia background industry
Variables of the experiment correctness dependent completion time CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced controlled academia background industry
Variables of the experiment correctness dependent completion time CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced controlled academia background industry
Variables of the experiment correctness dependent FindBugs completion time 1,320 classes 93,310 LOC CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced Azureus controlled academia 4,656 classes background industry 454,387 LOC
Variables of the experiment correctness dependent completion time CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced controlled academia background industry
Variables of the experiment correctness dependent completion time CodeCity tool Eclipse + Excel independent medium object system size large beginner experience level advanced controlled academia background industry
The experiment’s design between-subjects randomized-block
The experiment’s design between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
The experiment’s design background academia industry experience beginner advanced beginner advanced B1 B2 B3 B4 between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
The experiment’s design background academia industry experience beginner advanced beginner advanced B1 B2 B3 B4 between-subjects randomized-block CodeCity T1 large Tool T2 medium Size Ecl+Excl T3 large T4 medium
Execution
Experimental runs
Experimental runs day 1 time training session (1 hour)
Experimental runs day 1 time training session (1 hour) e1 experiment session (2 hours) c1
Experimental runs day 1 day 2 time training session (1 hour) e1 e2 experiment session (2 hours) c1 c2
Experimental runs day 1 day 2 day 3 time training session (1 hour) e1 e2 e3 experiment session (2 hours) c1 c2
Experimental runs day 1 day 2 day 3 day 4 time training session (1 hour) e1 e2 e3 experiment session (2 hours) c1 c2 c4
Testing the waters 2009 November December 18 24 25 2 9 Lugano 1 3 1 1 1 1 1
Timeline of the experiment 2009 2010 November December January February ... April 18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14 Lugano 1 3 1 1 1 1 1 1 1 1 3 Bologna 2 1 6 1 1 1 1 Antwerp 5 6 Bern 4 1 6
Timeline of the experiment 2009 2010 November December January February ... April 18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14 Lugano 1 3 1 1 1 1 1 1 1 1 3 remote sessions Bologna 2 1 6 1 1 1 1 Antwerp 5 6 remote session Bern 4 1 6
Treatments and subjects academia industry beginner advanced advanced large 2 2 6 10 CodeCity medium 3 2 7 12 large 2 3 3 8 Ecl+Excl medium 2 5 4 11 9 12 20 41
Collecting raw data
Collecting raw data solution
Collecting raw data completion time
Controlling time
Controlling time common time
Controlling time common time info on subjects Name (Task): Remaining time
Assessing correctness 1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2 A1 Multiple locations. The god class containing the largest number of methods in the system is There are 40/41 [0.5pts] classes class MainFrame [0.8pts] Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts] which contains 119 [0.1pts] methods. • edu.umd.cs.findbugs A2.1 • edu.umd.cs.findbugs.bcel B2.1 Localized [0.5pts] in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is DataClass [0.5pts] A2.2 A4.1 which affects a number of 67 [0.5pts] classes. Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]: 3 6 in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor • edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba contains 195 methods; • edu.umd.cs.findbugs.anttask 2. class MainFrame • edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2 contains 119 methods; • edu.umd.cs.findbugs.ba.deref 3. class BugInstance • edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs • edu.umd.cs.findbugs.ba.npe contains 118 methods or • edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor defined in package edu.umd.cs.findbugs.ba.type • edu.umd.cs.findbugs.bcel contains 118 methods. • edu.umd.cs.findbugs.classfile A4.2 • edu.umd.cs.findbugs.classfile.analysis The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each • edu.umd.cs.findbugs.classfile.engine 3 6 misplaced]: • edu.umd.cs.findbugs.classfile.impl 1. class DefaultNullnessAnnotations • edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba has an average of 124 lines of code per method; • edu.umd.cs.findbugs.cloud.db 2. class DBCloud.PopulateBugs • edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db has an average of 114.5 lines of code per method; • edu.umd.cs.findbugs.gui 3. class BytecodeScanner • edu.umd.cs.findbugs.gui2 defined in package edu.umd.cs.findbugs.ba • edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method. • edu.umd.cs.findbugs.model B1.1 • edu.umd.cs.findbugs.visitclass oracles The package with the highest percentage of god classes in the system is • edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts] which contains 1 [0.1pts] god classes out of a total of 3 [0.1pts] classes.
Assessing correctness blinding 1 T2: Findbugs, analyzed with CodeCity A3: Impact Analysis B1.2 A1 Multiple locations. The god class containing the largest number of methods in the system is There are 40/41 [0.5pts] classes class MainFrame [0.8pts] Dispersed. [1pt] defined in the following 3 packages [1/6pts for each]: defined in package edu.umd.cs.findbugs.gui2 [0.1pts] which contains 119 [0.1pts] methods. • edu.umd.cs.findbugs A2.1 • edu.umd.cs.findbugs.bcel B2.1 Localized [0.5pts] in package edu.umd.cs.findbugs.detect [0.5pts]. • edu.umd.cs.findbugs.detect The dominant class-level design problem is DataClass [0.5pts] A2.2 A4.1 which affects a number of 67 [0.5pts] classes. Dispersed The 3 classes with the highest number of methods are [ 1 pts each correctly placed and 1 pts each misplaced]: 3 6 in the following (max. 5) packages [0.2pts for each]: 1. class AbstractFrameModelingVisitor • edu.umd.cs.findbugs defined in package edu.umd.cs.findbugs.ba contains 195 methods; • edu.umd.cs.findbugs.anttask 2. class MainFrame • edu.umd.cs.findbugs.ba defined in package edu.umd.cs.findbugs.gui2 contains 119 methods; • edu.umd.cs.findbugs.ba.deref 3. class BugInstance • edu.umd.cs.findbugs.ba.jsr305 defined in package edu.umd.cs.findbugs • edu.umd.cs.findbugs.ba.npe contains 118 methods or • edu.umd.cs.findbugs.ba.vna class TypeFrameModelingVisitor defined in package edu.umd.cs.findbugs.ba.type • edu.umd.cs.findbugs.bcel contains 118 methods. • edu.umd.cs.findbugs.classfile A4.2 • edu.umd.cs.findbugs.classfile.analysis The 3 classes with the highest average number of lines of code per method are [ 1 pts each correctly placed and 1 pts each • edu.umd.cs.findbugs.classfile.engine 3 6 misplaced]: • edu.umd.cs.findbugs.classfile.impl 1. class DefaultNullnessAnnotations • edu.umd.cs.findbugs.cloud defined in package edu.umd.cs.findbugs.ba has an average of 124 lines of code per method; • edu.umd.cs.findbugs.cloud.db 2. class DBCloud.PopulateBugs • edu.umd.cs.findbugs.detect defined in package edu.umd.cs.findbugs.cloud.db has an average of 114.5 lines of code per method; • edu.umd.cs.findbugs.gui 3. class BytecodeScanner • edu.umd.cs.findbugs.gui2 defined in package edu.umd.cs.findbugs.ba • edu.umd.cs.findbugs.jaif has an average of 80.75 lines of code per method. • edu.umd.cs.findbugs.model B1.1 • edu.umd.cs.findbugs.visitclass oracles The package with the highest percentage of god classes in the system is • edu.umd.cs.findbugs.workflow edu.umd.cs.findbugs.ba.deref [0.8pts] which contains 1 [0.1pts] god classes out of a total of 3 [0.1pts] classes.
Results
Statistical test two-way analysis of variance (ANOVA) 95% confidence interval
Correctness Ecl+Excl CodeCity Does the use of CodeCity increase the 8 correctness of the solutions to program comprehension tasks, compared to non- 7 visual exploration tools, regardless of the 6 object system size? 5 4 3 2 1 0 medium large
Correctness Ecl+Excl CodeCity Does the use of CodeCity increase the 8 correctness of the solutions to program comprehension tasks, compared to non- 7 visual exploration tools, regardless of the 6 object system size? 5 24.26% 4 3 2 more correct with CodeCity 1 large effect size (d=0.89) 0 medium large
Correctness Ecl+Excl CodeCity Does the use of CodeCity increase the 8 correctness of the solutions to program comprehension tasks, compared to non- 7 visual exploration tools, regardless of the object system size? 6 24.26% 5 4 more correct with CodeCity 3 large effect size (d=0.89) 2 medium large
Completion time Ecl+Excl CodeCity Does the use of CodeCity reduce the time 60 needed to solve program comprehension tasks, compared to non-visual exploration 50 tools, regardless of the object system size? 40 30 20 10 0 medium large
Completion time Ecl+Excl CodeCity Does the use of CodeCity reduce the time 60 needed to solve program comprehension tasks, compared to non-visual exploration 50 tools, regardless of the object system size? 40 12.01% 30 20 less time with CodeCity 10 moderate effect size (d=0.63) 0 medium large
Completion time Ecl+Excl CodeCity Does the use of CodeCity reduce the time 60 needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size? 50 12.01% 40 30 less time with CodeCity moderate effect size (d=0.63) 20 medium large
after the first round CodeCity vs Ecl+Excl +24% correctness -12% completion time
193 a riteri nce ing C ie Block Exper d Replicability nd nce grou adva Back nced ent size indust ry adva Tr eatm S ystem nced indust ry adva Tool large nced b er ity in dustry adva Code Num CodeC y large ustry begin ner 1 it ind CodeC y large y ginner IA01 1 it large academ y be nced CodeC y adva IA02 1 odeC it la rge academ nced C ry adva IA03 1 ity large indust y nced CodeC y adva 4 1 it adem IA0 CodeC y large ac y nced 1 adem adva AB01 CodeC y it large ac vance d 1 indu stry ad AB02 CodeC y it large ry vance d IA05 1 CodeC y it large indust ad nced A01 1 eCit ium indust ry adva A Cod med ner 02 1 ity ium in dustry begin d AA CodeC y med y nce 2 m academ adva IA06 it re CodeC y mediu nced su indust ry adva res IA07 2 it m CodeC y mediu nced fair amo unt eP too amo unt indust ry adva very muc unt fair amo 2 m Tim IA08 it odeC mediu begin d ner fair litt h C ry very so m unt fair indust y no amo le eCity litt uch IA09 2 edium Cod m nce no amo uch adva fair so m le ity academ y very so m unt 03 2 ium iate t so ttle h AB CodeC y med ner uc begin d t litt uch adem ed IA10 2 it m inte ficult le t mediu ac rm .2 CodeC y no nce sim ple diate no amo uch m dif ossib li fair so m le y B2 adva t inte t academ y un IA11 2 it m mediu nced sim rme CodeC y p very no inte cult iate fair so m uch im adva no so m le academ no amo uch t 2 m triv ple litt it no IA12 dif rme diate mediu d nced t very e CodeC y un dif rme inte cult diate n ial ry adva indust too amo unt 2 m dif ficult diate inte rme AB04 it inte fair muc unt t CodeC y mediu no .1 ner fi ial fair amo t begin B2 litt unt fair e ry triv h triv ple indust y dif rme n A03 2 eCit ium sim ial fi ner very amo A med sim le ple Cod dif ficult begin d no amo unt dif ficult adem triv ial fair so m unt 5 2 ity sim ficult fair amo uch triv ple diate AB0 sim ial fair amo CodeC l large ac triv ple nce inte cult le no amo unt fair e emy sim ple adva dif rme diate n sim ial dif ossib t im ple inte ple t so un no inte ficult diate 2 very amo uch sim rme AA04 Exc acad nced triv large .2 sim ple diate litt unt ial Ecl+ inte ficult diate inte rme y B1 m adva p t triv academ y sim ple no so m uch fi Excl ial IA13 3 no so m le inte ple dif ossib diate sim ple no so m uch nced sim rme large sim ple dif rme no so m uch Ecl+ adva sim dif ple mu ch triv ficult academ y dif cult le fair Excl sim ple 3 im rme ch IA14 u large nced sim triv ple inte rme iate sim ple diate sim ial t Ecl+ no sim ple adva inte rme diate triv ial t academ sim ple Excl im ficult diate ult inte ficult inte rme diate dif ial d 3 p re 4– –3 triviaial >1 –6 inteterme ediate d intermssible ult t B06 t so e nced inte ple im cult inte rme rm diate .1 fi sim ple diate sim rme t A larg fic Ecl+ fic sim rme diate B1 dustry adva e im ple diate dif ple fi dif ossib le xcl dif ficult dif po 3 ult le 7 inte ple dif AB0 sim rme E in nced inte ficult le large ult im ossib dif ossib sim dif ossib diate e Ecl+ inte ficult le ustry adva fic te inte rme diate te sim rme le l sk dif ossib triv ple 3 if AA05 ind inte rme iate dia Exc ult le p dia inte ossib d p large im rme fic A4 er Ta rm diate vance p triv ial inte im Ecl+ le sim ple diate inte rme inte rme diate sim rme inte ial inte rme diate d ry p ssib Excl indust ad inte rme diate te 3 p im AA06 sim ple large nced dif ple P .2 sim rme diate p triv ficult diate inte ficult im ficult diate dia fic dif rme le inte ple Ecl+ po sim rme triv ple diate e inte rme el im ficult ry adva inte ossib 1– 0 sim rme diate xcl indust ev im AA07 3 edium dif ple diate dif rme cl+E ner im ossib le te g. in ple yL te begin dif inte E m inte ossib le 4 3 in rm p im ossib ry dia im ficult iate indust y rm le ult 3 Excl ium inte rme iate 15 te triv ial ner sim rme te im rme iate dif ficult inte ial IA dif ial med A4 iffic ed p Ecl+ begin d dif4 <1 1 trivia cult inte ult sim ossib diate sim 4– le 0 diateE of tr mple edia d sim rme diate p inte ficult le triv ple Excl academ y inte rme ssib te d sim ple diate IA16 4 m p ssib fic D mediu l 1– in6mple R diate .1 fi ple ub k adva owlle er eable simp rmed7ia1 10 sim7–rmea ber siinterm l 3 mp edia .En dia dif ple le nce ia Ecl+ inte rme 1– adva dif ficult le po sim 1– si3 term ev academ y im rme dif sim ial diate 4 Excl m 4–si etermse IA01 inte rme diate pin ar l mediu nced cli Ye ia inte im rme iate sim ficult dif Ecl+ p iv adva po < 3 dif oss iate inte ple dif ficult e em y im ossib le Excl inte ple rm inte rme 4 m acad im ficult d IA18 < 6 ult dif ossib le nced im ossib mediu le s dif d sim rme diate 7– OP sim ficult ible inte cult le Ecl+ 1 6 triv le y sim ple adva dif ficult diate dif ple diate 4– 6 simp academ y fic triv sim ple 4 Excl m p IA19 mediu nced – l inte ple inte fiault um ple inte rme p 1 1 Ecl+ sim rme diate e inte rme diate A3 p sim ficult adva p 10 d ple rm if J N sim academ y sim ple diate 7– –3 triv rme diate fi l 4– B08 4 +Exc ium v sim ple inte ficult nced inte ial diate inte ult sim ple diate 4– 0 sk c A med triv ple inte rme inte rme 1 Ecl dif 1 adva fic < 6 dif inte rme diate 6 academ y 1– d ta sim ial Excl 7–mp–3le p 4 ium ple 09 inte rme iate an– 6 sim rme iate nced 4– 1 sim10 le si 1 p AB inte rme iate med sim rme diate Ecl+ dif –6 ple 3 sim ple 7– triv3ficult 1– 6 adva inte rme triv rme diate sim ple diate d triv ple diate inte ed adem si ple O 1 sim ple l 4– 3 te 4 d m 4 3 AA1 0 Exc le sim l kn gin ng. si mple mediu ac 10 ial inte ple inte ple > –10 rm .2 inte rme ial dia 1– 6 triv plen ow ner si mple 7– –6 4 Ecl+ triv ial diate A2 sim rme stry 1– cks inte sim ialeg owle dg tr mple 1– 3 7 > 10 dif ple diate AA11 4 Excl m indu 0 su triv le 4 te sim ple inn dg eab ivial mediu d blo < 3 10 inte sim pele inn er eab mple pr – 10 dif ple > e3 s triv rme inte ficult Ecl+ – –6 triv ial nts an 1 1 si le < 1 sim ial > sim ficult diate sim rme sim ial Excl < ad gin led d able trivia 7l 1–3 4 m ple AA12 7– –3 te le inte iate 1– 1 1 mediu triv 7– 10 atme triv ple ial b ev.E inte ficult sim ial > 10 ple dia Ecl+ 1 0 sim ple 3 dif rme 4 0 im rme diate 1 sim ial triv ple diate l – to tre 10 ed tr ple 1 sim l inte ple 7– 10me > –3 le inte ssib te ainte va ssib rte able iv ple R 4 sim rme 7– <1 AA13 ia +Exc 7– –6 >ti 6 rm .1 im norminn dg eabsi ivial dia e 1– 0 ed 4– cts 1– 0 inte ple Ecl 1 0 knintea nce ia te er triv cult iate inte rme 1 A2 sim rme 1 dif rme le 1 inte subje 4– 3 sim ple < 3 7– –3 lem triv no inn er AA14 4 4 0 sim ple d 1– 6 te k si g rmc eiate b trivia 1 of the < 1 inte k sim 7– 10 sim ple triv gin le r 7– –6 w e triv ial tr po 1 3 n dg sim ple 4– 1 te le d ficu lip le el b > 10 b g sim ial bleeiv ment 4 0 triv ple be triv ple dif Essibev IA20 –3 sim kn ult wle e te 1 4– 6 intebeple wle dg 1– 0 kp g iffi knolt s po lt be cts nce ge dif ial > –6 triv ial fi 1 ad po w edia e triv ial im ficuL ial dia 1– 3 beia le ssign 1– 6 be va ne gearc 4– 3 sim ple diate sim ial – ial d e 7– 10 dif ple 1 c inte ple kn ia l nc sim ficult 1– 3 triv rme 4– 3 ea ult 1– 6 be tevan ncd legea e > 10 e ie .2. Th 1 3 ’p 1– 6 sim ple 4– 3 fic sim rme o A1 sim ficult in d rme d 10 c kn dif w plee dd te sim ple 7– –3 le 1– 3 4– 6 d dif ad tevrm edia Etep kn gin nce r k g A 7– –6 w sim ple ple no m inn ee ia x te 4– 0 inte be ow nce d able Table ad ow ne d ad pe ne d able ab k egin tece dge le able < 3 7– 6 sim ple jeno sim ple 1 4 in d rm dia 4– 0 dif ial dia triv ial sim ple 1 6 sim ple 1 ad intermncedia te de le now ne rmediaable 4– 1 1– 0 triv ficult iate sim ial ad van led r ad imw ficled r 4 6 kn va ce ge inte ple te triv ple b inan poss ge 1 inte ple inte v rme dsi ten dg ia te te 7– –3 triv rme < 6 Co sim 1– 3 ow rm e d v le lt IA0 1 va rme de te 7– –6 ed ple dia ib triv ial d 10 b IA0 u ble a 1– 1 IA0 2 3 sim ple diate dif rme sim ial va le e d iate adv mp d ia a 7– 10 triv inte ple IA0 3 eles 4– 6 sim rme Ja be gin led d 1 3 a be gin ne gea d 7– 10 4– AB 4 4– 0 ce d b AB 01 inte 4 6 1 1 3 – 7– 10 kn pe in cmple edia te Tha sim rme o sim ficult a be ne.5.led r dif ial v – 7 02 4– 6 ge te 7– –6 kn pe led d simble l diatex dva si rt term edia > –3 > 10 ex gin ne r ex gin tencple d le AA 5 7– 6 ple te p in nc rm g iate be sim nce AA 01 1– 0 inte ta IA0 1– 0 ble 4 0 sim ple sim ple be pe ne r a pein nermd ad an te ded iate Da le 1 1 7– 10 IA0 02 te a in 1 e Osi ple 1– 3 sim ple ial dia adv ansimed ed te inte ple diate adv inrt term edab be van letrivia edia A.5 Data 1– 3 7– –6 dia ex va te led ed ad va ne gea be gin rt r ad gin led r sim r – 0 kn inOP ple IA0 6 triv ple ete a si ce ple iate be ow ne te c rm ia A.5 4– 3 e e 7 1 noA 1– 3 a inte rme iate ad in wterm 4 0 7 > 10 ial dia triv ial kn gin 1 ad ow rt term m 4– 6 IA0 8 3 7– –6 inte ple diate ad ow ne d leble ed sim ial kn gin ne r ad va nce r IA0 4 0 be gin ne AB 9 inte ple kn gin cesimgea l triv rme 4– 6 le 1 IA1 03 kn va nce d 1– 0 ex ow ne r 7– –6 n inte ple diate o sim ple ex per t in mple edia ab 1 4– 6 p IA1 0 ad ow led ge ble kn ow nce d 1– 3 sim rme kn pe led r ge 4 0 d ex pe nce mple IA1 1 T 4– 6 dif ial diate ble 1 ad va led ge able d 4– 3 kn ow led gea e sim ficult ex pe rtsi term AB 2 d > –6 sim rme mp ed AA 04 7– 6 inte ad va nce ge able be ow nce gea 4 6 e in d 03 kn ow led be va led r 1– 0 triv ple kn per t si term ex gin nce ge 10 ble e 1 ad va nce d able > –6 AA 05 ad ow rt kn va led rm ad ow led simp le rm 4– 3 v 04 gin led d ad gin ne gea knedia n led ginte le 4– 0 195 AB ex ow tece geaivable e CodeCity Ex inte ficult d 4– 6 1 tr rm IA1 3 inte rme 4– 6 IA1 triv ad va rt r 4 6 AB 4 r 06 ia x 7– 6 ple b va ne r periment sim rme > –6 AA 07 e 10 kn va nce d ad va ce d AA 05 kn gin nce r 4– 0 ial kn ow t AB ad van nce triv rme kn va led d able 1 AA 06 ad ow ne d kn ow led d able ad ow nce d kn va nce d le 4 6 IA1 07 ab kn van led d ad va nce ge able kn ow nce d kn va led r > –6 ad ow nce ge ge 5 be dif ple ge be ow led d able ad ow led ge able kn van nce d able ad ow led d le IA0 6 n 10 ad va led ge ns IA1 ab sim io ud IA 1 ad va led ge able va kn ow nce ge kn ow ce kn ow led ge e sit IA1 8 T1 ge ne dge able tio ad va nce ge able en 1 AB 9 ex ow rt Po ad pe led 08 le r De TO, lop er ad va nce d able t pe na gin le ge AA 09 ab C ve lop b M ve De er lo ad va rt d able AA 10 ow ce d inte ad va nce d Jo CodeCity Experim ad van rt AB bi ad va ce as lo ve 11 De eve .S n e t ex van led d St ex va nce d Ph .D ulta tud den arAA 2 ad pe ce ge om 174 C ast er S er se AA1 So ead . St ude , Ph t r D AA her3 So ftw of ude nt .D. ad pe nce d n Ph ons er S tu c 1 t p e ent IA2 14 kn van nce d Ag ad va rt d tc 0 st ge ble kn ow ce d aly t t en Participant: 30 le ex ow led d ad per led gea ex va nce d ad van nce d De ftw are IT nt Compl. Time An ad va nce ab Re ad ad va nce 34 r de m ad pe nce d Co oje ent me ngin ee d T1 A.4 Task Solu r/ Co 42 Le eer er, M Pr ud lop e E ngin at A.4.1 T1: Azu ad va rt d ad pe nce d ge 174 tion Oracles IA 01 ex va ce ad 37 Total (excl. A4.2) tre t na ex va rt d reus, analyze IA IA 02 St ve ar E 21 H .D ed ec a va ce d va t e Per Task Co .D er S tu den chit m M d with CodeCi IA 03 ad van nce by nt 21 nc B2.1 Completion Tim ad va nce d kn va rt d 194 B1.2 7 A.5 48.5Data M ast er S ava , Sy er A 04 5 ad pe nce ty 29 CodeCity Experiment A1 ad va nce d B1.1 ge ble A.4 Task Solution Ph ast er S tu Ar ste M ast r J ant nag AB B01 d 26 A4.2 52.9 6 re ex va nce d Oracles le ex ow led d ad per led gea IA 02 19 A4.1 1.02 kn ow nce M nio ult Ma 26 A.4.1 T1: Azureus ab A3 1.90 34.67 ex per nce d te T1 AA 05 9.85 a A2.2 35 44.67 So ns . St tu den t at pe t d ns ct us , analyzed with Code 4.42 AA 01 A2.1 ad Either 27 A1 7.28 2.33 M ftw ulta ude den t Participant: Cod cl 02 0 2.92 37.97 D 174 eCity A.4 Task Solution Oracles 10.0 25 M ast are nt nt t 5.38 5.25 ad va nce d 10.00 47.33 ad van t 5 IA 06 , va ce A. 32 A1 4.32 on Introduction There are no 1.83 ex va nce d 5.33 rt 8.80 IA 07 2.95 45.50 Se Pr .D . St ud den t ct 28 unit tests in IA01 8.58 ad pe nce d 2.55 i Ph .D . St tu den ite 1.75 IA 08 55.50 at 39 the system [1pt 6.08 9.37 ex va rt d or Ph .D er S tu rch 1.08 IA A 09 5.62 m 2.42 38 ], 9.43 1.33 32.83 CodeCity Experiment IA B03 ct Either IA02 10.00 S A or 34 ad 2.42 So oje . St ude ent t A.4.1 T1: Azureus, analyzed with CodeCity 42.83 ite 10.00 va t d T1 6.92 IA 10 174 Centralized A.4 Task Solution Oracles ad per nce 22 nf 7.92 2.58 ch There are no unit in a single package 6.25 2.25 43.25 IA 11 6.08 22 The aim of this Ar li IA03 6.92 as er ed 4.00 10.00 53.25 D ftw ct M ude nt A 12 22 [1pt]. Sinctests in the system hierarchy whose na 6.00 er, nc experim 1.67 AA B04 6.25 So eve ar an nt 9.08 M ns ar er ch er e there is only ct 29 27.08 ag 2.58 Ph t ent A1 or [1pt], root is in org. 4.75 rso is to compare IA04 A 03 Co ftw lop e Ar ag ite practit 3.92 9.42 Participant: ioners analyzing me one test class an 37.08 32 answer, the answ gudy.azureus2.u 4.83 10.00 A B05 tool efficiency (i.e., TestUser 2.17 Ph ofe . St ud ent t ct M 8.08 31 Pr .D . St ud den oje r pe A.4.1 T1: Azureus, analyzed with CodeCityely i.console.multiu 1.33 Ph .D . St tu , Pr ee 1.58 36.00 IA A04 Introdu Centralized in a er is complet Manager), if they AB0 1 4.25 3.75 23 ction dium to large-s in supporting Ph .D er S ant ngin single package hiera ser 1.67 10.00 46.00 ts’ wrong. 13 Either don’t give the 5.58 23 cale software software rchy whose root is 10.0 0 3.50 1.33 1.42 45.92 A 14 Ph ast ult e E full correct AB02 ec 4.75 30 [1pt]. Since there 5.17 systems. A B06 in org.gudy.azure 1.33 10.00 55.67 IA is only one [1pt], 26 bj You will use Co A1 There are no unit tests in the system test class (i.e., us2.ui.console.mult 3.67 1.33 AA B07 A2.1 6.33 3.17 30 3.25 44.67 su answer, the answ TestUserManager), IA05 8.75 A 05 deCity to analyze iuser M .D sso ude ent 2.08 8.33 40 The aim of this or er is completely wron if they don’t give 9.75 54.67 A A06 4.42 nt e 1.50 39 experiment is to Azureus, a BitTorre g. the full correct 9.67 5.00 Th 10.00 1.33 IA A07 Either AA01 36.25 30 7.17 Correctness Per Task 7.33 Correctness Participant:You are analyzin compare tool effic nt client wri Dispersed [0pt 2.50 10.00 46.25 pe d t IA 15 Centralized in a single package hierarchy whose root is in org.gudy.azureus2 lo Stu en practitioners 27 t 1. s othe Code 4.17 Total 2.33 .S r r en tten in A2.1 in the system [1pt], rwise] 8.33 IA 16 iency in supporting Java. .ui.console.multiuser 10.00 A4.2 ve er tud AA02 A2.1 1.42 24.67 39 There are no unit tests in the follo A. g med given maximum ium to large-sc A1 A2.2 A34.42 A4.1 B1.1 0 10.0 B1.2 2.42 B2.1 (excl. A4.2) 01 34.67 21 Introduction software [1pt]. Since there is only one test (max. wing class (i.e., TestUserManager), if they don’t give 5.92 10.00 1.17 e IA 17 ale 4.58 23 100 minutes for software systems 5) packages the full correct 1.00 4.00 39.50 bl or IA06 1.00 6.33 0.00 IA A 19 10.0 1.00 3.33 24 De ast solving 10 tas . answer, the answer is completely wrong. [0.2pts for each IA01 0.00 1.00 5.67 1.00 6.25 1.00 0.00 0 6.00 1.42 6.00 47.33 Ta A B08 23 You will use CodeCit Centralized in aDispersed [0pts otherwise]zure root is in org.gudy.azureus2.ui.console.multiuser]: 7.25 3.75 A B09 ks (10 minute • com.aelitis.a single package hierarchy whose 4.55 0.00 2.50 37.58 52 You are asked: y IA07 0.80 2.95 1.00 1.00 3.67 A A10 s per task). us.core IA02 1.00 0.50 2.25 1.00 0.00 7.83 1.00 6.30 1.25 6.308 46.0 28 The aim of this experiment to anacompare tool a BitTorre in supporting in the following A A11 is to lyze Azureus, efficiencynt clien 2.67 5.25 24 [1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correct (max. 5) packages IA08 0.00 3.00 1.00 0 0.00 10.0 1.50 37.83 AA A12 1.00 7.17 36 • not to software • com.aelitis.a IA03 0.00 1.00 3.67 0.00 0.00 8.50 0.00 3.00 8.08 3.00 8 45.5 A 13 t written in Java. A2.1 [0.2pts answer, the answer•is completely wrong. zureus.core.cont for each] : 10.00 5.00 practitioners giveconsult any other particip 1.75 35.32 IA A14 IA09 1.00 5.00 Introduction• areanalyzing medium to large-scale software systems. You to com.aelitis.azureus. ent IA04 0.00 6.33 0.00 2.00 1.00 9.33 1.00 0.00 3.83 0.00 7.75 1.00 3.08 1.00 5.00 1.75 5.00 3 42.2 20 n maximum perform the tas 100 minutesant during the exp Dispersed [0pts otherwise] • com.aelitis.a core AB0 3 6.83 2.83 30.33 for ks in the specifie solving 10 task eriment; zureus.core. AB01 0.00 1.00 3.25 1.00 3.67 0.807.33 1.00 0.00 4.00 0.00 6.92 1.00 7.42 1.00 5.80 1.25 5.80 3 40.3 • to write dow d order; s (10 A2.1 in the following (max. 5) packages [0.2pts fordownload • com.aelitis.azu 7.17 2.00 24.75 You willthis experimentnis to current timtool efficiency client written minutes per task). IA10 You are asked: the compare use CodeCity to analyze Azureus, a BitTorrent in supportingin Java. reus.core.content each]: • com.aelitis.a AB02 0.00 1.00 3.83 1.002.40 0.705.921.00 0.00 5.17 10.00 0.00 0.00 5.75 1.00 0.50 4.70 3.83 4.70 0 34.5 The aim of after com e each time bef software zureus.core.imp IA11 4.171.00 5.58 2.00 36.83 30.50 pleting • • to ann medium all the ore sta • com.aelitis.azu [0pts otherwise] Dispersed• com.aelitis.azureus.core l IA05 0.00 1.003.75 1.00 2.67 0.20 0.00 5.58 0.00 9.75 1.00 1.00 1.25 5.20 2.58 5.20 practitionersnot to consult any othe tasks; software systems. rting to read a task and analyzing oun to large-scale reus.core.down in the following (max. 5) • com.aelitis.azureus.c load IA12 3.920.40 2.58 3.67 0.00 6.33 4.08 4.20 1.67 4.20 0 49.0 39.00 You areto perform ce the exprerimenter nt during the exp (10 minutes participa • given maximum 100 minutes for that you tasks eriment; once packages [0.2pts for each]: ore.lws AA01 0.00 4 0.802.671.00 1.0010.00.00 3.17 0 1.00 0.00 1.83 35.83 to reset your tasks in the spe solving 10 mo the per task). • com.aelitis.azu • com.aelitis.azureus.core.content AB0 3.75 0.00 5.83 10.00 9.58 6.00 2.83 6.00 7 2 reus.core.impl 0.00 3 1.00 2.501.003.170.00 43.6 10-minute cified order; are ving on Java. • com AA02 AA0 1.00 5.58 1.00 1.00 1.00 1.25 6.00 26.58 19 You will use to not e down the current s-per-task allocatclient written into another task, in • CodeCity to analyze Azureus, a BitTorrent • writ to retu • com.aelitis.azureus.core .aelitis.azureus.c • com.aelitis.azu ore.peermanager IA06 0.00 51.00 5.50 0.004.67 0.30 4.33 1.00 6.33 0.00 3.50 0.00 7.83 1.00 1.00 1.67 4.30 2.08 4.30 36.58 time eac You are asked: plet rn to earlier tasks bech time before star ed timer; order • com.aelitis.azureus.core.download reus. .peerdb AB0 5.17 4.83 43.92 after com ing • for each task,all ause it affects ting to read a task • com.aelitis.acore.lws • com.aelitis.azureus.core.content zureus.c IA07 1.00 41.00 7.08 1.00 3.670.17 1.00 4.75 0.00 3.25 0.00 10.00 1.00 1.00 1.83 6.17 4.17 6.17 49.75 • not to maximum otherthein for solving the experiment; tasks; AA0 5.33 46.27 You are givenconsult any100experimthe required 10 tasks (10the timing; task). and once 2.33 • tocheck ce the to fill • com.aelitis.azu ore.stats 56.27 announ participant duringinfo minutes minutes per • com.aelitis.azureus.core.impl reus. • com.aelitis.acore.peerm IA08 1.00 IA13 0.50 3.00 1.00 4.67 0.83 1.00 10.000.00 4.50 1.00 5.83 1.00 10.001.00 7.33 2.00 7.33 3.85 55.92 the rmation. In the • to perform tthe tasks st appropenteranswe are moving on case of mu • com.aelitis.azureus.core.download us.c anager.peerdb to req your mo in the specifiedthat you 2.42 10.00 65.92 rese riat 10-minutes-p e order; and ltip to another task le choice • com.aelitis.azu zure ore.torrent IA09 1.00 IA141.00 6.67 1.00 9.00 1.00 2.05 1.00 6.95 10.00 0.00 1.00 1.00 10.001.00 8.00 2.67 8.00 3.67 50.33 ues r • com.aelitis.azureus.core.lws • to • not downted. current time er-task allocatedprovide addread a informa , in order s 10.00 6.67 8.00 5 reus.core.stats 5.75 54.7 write to retuthe You are asked: rn to earlier taskeach time before timer; to itional task and once starting • com.aeli • com.aelitis.azureus.core.impltis.azureus.p AB03 1.00AB06 1.00 5.67 1.00 1.00 4.67 10.0 1.00 0.00 0 6.83 1.00 1.00 10.001.00 8.00 3.17 30.83 tion, if • com.aelitis.azu lugins.net.b 9.33 4.42 1.58 6.17 837.5 • completing all participant s because experiment; • com.aelitis.azureus.core.peermanager.peerdb uddy 1.00 8.75 5.25 • not after for each erim , to filltasks; during the it affects the timing; 22.83 IA10 1.00AB07 1.00 0.17 3.75 1.00 8.42 0.00 0.00 1.00 1.00 1.58 6.17 to consultexp task the in the The any other • com.aeli reus.core.torrent • com.aelitis.azureus.core.lws tis.azureus.p 6.08 3.25 6.75 5.33 1.835.00 28.00 ent is con required • to announce the in the specified order; information. In the another task, in order chethe the most app cluded with a sho experimenter that you are moving on to case • com.aelitis.azu lugins.net.budd IA11 1.00 AA01.00 7.00 5 1.00 0.00 3.67 1.00 6.58 0.16 0.00 0.00 1.00 2.75 3.33 5.16 52.67 • to perform ck tasks ropriate answer and rt debriefing que of multiple choices • com.aelitis.azureus.core.stats reus. • com.aelitis.a plugins.net.buddy y.swt IA12 1.00 AA06 1.00 6.83 1.00 2.00 1.00 2.50 1.00 2.17 0.00 2.92 5.17 1.00 10.00 1.00 1.00 1.75 7.83 62.6 7 50.25 to reset este 10-minutes-per-task allocated starting add 5.08 8.00 8.0056.75 • to write requyour d.current time each time before timer;ide to read astionnaire.once prov • com.aelitis.azureus.core.peermanager.peerdbzureus.p 3.67 down the itionaltask rmation info and , if • com.aelitis.azureus.core.torrent lugins.net.buddy.tracker • com.aelitis.azu AB04 1.00 AA0 7 1.00 3.67 1.00 0.67 5.08 1.00 10.00 0.00 4.33 0.00 1.00 2.00 4.50 39.02 reus. • com.aelitis.a plugins.net.buddy.swt 6.50 1.00 8.50 6.67 6.6743.32 after not to return tothe tasks; • completing all earlier tasks because it affects the timing; • com.aelitis.azureus.core.stats zureus.plugi AA03 1.00 IA15 1.00 9.75 8.83 0.80 7.42 0.33 4.33 1.00 9.50 0.16 4.00 0.00 1.00 2.10 2.93 46.55 • com.aelitis.azureus.plugins.net.buddy ns.removerules • com.aelitis.azu 4.30 1.00 9.92 6.29 6.13 50.53 • to • for each task, entfill in the required information. In the another multiple order The experim to is 0 4.03 3.12 announce the experimenter that you are moving on to case of task, in choices Thank you for concluded reus. • com.aelitis.a plugins.net.buddy.tracker AB05 1.00 IA16 1.00 10.0 1.00 3.90 1.00 1.00 4.38 9.20 0.00 0.00 1.00 3.98 1.00 9.77 7.00 4.13 7.00 38.17 33.08 participatin with a short deb check the most appropriate answer andtimer; riefing que information, • com.aelitis.azureus.core.torrentzureus.plugi 2.55 4.72 5.58 44.42 to reset your 10-minutes-per-task g in this experimentadditionalstionnaire. if allocated provide • com.aelitis.azu reus. ns.s • com.aelitis.azureus.plugins.net.buddy.swtharing.hoster AA04 1.00 IA01 1.00 1.00 5.13 0.50 1.00 4.58 9.82 0.00 0.00 1.00 5.08 1.00 6.58 6.50 1.92 6.50 50.33 ! • com.aelitis.a plugins.remove 5.28 3.83 1.92 • not requested. earlier tasks because it affects the timing; • com.aelitis.azureus.plugins.net.buddy lugi rules IA13 1.00 IA181.00 1.00 3.83 0.83 1.00 3.50 1.00 0.00 0.00 9.42 6.83 3.58 33.83 to return to ! ! • com.aelitis.azu zureus.p ns.starts 3.33 4.50 3.50 5.92 1.00 5.83 39.83 1.50 ! ! Richard Wettel • com.aelitis.azureus.plugins.net.buddy.tracker toprules.default reus. 0.00 IA19 0.30 10.00 8.83 0.00 2.00 5.10 1.92 38.00 • for each task, to fill in the required information., In the case of multiple choices • com.aelitis.a plugins.sharing.hoster plugin IA14 0.80 0.33 6.08 1.08 1.00 0.67 3.42 1.00 6.00 1.00 4.43 41.33 2.33 The thenk you appropriate answer and provideMichele Lanza, Romain if • com.aelitis.azureus.plugins.net.buddy.swtzureus.plugi 8 10.00 1.17 31.92 experiment is concluded with a short debriefing questionnaire. Ro Tha most for part • com.aelitis.azureus.plugins.removeruns.tracker.dht • com.aelitis.azu AB06 1.00 AB0 1.00 1.00 0.00 4.83 4.33 0.00 0.00 0.00 1.00 3.33 0.00 4.83 4.00 4.00 37.08 check icipating in this exp additional information, bbes reus. • com.aelitis.a plugins.startsto les AB09 5.83 7.33 8.00 3.67 2.27 1.67 2.92 34.25 eriment! • com.aelitis.azureus.plugins.net.buddy.tracker prules.defaultplugin zureus.plugi AB07 0.00 0.60 0.67 3.17 0.00 6.08 5.08 0.00 0.00 0.00 3.50 0.00 5.17 1.00 2.27 3.00 37.58 requested. • com.aelitis.azureus.plugins.sharing.hns.tracker.local • com.aelitis.azu AA10 6.08 1.00 3.83 4.42 7.00 1.42 53.08 ! reus. • com.aelitis.a plugins.tracker.dht oster AA05 1.00 1.00 1.00 6.17 0.00 4.08 1.00 0.00 5.25 1.00 3.33 1.00 6.00 58.75 5.42 ! ! ! • com.aelitis.azureus.plugins.removeruleszureus.plugins.tr AA11 3.92 1.00 4.75 6.33 1.42 3.00 43.33 40.08 Richard Wettel, Mic • com.aelitis.azu acker.peerauth • com.aelitis.azureus.plugins.startstoprules.defaultplugin AA06 0.00 1.00 AA12 0.00 6.75 0.00 4.75 1.00 0 0.00 5.92 0.00 5.67 1.00 4.002.42 4.50 32.67 hele Lanza, Rom The experiment is concluded with a short debriefing questionnaire. ain Rob reus. • com.aelitis.a plugins.tracker. 10.0 7.00 1.00 0 1.00 0 0.00 10.0 3.25 0.009.17 5.00 4.00 36.08 bes • com.aelitis.azureus.plugins.sharing.hoster local zureus.ui.swt.co AA07 1.00 1.00 0.00 7.00 0.00 2.83 1.00 1.42 1.33 Thank you for participating in this experiment! • com.aelitis.azu • com.aelitis.azureus.plugins.tracker.dht ntent.columns AA13 1.50 1.00 3.33 10.0 3.42 0.002.424.80 reus.plugins.tracker. IA15 1.00 0.80 0.00 6.33 0.00 1.008.00 0.00 2.33 1.00 3.80 • com.aelitis.a • com.aelitis.azureus.plugins.startstoprules.defaultplugin zureus.ui.swt.she peerauth AA14 4.75 • com.aelitis.azu • com.aelitis.azureus.plugins.tracker.local lls.main IA16 0.00 0.00 0.67 6.83 0.00 5.58 1.00 0.67 0.00 1.00 1.00 4.34 e, in3.67 utes min IA20 tim task 1.00 tion reus. ! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes • com.aelitis.a ui.swt.content.columns IA01 1.00 1.00 1.00 0.50 1.00 1.00 subjects’ 1.00 comple 7.50 • com.aelitis.azureus.plugins.tracker.dht • com.aelitis.azu zureus.util • com.aelitis.azureus.plugins.tracker.peerauth The 0.00 Table A.4. 1.00 0.00 1.00 1.00 7.50 6.50 Thank you for participating in this experiment! reus.ui.swt.shells.m IA18 1.00 1.00 1.00 0.50 1.00 6.50 • com.aelitis.azureus.plugins.tracker.local ain • com.aelitis.azu • com.aelitis.azureus.ui.swt.content.columns IA19 0.00 1.00 0.60 0.00 1.00 1.00 0.00 1.00 0.00 4.60 3.60 reus.util • com.aelitis.azureus.plugins.tracker.peerauth AB08 0.00 1.00 0.40 0.38 1.00 1.00 1.00 1.00 1.00 6.78 5.78 ! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes • com.aelitis.azureus.ui.swt.shells.main AB09 1.00 1.00 0.60 0.50 1.00 1.00 0.00 1.00 1.00 7.10 6.10 • com.aelitis.azureus.ui.swt.content.columns AA10 1.00 1.00 0.80 0.38 1.00 1.00 0.00 1.00 1.00 7.18 6.18 • com.aelitis.azureus.util • com.aelitis.azureus.ui.swt.shells.main AA11 1.00 1.00 0.80 0.00 1.00 1.00 0.00 1.00 1.00 6.80 5.80 AA12 0.00 0.50 0.00 0.00 1.00 1.00 0.00 1.00 1.00 4.50 3.50 • com.aelitis.azureus.util AA13 0.00 1.00 0.00 0.25 1.00 1.00 0.00 1.00 1.00 5.25 4.25 AA14 1.00 1.00 0.60 0.00 1.00 1.00 0.90 1.00 1.00 7.50 6.50 IA20 0.00 1.00 1.00 0.38 1.00 1.00 0.00 1.00 1.00 6.38 5.38 Table A.3. The correctness of the subjects’ solutions to the tasks Pre-experiment Handouts Oracles Data questionnaire technical report 2010
Conclusions
Summary
Summary list of design desiderata
Summary list of design desiderata 6 months 4 locations 41 subjects
Summary list of design +24.26% correctness desiderata -12.01% completion 6 months time 4 locations 41 subjects
Take-away message Visualizing software systems as cities is a viable alternative to the state of the practice approaches for software exploration.
We thank Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and
We thank Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and None of the participants has been harmed during the experiment.
Richard Wettel http://www.inf.usi.ch/phd/wettel http://creativecommons.org/licenses/by/3.0/