{"id":118479,"date":"2017-05-30T18:56:01","date_gmt":"2017-05-30T18:56:01","guid":{"rendered":"https:\/\/livablesoftware.com\/?p=118479"},"modified":"2017-11-04T18:59:00","modified_gmt":"2017-11-04T18:59:00","slug":"learned-software-development-mining-github-plus-concerns","status":"publish","type":"post","link":"https:\/\/livablesoftware.com\/learned-software-development-mining-github-plus-concerns\/","title":{"rendered":"All we have learned about software development by mining GitHub (plus some concerns)"},"content":{"rendered":"<p>We (<a href=\"http:\/\/som-research.uoc.edu\/valerio-cosentino\/\" target=\"_blank\" rel=\"noopener\">Valerio<\/a>,\u00a0<a href=\"https:\/\/jordicabot.com\/\" target=\"_blank\" rel=\"noopener\">Jordi<\/a>\u00a0and <a href=\"http:\/\/jlcanovas.es\" target=\"_blank\" rel=\"noopener\">myself<\/a>)\u00a0have continued our meta-analysis of research papers on how software development practices are influenced by the use of a distributed social coding platform like <a href=\"https:\/\/github.com\/\" target=\"_blank\" rel=\"noopener\">GitHub<\/a> (see <a href=\"https:\/\/modeling-languages.com\/believe-research-github-mining\/\" target=\"_blank\" rel=\"noopener\">our initial work in this area<\/a>).<\/p>\n<p>Our objective is to identify the\u00a0quantity, topic and empirical methods\u00a0used in these works to see what we have learned as a software community thanks to the large number of papers mining GitHub with the hope of extracting some useful information that can help improve software development.<\/p>\n<p>To do this, <strong>we performed a systematic mapping study<\/strong>, which was conducted with four research questions and <strong>assessed 80 publications from 2009 to 2016<\/strong>.<\/p>\n<p>Our study attested the high activity of research work around the field of Open Source collaboration, especially in the software domain, revealed a set of shortcomings and proposed some actions to mitigate them. We hope this work can also create the basis for additional studies on other collaborative activities (like book writing for instance) that are also moving to GitHub.<\/p>\n<p>We provide a summary of our findings in this post. Full results have been published in the IEEE Access journal. Read them <a href=\"http:\/\/ieeexplore.ieee.org\/document\/7887704\/\" target=\"_blank\" rel=\"noopener\">here<\/a> (free for all, it&#8217;s an open access journal).<\/p>\n<h2>What topics\/areas have been studied? What have we\u00a0learned thanks to GitHub data?<\/h2>\n<p>The following table lists the main topics and areas that papers mining GitHub data focused on. We list a sample of the main findings reported by those papers. As you&#8217;ll see some of the findings are expected but others are more surprising. Let me emphasize again that these are NOT our findings but the findings of a large community of researchers working on this topic. Refer to the <a href=\"http:\/\/ieeexplore.ieee.org\/document\/7887704\/\" target=\"_blank\" rel=\"noopener\">paper<\/a> to go to the sources of each claim and get all the data you need to understand the context of the claim.<\/p>\n<p><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-areasTopics.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-5770 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-areasTopics.png\" alt=\"\" width=\"573\" height=\"303\" \/><\/a><\/p>\n<ul>\n<li>On Development\n<ul>\n<li>Code contributions\n<ul>\n<li>Contributions are unevenly distributed across the project<\/li>\n<li>Few developers are responsible for a large set of code contributions<\/li>\n<li>Most contributions come in the form of direct code modifications<\/li>\n<li>Most pull requests are small (less than 20 lines), processed in less than 1 day and the discussion spans on average to 3 comments<\/li>\n<li>Importance of casual contributions:\u00a0their contributions are becoming in fact rather common<\/li>\n<li>Technical (e.g., documentation or code length) and social (e.g., friendship or external contributions) factors are crucial to accept\/reject pull requests<\/li>\n<li>Geographical factors also affects the acceptance of pull requests<\/li>\n<li>Pull request evaluation time depends on their complexity, author of the request or timing (e.g., sent out of business hours)<\/li>\n<\/ul>\n<\/li>\n<li>Issues\n<ul>\n<li>Issues tracker are scarcely used<\/li>\n<li>Few projects concentrate most of the issues<\/li>\n<li>Distribution of issues in a project: numerous when the project is created, stable when the project matures and pending issues always growing<\/li>\n<li>A small number of labels are used to tag issues<\/li>\n<li>Issue lifetime is generally stable<\/li>\n<li>The use of labels and @-mentions is effective in solving issues<\/li>\n<\/ul>\n<\/li>\n<li>Forking\n<ul>\n<li>Forking is unevenly distributed<\/li>\n<li>Forking is a good indicator of project liveliness<\/li>\n<li>Forks are mostly used to fix bugs<\/li>\n<li>Forking is beneficial for the project<\/li>\n<li>Forkability depends on several factors such as activity, publicity or community<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Projects\n<ul>\n<li>Characterization\n<ul>\n<li>Most projects are personal<\/li>\n<li>Most projects choose not to benefit from GitHub features (beyond pure project hosting)<\/li>\n<li>Commercial projects also rely on GitHub<\/li>\n<li>Project domains mainly includes application and system software, documentation and libraries<\/li>\n<li>The top\u00a05 project languages are JavaScript, Ruby, Python, Objective-C and Java<\/li>\n<li>User base and documentation keep the project alive<\/li>\n<li>Attracting and retaining contributors is key for project survival<\/li>\n<\/ul>\n<\/li>\n<li>Popularity\n<ul>\n<li>Few projects are popular<\/li>\n<li>The main benefit of project popularity is the attraction of new developers<\/li>\n<li>Project activity and project popularity is usually measured by the number of forks, start, followers and\/or watchers<\/li>\n<li>Documentation as a factor for project popularity<\/li>\n<li>Fame of projects driven by organizations, which tend to be more popular than the ones owned by individuals<\/li>\n<li>Trends in programming languages and domain affect project popularity<\/li>\n<\/ul>\n<\/li>\n<li>Communities and teams\n<ul>\n<li>Small development teams are the rule<\/li>\n<li>Types of teams can be fluid, commercial, academic or stable<\/li>\n<li>Team diversity has a positive impact on the collaboration experience and productivity<\/li>\n<li>Community composition is usually difficult to measure<\/li>\n<li>Stability of core developers<\/li>\n<li>Few users become core developers<\/li>\n<\/ul>\n<\/li>\n<li>Global discussions\n<ul>\n<li>Most discussions revolve around code-center issues<\/li>\n<li>Long discussions have a negative effect<\/li>\n<li>Mood in discussions is generally neutral<\/li>\n<li>Barriers and enablers in discussions include having paid developers (barrier) and using label\/mention mechanisms (enabler)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Users\n<ul>\n<li>Characterization\n<ul>\n<li>A majority of users come from the United States<\/li>\n<li>GitHub is mainly driven by males and young developers<\/li>\n<li>Hobbyists as main citizens<\/li>\n<li>Approximately half of GitHub\u2019s registered users work in private projects<\/li>\n<li>Indicators to assess the expertise, interest, and importance of users rely on main GitHub&#8217;s features (e.g., forking, project popularity, etc.)<\/li>\n<li>Sustained contributions and volume of activity of a user are considered as a signal of commitment to the project<\/li>\n<li>User productivity depends on factors such as the number of projects the user is involved and her commitment to the project<\/li>\n<\/ul>\n<\/li>\n<li>Rockstars\n<ul>\n<li>A rockstar, or popular user, is characterized by having a large number of followers, which are interested in how she codes, what projects she is following or working on<\/li>\n<li>Users become popular as they write more code and monitor more projects<\/li>\n<li>Rockstars play an important role in the project dissemination and attractiveness<\/li>\n<\/ul>\n<\/li>\n<li>Issue reporters and assignees\n<ul>\n<li>A vast majority of issue reporters are not active on code contributions, do not participate in any project and do not have followers, in addition, they often have an empty profile<\/li>\n<\/ul>\n<\/li>\n<li>Followers\n<ul>\n<li>Following is not mutual<\/li>\n<li>Reasons to follow include: to discover new projects and trends, for\u00a0learning, socializing and collaborating as well as for implicit coordination and when looking for a job<\/li>\n<li>Following networks in GitHub fit the theory of the \u201csix degrees of separation\u201d in the \u201csmall world\u201d phenomenon, meaning that any two people are on average separated by six intermediate connections<\/li>\n<\/ul>\n<\/li>\n<li>Watchers\n<ul>\n<li>Watching is generally used to assess the quality, popularity and worthwhile of a project<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>Ecosystem\n<ul>\n<li>Characterization\n<ul>\n<li>Most ecosystems in GitHub revolve around one\u00a0central project, whose purpose is to support software development, such as frameworks and libraries\u00a0on which the other projects in the ecosystem rely on<\/li>\n<li>Popular ecosystems are usually interconnected<\/li>\n<li>Mutual evolution in linked ecosystems<\/li>\n<li>JavaScript, Java and Python dominate the ecosystems<\/li>\n<\/ul>\n<\/li>\n<li>Transparency\n<ul>\n<li>GitHub\u2019s transparency helps in identifying user skills as well as enabling learning from the actions of others, facilitating the coordination between users in a given ecosystem and lowering the barriers for joining specific projects<\/li>\n<li>The transparency provided by GitHub can help in making sense of the project (or ecosystem) evolution over time<\/li>\n<\/ul>\n<\/li>\n<li>Relationships with other platforms\n<ul>\n<li>Existence of an ecosystem on the Internet for software developers which includes many platforms, such as GitHub, Twitter and StackOverflow, among others<\/li>\n<li>Twitter is used by developers to stay aware of industry changes, for learning, and for building relationships, thus it helps developers to keep up with the fast-paced development landscape<\/li>\n<li>StackOverflow is often used as a communication channel for project dissemination<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>How were these GitHub mining studies conducted?<\/h2>\n<p>We characterized the methodological aspects of the empirical process followed by the selected papers when inferring their results and take a critical look at them to evaluate how confident we can be about those results and how generalizable they might be.<\/p>\n<h3>What empirical methods have been used to reach the previous statements?<\/h3>\n<p>Regarding the details of the empirical methods used in the papers, we classified the selected papers according to:<\/p>\n<ul>\n<li>Type of methods employed for the analysis. This dimension can take four values, namely: (1) Metadata observation, when the work relies on the study of Github metadata; (2) surveys; (3) interviews; or (4) a mixture of methods. Other kinds of empirical methods are not added as dimensions since no instance of them were found.<\/li>\n<li>Sampling techniques used, which classifies the sampling techniques used to build the datasets out of GitHub (i.e., subsets of projects and users input of the analysis phase for the papers). This dimension can take three values, namely: (1) non-probability sampling, (2) probability sampling; or (3) no sampling.<\/li>\n<li>Self-awareness, which analyzes whether a paper reports on its own limitations (i.e., threats to validity).<\/li>\n<\/ul>\n<p>The following figures summarize the results.<br \/>\n<a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/rq2Color.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-5781 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/rq2Color.png\" alt=\"\" width=\"686\" height=\"380\" \/><\/a><\/p>\n<h3>Which technologies have been used to extract and build datasets from GitHub?<\/h3>\n<p>Papers studying GitHub must first collect the data they need to analyze, which can either be created from scratch or by reusing existing datasets. We classified the selected papers according to:<\/p>\n<ul>\n<li>Data collection process, which reports on the tool\/s used to retrieve the data. Currently there are six possible values for this dimension according to the tool used: (1) GHTorrent [80], (2) GitHub Archive, (3) GitHub API, (4) Others (e.g., BOA [101]), (5) manual approach and (6) a mixture of them.<\/li>\n<li>Dataset size and availability, which reports on the number of users and\/or projects of the dataset used in the study, and indicates whether the dataset is provided together with the publication. This may help to evaluate the replicability of the work.<\/li>\n<\/ul>\n<p>The following figures summarize the results.<\/p>\n<p><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/rq3Color.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-5782 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/rq3Color.png\" alt=\"\" width=\"581\" height=\"656\" \/><\/a><\/p>\n<h3>What are the research communities and the publication fora used?<\/h3>\n<p>We believe it is also interesting as part of a systematic mapping study to characterize the research community behind the field, both in terms of the people and the venues where the works are being published.\u00a0According to this, we analyzed the two following dimensions:<\/p>\n<ul>\n<li>Researchers, which we propose to analyze by building the co-authorship graph of the selected papers. In this kind of graphs, authors are represented as nodes while co-authorship is represented as an edge between the involved author nodes. Furthermore, the weight of a node represents the number of papers included in the set of selected papers for the corresponding author while the weight of an edge indicates the number of times the involved author nodes have coauthored a paper. By using co-authorship graphs, we can apply well-known graph metrics to analyze the set of selected papers from a community dimension perspective.<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-5772 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-rq4a.png\" alt=\"\" width=\"596\" height=\"400\" \/><\/p>\n<ul>\n<li>Publication fora, which requires a straightforward analysis as we only have to use the venue where each selected paper was published.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-rq4c.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-5776 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-rq4c.png\" alt=\"\" width=\"626\" height=\"182\" \/><\/a><\/p>\n<h2><a href=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-rq4b.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-5771 aligncenter\" src=\"https:\/\/modeling-languages.com\/wp-content\/uploads\/2017\/05\/ieeeAccess-rq4b.png\" alt=\"\" width=\"931\" height=\"212\" \/><\/a><\/h2>\n<h2>Discussion<\/h2>\n<p>It&#8217;s undeniable the amazing opportunity that GitHub (or more specifically the millions of projects hosted there and all the available data that comes with them) offers to the software research community. Many useful findings have already been reported that I believe can make a significant impact on the way open source projects are managed (hopefully, <a href=\"https:\/\/modeling-languages.com\/very-large-community-software-development-erc\/\" target=\"_blank\" rel=\"noopener\">making them more community-aware<\/a> \ud83d\ude42 ).<\/p>\n<p>Still, we have also seen some issues regarding how some of these previous studies have done that could hinder the generalization of the results. I think it&#8217;s only fair to finish our metastudy with a list of concerns that we would like to see addressed in the future.<\/p>\n<p><strong>Lack of specific development areas and interdisciplinary studies<\/strong>. Our analysis of areas\/topics reveals a lack of works studying GitHub with a more interdisciplinary perspective, e.g., from a political point of view (e.g., study of governance models in OSS), complex systems (e.g., network analysis) or social sciences (e.g., quality of discussions in issue trackers). We also miss works targeting early phases of the development\u00a0cycle (e.g.s studying requirements and design aspects). Obviously, these kinds of studies are more challenging since there is typically fewer data available in GitHub or needs more processing.<\/p>\n<p><strong>Overuse of quantitative analysis<\/strong>. Right now, a vast majority of studies rely only on the analysis of GitHub metadata. We hope to see in the future a better combination of such studies (typically large regarding the spectrum of analyzed projects but shallow in the analysis of each individual project) with other studies targeting the same research question but performing a deeper analysis (e.g., including also interviews to understand the reasons behind those results) for a smaller subset of projects. We detected some evidences of this shift in the number of works applying longitudinal studies, where we found a positive trend, thus improving the situation detected in OSS by other works.<\/p>\n<p><strong>Poor sampling techniques<\/strong>. We believe that the GitHub research community could benefit from a set of benchmarks (with predefined sets of GitHub projects chosen and grouped according to different characteristics) and\/or from having trustworthy algorithms responsible for generating diverse and representative samples according to a provided set of criteria to study. These samples could, in turn, be stored and made available for further replicability studies.<\/p>\n<p><strong>Small datasets size<\/strong>. Most papers use datasets of small-medium size, which are an important threat to the validity of results when trying to generalize them. Also, more than half of the analyzed paper did not provide a link to the dataset they used nor an automatic process to regenerate it, which may also hamper the replicability of the studies. However, it is important to note that the number of works sharing their datasets is increasing over time.<\/p>\n<p><strong>Low level of self-awareness<\/strong>. We believe the limited acknowledgment of their threats to validity (around 34% do not report any) puts the reader in the very difficult situation of having to decide by herself the confidence in the reported results the work deserves. Works reporting partial results or findings on very concrete datasets are perfectly fine but only as long as this is clearly stated. However, we detected an upward trend in reporting threats to validity in recent years.<\/p>\n<p><strong>Need of replication and comparative studies<\/strong>. Clearly urgent since almost none exist at the moment. replicability is hampered by some of issues above and the difficulties of performing (and publishing) replicability studies in software engineering. Comparative studies need to first set on a fixed terminology. Some papers may seem inconsistent when reporting on a given metric but a closer look may reveal that the discrepancy is due to their different interpretation. For instance, this is common when talking about success, popularity or activity in a project (e.g., one may consider a project successful as equivalent to popular, and by popular mean to be starred a lot, while the other may interpret successful as a project with a high commit frequency). Replicability<br \/>\nis also threaten in studies involving user interviews, as the material produced as a result of such interviews is normally not provided.<\/p>\n<p><strong>API restrictions limits the data collection process<\/strong>. GitHub self-imposed limitations on the use of its API hinders the data collection process required to perform wide studies. A workaround can be the use of OAuth access tokens from different users to collect the information needed. An OAuth access token is a form to interact with the API via automated scripts. It can be generated by a user to allow someone else to make requests on his behalf without revealing his password. Still, researchers may need a large number of tokens for some studies. To deal with this issue, we propose that GitHub either lifts the API request limit for research projects (pre-approving them first if needed) or offers an easy way for individual users to donate their tokens to research works they want to support.<\/p>\n<p><strong>Privacy concerns<\/strong>. Data collection can also bring up privacy issues. Through our study, we discovered that GitHub and third-party services, built on top of it, are sometimes used to contact users (using the email they provide in their GitHub profile) to find potential participants for surveys and interviews. This potential misuse of user email addresses can raise discontent and complaints from the GitHub users. Even when it is not clear whether such complaints are reasonable, the controversy alone may hamper further research works and therefore must be dealt with care. As with the tokens before, we would encourage GitHub to also have an option in the user profile page to let users say whether they allow their public data to be mined as part of research works and under what conditions.<\/p>\n<span class=\"et_bloom_bottom_trigger\"><\/span>","protected":false},"excerpt":{"rendered":"<p>Mining information on software projects hosted on GitHub can reveal a lot of useful information on how software projects (and the community behind them) should be managed to optimize your chances of success<\/p>\n","protected":false},"author":3,"featured_media":118480,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[10,9],"tags":[12,25,33],"_links":{"self":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts\/118479"}],"collection":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/comments?post=118479"}],"version-history":[{"count":0,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/posts\/118479\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/media\/118480"}],"wp:attachment":[{"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/media?parent=118479"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/categories?post=118479"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/livablesoftware.com\/wp-json\/wp\/v2\/tags?post=118479"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}