AI Coding Agents on GitHub: Repository Popularity as a Hidden Confounder in the Acceptance of 2.4 Million Agent-Authored Pull Requests

Abstract

Autonomous coding agents now open pull requests on GitHub without a person writing the code, and the share of those pull requests that maintainers merge has become the usual shorthand for how well the agents work. This paper argues that the shorthand measures the projects as much as it measures the agents. Using the public AIDev dataset, a snapshot of 2,743,854 pull requests written by six agents across 326,798 repositories between December 2024 and October 2025, we find that the merge rate falls from 91.58 percent in repositories with no stars to 60.05 percent in repositories with a thousand or more. The decline appears inside every agent separately, and in the five agents with enough volume in the top band to measure it the fall is between 20.3 and 27.6 percentage points, so it cannot be an artefact of which agent is used where. The enriched portion of the dataset that previously published analyses rely on contains only repositories with at least 100 stars, which we verify directly; it holds 2.7 percent of the decided pull requests and merges them at 73.67 percent against 90.63 percent elsewhere. Holding the repository fixed with a Mantel-Haenszel estimator changes the picture between agents: of fifteen comparable pairs, two reverse direction and one loses more than half its effect. Time to a decision separates the same way, a median of 51 seconds in unstarred repositories against 5.4 hours in the most popular ones, with a medium effect size. Even within the enriched subset, 75.8 percent of agent pull requests carry no recorded human review. We conclude that published acceptance figures describe a narrow and unusually demanding slice of GitHub, and we recommend that studies of agent contributions report the popularity distribution of their repositories and estimate effects within repositories rather than across them.

Introduction

The paper investigates whether GitHub pull-request (PR) merge rates are a reliable measure of coding-agent performance. It argues that commonly reported acceptance rates may be misleading because the decision to merge a PR depends not only on the quality of the AI-generated code, but also on the characteristics and popularity of the repository receiving it.

Main idea

Coding agents such as OpenAI Codex, GitHub Copilot, Cursor, Devin, Claude Code, and Google Jules can now autonomously generate software changes and submit them as pull requests. Previous studies have often used the percentage of PRs that are merged as a measure of agent success.

The paper challenges this approach because merging is a repository-level decision. Popular and highly visible repositories may have stricter review standards, more maintainers, and greater risk sensitivity than small repositories. Therefore, an overall merge rate can partly measure the characteristics of the repositories where an agent is used rather than the agent's actual capability.

Data and methodology

The study analyzes the AIDev dataset, using a September 2026 snapshot containing:

• 2,743,854 agent-authored pull requests.
• 2,451,452 decided PRs after excluding still-open PRs.
• 326,798 repositories.
• 6 coding agents.
• Repository popularity measured through GitHub star counts.

The researchers divide repositories into five popularity groups: 0 stars, 1–9, 10–99, 100–999, and 1,000+ stars.

They use statistical tests including chi-square tests, effect sizes, odds ratios, and Mantel-Haenszel analysis. The Mantel-Haenszel method is particularly important because it compares agents within the same repositories, reducing the influence of repository-level differences.

Key findings

1. Agent merge rates differ significantly.
   Across 2.45 million decided PRs, the overall merge rate was 90.15%, but individual agents varied considerably:

   • OpenAI Codex: 93.20%
   • Google Jules: 88.74%
   • Claude Code: 87.84%
   • Cursor: 80.39%
   • GitHub Copilot: 77.63%
   • Devin: 69.74%

2. Repository popularity has a strong relationship with acceptance.
   Merge rates decrease as repository popularity increases:

   • 0 stars: 91.58%
   • 1–9 stars: 88.35%
   • 10–99 stars: 81.00%
   • 100–999 stars: 80.52%
   • 1,000+ stars: 60.05%

3. The effect is not simply caused by different agents being used in different repositories.
   The decline in merge rates with increasing repository popularity appears within individual agents as well, suggesting that repository characteristics themselves play an important role.

4. Previous research may be affected by selection bias.
   The enriched portion of AIDev used in earlier research contains only repositories with at least 100 stars. These repositories represent just 2.7% of decided PRs, yet their merge rate is 73.67%, compared with 90.63% for the rest of the dataset—a difference of almost 17 percentage points.

5. Agent comparisons can change when repository effects are controlled.
   When agents are compared within the same repositories using the Mantel-Haenszel approach, some previously reported differences change substantially. Two of fifteen pairwise comparisons reverse direction, while another loses more than half of its original magnitude.

6. Decision time also varies with repository popularity.
   The paper finds that the relationship is not limited to whether a PR is merged. The time required to reach a decision also differs across repository popularity levels.

7. Human review is surprisingly limited.
   Even among highly watched repositories, approximately three quarters of agent PRs have no recorded human review. This suggests that a high merge rate should not automatically be interpreted as evidence of extensive human validation.

Overall conclusion

The central conclusion is that an AI coding agent's merge rate is not purely a property of the agent. It is influenced substantially by the repository in which the agent operates.

In simple terms:

Observed agent performance = agent effects + repository effects + other contextual factors

Therefore, comparing agents using a single pooled merge rate can produce misleading conclusions. Future studies should control for repository characteristics, particularly repository popularity, rather than treating every merge or rejection as an independent indicator of agent quality.

Conclusion

Across 2,451,452 decided pull requests written by six coding agents, the single strongest thing determining whether an agent’s work is accepted is not which agent wrote it but where it was sent. Acceptance falls from 91.58 percent in repositories with no stars to 60.05 percent in those with a thousand or more, and that decline is present inside every agent examined. The portion of the data on which the published literature rests consists entirely of repositories above 100 stars, holds 2.7 percent of the decided pull requests, and merges them roughly seventeen percentage points below the rest of the population. Holding the repository fixed changes the comparison between agents in three of fifteen cases and reverses it in two, including one where a threefold apparent advantage becomes no measurable difference. The same split shows in decision time, from a median of 51 seconds in unstarred repositories to 5.4 hours in the most popular. Even inside the most scrutinised repositories available, three quarters of agent pull requests carry no recorded human review. The conclusion is not that coding agents are worse than reported, nor better. It is that the number by which they are commonly judged belongs partly to the projects that receive their work, and that any study reporting it should say which projects those were.

References

[1] H. Li, H. Zhang, and A. E. Hassan, “AIDev: Studying AI Coding Agents on GitHub,” arXiv preprint arXiv:2602.09185, 2026. [Online]. Available: https://arxiv.org/abs/2602.09185 [2] R. Robbes, T. Matricon, T. Degueule, A. Hora, and S. Zacchiroli, “Agentic Much? Adoption of Coding Agents on GitHub,” arXiv preprint arXiv:2601.18341, 2026. [Online]. Available: https://arxiv.org/abs/2601.18341 [3] T. A. Ghaleb, “Fingerprinting AI Coding Agents on GitHub,” arXiv preprint arXiv:2601.17406, 2026. [Online]. Available: https://arxiv.org/abs/2601.17406 [4] [4] M. Watanabe, H. Li, Y. Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub,” arXiv preprint arXiv:2509.14745, 2025. [Online]. Available: https://arxiv.org/abs/2509.14745 [5] G. Pinna, J. Gong, D. Williams, and F. Sarro, “Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance,” arXiv preprint arXiv:2602.08915, 2026. [Online]. Available: https://arxiv.org/abs/2602.08915 [6] H. Yoshioka, T. Monno, H. Tokumasu, T. Wakamatsu, Y. Ota, N. Weeraddana, and K. Matsumoto, “Let’s Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests,” arXiv preprint arXiv:2601.18749, 2026. [Online]. Available: https://arxiv.org/abs/2601.18749 [7] K. Duma, P. Wróblewski, J. Bobi?ska, J. Winiarska, and P. Przymus, “These Aren’t the Reviews You’re Looking For How Humans Review AI-Generated Pull Requests,” arXiv preprint arXiv:2605.02273, 2026. [Online]. Available: https://arxiv.org/abs/2605.02273 [8] C. Nachuma and M. Zibran, “When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests,” arXiv preprint arXiv:2602.19441, 2026. [Online]. Available: https://arxiv.org/abs/2602.19441 [9] M. L. Siddiq, X. Zhao, V. C. Lopes, B. Casey, and J. C. S. Santos, “Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub,” arXiv preprint arXiv:2601.00477, 2026. [Online]. Available: https://arxiv.org/abs/2601.00477 [10] M. Abujadallah, A. Arabat, and M. Sayagh, “Understanding the Rejection of Fixes Generated by Agentic Pull Requests — Insights from the AIDev Dataset,” arXiv preprint arXiv:2606.13468, 2026. [Online]. Available: https://arxiv.org/abs/2606.13468 [11] S. Sawada, T. Shirai, Y. Kashiwa, K. Yamaguchi, H. Iwata, and H. Iida, “To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study,” arXiv preprint arXiv:2605.06464, 2026. [Online]. Available: https://arxiv.org/abs/2605.06464 [12] M. Wessel, A. Serebrenik, I. Wiese, I. Steinmacher, and M. A. Gerosa, “Quality Gatekeepers: Investigating the Effects of Code Review Bots on Pull Request Activities,” arXiv preprint arXiv:2103.13547, 2021. [Online]. Available: https://arxiv.org/abs/2103.13547 [13] M. Wyrich, R. Ghit, T. Haller, and C. Müller, “Bots Don’t Mind Waiting, Do They? Comparing the Interaction With Automatically and Manually Created Pull Requests,” arXiv preprint arXiv:2103.03591, 2021. [Online]. Available: https://arxiv.org/abs/2103.03591 [14] M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments,” arXiv preprint arXiv:2010.03303, 2020. [Online]. Available: https://arxiv.org/abs/2010.03303 [15] X. Zhang, Y. Yu, G. Gousios, and A. Rastogi, “Pull Request Decision Explained: An Empirical Overview,” arXiv preprint arXiv:2105.13970, 2021. [Online]. Available: https://arxiv.org/abs/2105.13970 [16] D. Legay, A. Decan, and T. Mens, “On the impact of pull request decisions on future contributions,” arXiv preprint arXiv:1812.06269, 2018. [Online]. Available: https://arxiv.org/abs/1812.06269 [17] K. A. Hasan, M. Macedo, Y. Tian, B. Adams, and S. Ding, “Understanding the Time to First Response In GitHub Pull Requests,” arXiv preprint arXiv:2304.08426, 2023. [Online]. Available: https://arxiv.org/abs/2304.08426 [18] S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” arXiv preprint arXiv:2302.06590, 2023. [Online]. Available: https://arxiv.org/abs/2302.06590 [19] F. Xu, P. K. Medappa, M. M. Tunc, M. Vroegindeweij, and J. C. Fransoo, “AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden,” arXiv preprint arXiv:2510.10165, 2025. [Online]. Available: https://arxiv.org/abs/2510.10165 [20] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv preprint arXiv:2310.06770, 2023. [Online]. Available: https://arxiv.org/abs/2310.06770

Copyright

Copyright © 2026 Perseus Bhavnagri. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.