Benchmarking Security and Efficiency of Locally Hosted Large Language Models for Code Generation

Abstract

The rapid evolution of Large Language Models (LLMs) has revolutionized software development, with tools like GitHub Copilot and ChatGPT setting the standard for automated code generation. However, the reliance on cloud-based models raises significant concerns regarding data privacy, latency, and operational costs. Consequently, there is a growing shift towards locally hosted open-weights models that can run on consumer-grade hardware (16-32GB RAM). Despite this trend, limited empirical research exists to quantify whether these smaller, resource-constrained models (7B-9B parameters) can match the security and efficiency of human-written code or larger cloud counterparts. This research aims to conduct a comparative analysis of the security and efficiency of five state-of-the-art local LLMs: Mistral 7B, LLaMA 3.1 8B, Gemma 2 9B, Qwen 2.5 7B, and Phi-3 3.8B. Using a controlled experimental setup on medium-configuration hardware, the study evaluates Python code generation across a suite of algorithmic tasks. Security vulnerabilities were assessed using static analysis tools (Bandit), while efficiency metrics—including execution runtime (ms) and peak memory usage (KB)—were measured to determine the suitability of these models for edge deployment. The results demonstrate that Mistral 7B and Qwen 2.5 achieve superior performance, delivering 100% functional correctness with zero security flaws and minimal resource consumption. By benchmarking these models, this research provides critical insights into the viability of privacy-preserving, local AI coding assistants, helping developers select optimal models for secure and efficient offline software development.

Introduction

Rapid advances in artificial intelligence and large language models (LLMs) have reshaped software engineering by enabling automated code generation, significantly improving developer productivity. However, reliance on cloud-based proprietary models such as GitHub Copilot and ChatGPT raises concerns around data privacy, intellectual property leakage, and dependency on external APIs. To address these issues, organizations are increasingly exploring locally hosted, open-weight LLMs (e.g., LLaMA, Mistral, Gemma, Qwen) that run offline on consumer-grade hardware. While these models offer privacy and autonomy benefits, their ability to generate secure and efficient code comparable to human-written solutions remains uncertain.

This study investigates whether smaller, resource-constrained local LLMs (3B–9B parameters) can serve as viable alternatives for software development. It focuses on two critical dimensions: security, examining vulnerabilities such as hardcoded credentials and injection flaws, and efficiency, assessing execution time and memory usage compared to human-written code. Existing research largely centers on cloud-based models and often evaluates security or efficiency in isolation, leaving a gap in understanding the trade-offs faced by local models deployed on edge devices.

To bridge this gap, the research conducts a controlled, offline experimental comparison between five local LLMs (Mistral, LLaMA, Gemma, Phi-3, and Qwen) and a human-written Python baseline. A custom benchmark of six algorithmic tasks was created, covering string manipulation, numeric computation, and array operations. All models were tested under identical conditions on consumer hardware (32GB RAM), using standardized prompts, quantized models, and deterministic inference settings.

The evaluation framework measures correctness (functional test pass rate), efficiency (runtime and memory usage), and security (static analysis using Bandit with CWE-based vulnerability detection). Automated pipelines ensured reproducibility through sandboxed execution, consistent logging, and structured performance analysis.

References

[1] M. Ahmed, \"Integrating AI-Driven Automated Code Review in Agile Development: Benefits, Challenges, and Best Practices,\" Proc. Of the International Conference on Software Engineering, 2025. [2] S. Mishra, “AI-Augmented Vulnerability Detection and Patching,” IEEE Transactions on Software Engineering, vol. 50, no.1, pp. 72-84, 2024. [3] Y. Li, A. Ramanathan, and K. Zhao, “Assessing the Performance of AI-Generated Code: A Case Study on GitHub Copilot,” arXiv preprint arXiv:2401.01234, 2024.

Copyright

Copyright © 2025 Ram Krishna Kumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.