Forensic Analysis of Encrypted Communication Using OpenSSL Key Structures: A Comparative Study of RSA and Elliptic Curve Cryptography

Abstract

The rapid growth of digital communication has significantly increased the need for secure data transmission across modern networks. Cryptographic techniques play a vital role in protecting sensitive information by ensuring confidentiality, integrity, and authenticity. Among the most widely used public-key cryptographic systems, RSA and Elliptic Curve Cryptography (ECC) are extensively applied in areas such as secure web communication, digital certificates, authentication systems, and encrypted messaging platforms. While these techniques provide strong security, the processes involved in encryption and decryption often leave behind subtle traces within computing environments. These traces, known as cryptographic artefacts, can include key files, encrypted data, metadata, and command execution logs. Such artefacts can serve as valuable sources of evidence in digital forensic investigations, particularly when direct access to encrypted data is not possible. This study investigates the forensic artefacts generated during encryption and decryption processes using the OpenSSL toolkit. RSA and ECC key pairs were generated and used to encrypt and decrypt a sample plaintext file. The resulting outputs were analysed through command-line inspection and structural examination of key components. The findings reveal that encrypted data exhibits a high degree of randomness at the binary level, reflecting strong encryption characteristics. In addition, RSA private keys contain identifiable mathematical components, including modulus values and parameters associated with the Chinese Remainder Theorem. ECC, on the other hand, demonstrates comparable security while requiring significantly smaller key sizes, highlighting its efficiency in modern computing environments. Overall, this research demonstrates that encryption processes, although designed to protect data, leave behind detectable artefacts that can provide important insights for forensic investigators. These findings reinforce the importance of incorporating cryptographic awareness into digital forensic analysis.

Introduction

This study examines the forensic artifacts generated during cryptographic operations using the OpenSSL toolkit, focusing on two widely used public-key algorithms: RSA and Elliptic Curve Cryptography (ECC). As digital communication grows, encryption has become essential for protecting sensitive information, while also creating challenges for digital forensic investigations. Although encrypted data may be inaccessible without decryption keys, cryptographic processes leave behind identifiable traces such as key files, encrypted outputs, metadata, and command logs.

The research aims to analyze these artifacts by generating RSA and ECC key pairs, performing encryption and decryption on sample files, and examining the resulting outputs. Using OpenSSL in a Windows environment, the study investigates key structures, encrypted file characteristics, metadata, and hexadecimal representations of ciphertext. The scope is limited to controlled experiments and does not include attempts to break the encryption.

A literature review highlights that RSA remains popular due to its reliability and compatibility, while ECC offers comparable security with significantly smaller key sizes and better efficiency, especially for mobile and IoT devices. However, previous studies have focused mainly on security and performance, leaving a gap in understanding the forensic implications of real-world cryptographic implementations.

The experimental results show that encryption activities generate valuable forensic artifacts, including RSA and ECC key files, encrypted data, metadata, and command execution traces. Hexadecimal analysis revealed highly randomized ciphertext patterns, a characteristic of secure encryption. Metadata such as file creation times, modification dates, sizes, and storage locations can help investigators reconstruct encryption-related activities. The study also confirmed that ECC keys are much smaller than RSA keys while providing similar security levels, making ECC more resource-efficient.

Conclusion

This study examined the forensic artefacts generated during encryption experiments using the OpenSSL cryptographic toolkit. The primary objective of the research was to analyze how cryptographic operations leave identifiable traces within computing systems and to compare the structural characteristics of two widely used public-key cryptographic algorithms, RSA and Elliptic Curve Cryptography (ECC). The experimental procedure involved generating RSA and ECC key pairs, encrypting a plaintext file using the RSA public key, and subsequently decrypting the encrypted file using the corresponding private key. The successful completion of the encryption and decryption processes confirmed the correctness of the implemented cryptographic operations. During the experiment, several artefacts were generated including cryptographic key files, encrypted binary files, decrypted output files, and associated metadata. The forensic examination of these artefacts demonstrated that encryption processes leave identifiable traces within the file system. Files such as rsa_private.pem, rsa_public.pem, ecc_private.pem, and ecc_public.pem represent important indicators of cryptographic activity within a system. Additionally, encrypted files such as rsa_encrypted.bin exhibit high-entropy randomized byte patterns when examined at the hexadecimal level. These characteristics distinguish encrypted data from normal plaintext files and may assist digital forensic investigators in identifying encrypted evidence during investigations. Metadata analysis further revealed useful information including file creation time, modification time, file size, and file path location. Such metadata attributes can assist investigators in reconstructing timelines of cryptographic operations and determining when encryption activities occurred within a system environment. The comparative analysis between RSA and Elliptic Curve Cryptography also highlighted significant differences in key size efficiency. While both algorithms provide strong cryptographic security, ECC achieves comparable levels of security with significantly smaller key sizes compared to RSA. This efficiency makes ECC particularly suitable for modern computing environments including mobile devices, embedded systems, and resource-constrained platforms. Overall, the findings of this research emphasize the importance of understanding cryptographic artefacts in digital forensic investigations. Even when encrypted data cannot be directly decrypted, the presence of key files, encrypted data structures, and associated metadata may provide valuable indicators of encryption activity. The ability to identify and analyze such artefacts can assist investigators in reconstructing digital events and understanding the use of encryption within a system. Future research may explore automated detection of encrypted artefacts using entropy analysis tools, forensic frameworks for analyzing cryptographic key structures, and the integration of cryptographic artefact detection into digital forensic investigation workflows

References

[1] Yan, Y. (2022). The overview of elliptic curve cryptography (ECC). Journal of Physics: Conference Series, 2386(1). [2] Cao, Z., & Liu, L. (2024). The practical advantage of RSA over ECC and pairings. [3] Shah, A. M., & Gor, A. (2025). Comprehensive survey of symmetric and public-key cryptographic algorithms: Foundations, attacks, and applications. International Journal of Informative & Futuristic Research, 12(10). [4] Ramakrishna, D., & Shaik, M. A. (2025). A comprehensive analysis of cryptographic algorithms: Evaluating security, efficiency, and future challenges. IEEE Access, 13. [5] Arunkumar, B., & Kousalya, G. (2021). Secure and lightweight elliptic curve cipher suites in SSL/TLS. Computer Systems Science & Engineering. [6] Khan, M. R., et al. (2023). Analysis of elliptic curve cryptography & RSA. Journal of ICT Standardization, 11(4). [7] Ketha, A. (2024). The evolution of cryptography and a contextual analysis of the major modern schemes. [8] Weng, Z. (2025). Modern encryption algorithms comparative study: From symmetric to asymmetric systems. Academic Journal of Science and Technology. [9] Khalaf, A. O., Salah, S. K., Sartep, H. J., &Abdalrdha, Z. K. (2019). Comparison between RSA, ECC & NTRU algorithms. International Journal of Engineering Research and Advanced Technology, 5(11). https://doi.org/10.31695/IJERAT.2019.3582 [10] Mahto, D., & Yadav, D. K. (2017). RSA and ECC: A comparative analysis. International Journal of Applied Engineering Research. [11] Kumar, R. (2024). Design and analysis of computations using ECC and RSA. International Journal of Intelligent Systems and Applications in Engineering. [12] Dar, M. A., Askar, A., Alyahya, D., & Bhat, S. A. (2021). Lightweight and secure ECC key exchange for mobile phones. International Journal of Interactive Mobile Technologies. https://doi.org/10.3991/ijim.v15i23.26337

Copyright

Copyright © 2026 Divya U, Dr. Divya . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.