Human-Centered Image Manipulation Using Deep Learning

Abstract

The ability to intuitively manipulate human images with precision is a key requirement in fields like digital art, content creation, and virtual avatars. This project presents a deep learning-based framework for human-centered image manipulation, allowing users to interactively control facial features, expressions, and poses using simple point-based inputs. Built upon a generative adversarial network (GAN) architecture, the system enables users to click and drag specific points on a human image to reposition them in real-time, maintaining visual coherence throughout the transformation. Unlike traditional methods that rely on semantic labels or 3D models, our approach employs feature-guided motion supervision and point tracking within the latent space of StyleGAN2 to achieve fine-grained, photo-realistic editing. A key enhancement in this work is the integration of real image support through GAN inversion, enabling users to upload actual human photos for personalized manipulation. Experimental evaluations demonstrate the system\'s effectiveness across various human-centered attributes, achieving smooth and realistic results. This work contributes an intuitive, flexible, and efficient solution for interactive human image editing using deep generative models.

Introduction

The rapid progress in deep learning, particularly Generative Adversarial Networks (GANs) like StyleGAN2, has revolutionized image synthesis and editing, enabling photorealistic and controllable image generation. However, precise, intuitive, and user-friendly manipulation of human-centric image attributes (e.g., facial expressions, poses) remains challenging. Existing methods often depend on semantic labels, text prompts, or 3D models, which can limit flexibility and real-time interactivity.

This project introduces a novel framework inspired by DragGAN that allows interactive, point-based editing of human images by dragging "handle points" to "target points," optimizing changes within the GAN’s latent space using feature-based motion supervision and point tracking. A significant advancement is integrating GAN inversion techniques to enable editing of real photographs, broadening practical applications such as photo retouching and avatar creation.

The system leverages StyleGAN2 for image generation, uses datasets like FFHQ and StyleGAN-Human for training and evaluation, and incorporates GAN inversion methods (PTI, ReStyle) for real image editing. The method optimizes latent vectors to reflect spatial transformations with precise motion supervision and tracking, preserving photorealism and identity during edits.

Extensive experiments show that the framework supports realistic, high-quality human image manipulations including facial pose, expression changes, and full-body adjustments, with strong visual fidelity, accuracy, and interactive usability.

Conclusion

This paper presented a novel and interactive framework for human-centered image manipulation using deep generative models. By leveraging the power of StyleGAN2 and incorporating a point-based editing mechanism inspired by DragGAN [69], the proposed system enables precise and intuitive control over facial features, poses, and expressions in both synthetic and real images. Unlike traditional approaches that rely on text prompts, semantic labels, or 3D priors, our method allows users to directly manipulate image content by dragging selected points to desired positions. The integration of motion supervision in feature space, along with feature-based point tracking, ensures photorealistic and semantically consistent transformations. Additionally, the system supports real image editing via GAN inversion techniques such as PTI and ReStyle, significantly enhancing its practical applicability. Experimental results demonstrate the system\'s effectiveness in terms of editing accuracy, visual quality, and usability. Compared to existing baseline methods, our approach achieves lower landmark error, better perceptual similarity, and real-time responsiveness with a user-friendly interface. Overall, the proposed system offers a powerful, flexible, and accessible solution for personalized image editing, virtual avatars, and creative content generation..

References

[1] S. R. Abdal, Y. Qin, and P. Wonka, \"Image2StyleGAN: How to Embed Images into the StyleGAN Latent Space?,\" in Proc. ICCV, 2019. [2] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka, \"StyleFlow: Attribute-conditioned Exploration of StyleGAN-generated Images using Conditional Continuous Normalizing Flows,\" ACM Trans. Graph., vol. 40, no. 3, pp. 1–21, 2021. [3] T. Beier and S. Neely, \"Feature-based Image Metamorphosis,\" in Seminal Graphics Papers: Pushing the Boundaries, vol. 2, pp. 529–536, 2023. [4] M. Botsch and O. Sorkine, \"On Linear Variational Surface Deformation Methods,\" IEEE Trans. Vis. Comput. Graph., vol. 14, no. 1, pp. 213–230, 2007. [5] T. Brox and J. Malik, \"Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation,\" IEEE TPAMI, vol. 33, no. 3, pp. 500–513, 2010. [6] E. R. Chan et al., \"Efficient Geometry-aware 3D Generative Adversarial Networks,\" in Proc. CVPR, 2022. [7] E. R. Chan et al., \"pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis,\" in Proc. CVPR, 2021. [8] A. Chen et al., \"SofGAN: A Portrait Image Generator with Dynamic Styling,\" ACM Trans. Graph., vol. 41, no. 1, pp. 1–26, 2022. [9] Y. Choi et al., \"StarGAN v2: Diverse Image Synthesis for Multiple Domains,\" in Proc. CVPR, 2020. [10] E. Collins et al., \"Editing in Style: Uncovering the Local Semantics of GANs,\" in Proc. CVPR, pp. 5771–5780, 2020. [11] A. Creswell et al., \"Generative Adversarial Networks: An Overview,\" IEEE Signal Process. Mag., vol. 35, no. 1, pp. 53–65, 2018. [12] Y. Deng et al., \"Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning,\" in Proc. CVPR, 2020. [13] A. Dosovitskiy et al., \"FlowNet: Learning Optical Flow with Convolutional Networks,\" in Proc. ICCV, 2015. [14] Y. Endo, \"User-Controllable Latent Transformer for StyleGAN Image Layout Editing,\" Comput. Graph. Forum, vol. 41, no. 7, pp. 395–406, 2022. [15] D. Epstein et al., \"BlobGAN: Spatially Disentangled Scene Representations,\" in Proc. ECCV, pp. 616–635, 2022. [16] J. Fu et al., \"StyleGAN-Human: A Data-Centric Odyssey of Human Generation,\" in Proc. ECCV, 2022. [17] P. Ghosh et al., \"GIF: Generative Interpretable Faces,\" in Proc. 3DV, 2020. [18] D. B. Goldman et al., \"Video Object Annotation, Navigation, and Composition,\" in Proc. ACM UIST, pp. 3–12, 2008. [19] I. Goodfellow et al., \"Generative Adversarial Nets,\" in Proc. NeurIPS, 2014. [20] J. Gu et al., \"StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis,\" in Proc. ICLR, 2022. [21] E. Härkönen et al., \"GANSpace: Discovering Interpretable GAN Controls,\" arXiv preprint arXiv:2004.02546, 2020. [22] A. W. Harley et al., \"Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories,\" in Proc. ECCV, 2022. [23] J. Ho et al., \"Denoising Diffusion Probabilistic Models,\" in Proc. NeurIPS, 2020. [24] T. Igarashi et al., \"As-Rigid-as-Possible Shape Manipulation,\" ACM Trans. Graph., vol. 24, no. 3, pp. 1134–1141, 2005. [25] E. Ilg et al., \"FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks,\" in Proc. CVPR, 2017. [26] P. Isola et al., \"Image-to-Image Translation with Conditional Adversarial Networks,\" in Proc. CVPR, 2017. [27] T. Karras et al., \"Alias-Free Generative Adversarial Networks,\" in Proc. NeurIPS, 2021. [28] T. Karras, S. Laine, and T. Aila, \"A Style-based Generator Architecture for Generative Adversarial Networks,\" in Proc. CVPR, pp. 4401–4410, 2019. [29] T. Karras et al., \"Analyzing and Improving the Image Quality of StyleGAN,\" in Proc. CVPR, pp. 8110–8119, 2020. [30] D. E. King, \"Dlib-ml: A Machine Learning Toolkit,\" JMLR, vol. 10, pp. 1755–1758, 2009. [31] D. P. Kingma and J. Ba, \"Adam: A Method for Stochastic Optimization,\" arXiv preprint arXiv:1412.6980, 2014. [32] T. Leimkühler and G. Drettakis, \"FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold,\" ACM Trans. Graph., vol. 40, no. 6, 2021. [33] H. Ling et al., \"EditGAN: High-Precision Semantic Image Editing,\" in Proc. NeurIPS, 2021. [34] Y. Lipman et al., \"Differential Coordinates for Interactive Mesh Editing,\" in Proc. Shape Modeling Applications, IEEE, pp. 181–190, 2004. [35] Y. Lipman et al., \"Linear Rotation-Invariant Coordinates for Meshes,\" ACM Trans. Graph., vol. 24, no. 3, pp. 479–487, 2005. [36] R. Mokady et al., \"Self-distilled StyleGAN: Towards Generation from Internet Photos,\" in ACM SIGGRAPH Conf. Proc., pp. 1–9, 2022. [37] X. Pan et al., \"A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis,\" in Proc. NeurIPS, 2021. [38] T. Park et al., \"Semantic Image Synthesis with Spatially-Adaptive Normalization,\" in Proc. CVPR, 2019. [39] A. Paszke et al., \"Automatic Differentiation in PyTorch,\" 2017. [40] O. Patashnik et al., \"StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery,\" in Proc. ICCV, 2021. [41] J. N. M. Pinkney, \"Awesome Pretrained StyleGAN2,\" GitHub, 2020. [Online]. Available: https://github.com/justinpinkney/awesome-pretrained-stylegan2 [42] A. Ramesh et al., \"Hierarchical Text-Conditional Image Generation with CLIP Latents,\" arXiv preprint arXiv:2204.06125, 2022. [43] D. Roich et al., \"Pivotal Tuning for Latent-Based Editing of Real Images,\" ACM Trans. Graph., vol. 42, no. 1, pp. 1–13, 2022. [44] R. Rombach et al., \"High-Resolution Image Synthesis with Latent Diffusion Models,\" arXiv preprint arXiv:2112.10752, 2021. [45] C. Saharia et al., \"Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding,\" arXiv preprint arXiv:2205.11487, 2022. [46] K. Schwarz et al., \"GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis,\" in Proc. NeurIPS, 2020. [47] Y. Shen et al., \"Interpreting the Latent Space of GANs for Semantic Face Editing,\" in Proc. CVPR, 2020. [48] Y. Shen and B. Zhou, \"Closed-Form Factorization of Latent Semantics in GANs,\" arXiv preprint arXiv:2007.06600, 2020. [49] I. Skorokhodov et al., \"Aligning Latent and Image Spaces to Connect the Unconnectable,\" arXiv preprint arXiv:2104.06954, 2021. [50] J. Sohl-Dickstein et al., \"Deep Unsupervised Learning Using Nonequilibrium Thermodynamics,\" in Proc. ICML, PMLR, pp. 2256–2265, 2015. [51] J. Song, C. Meng, and S. Ermon, \"Denoising Diffusion Implicit Models,\" in Proc. ICLR, 2020. [52] Y. Song et al., \"Score-Based Generative Modeling through Stochastic Differential Equations,\" in Proc. ICLR, 2021. [53] O. Sorkine and M. Alexa, \"As-Rigid-as-Possible Surface Modeling,\" in Symp. Geometry Processing, vol. 4, pp. 109–116, 2007. [54] O. Sorkine et al., \"Laplacian Surface Editing,\" in Proc. Eurographics/ACM SIGGRAPH Symp. Geometry Processing, pp. 175–184, 2004. [55] N. Sundaram, T. Brox, and K. Keutzer, \"Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow,\" in Proc. ECCV, 2010. [56] R. Suzuki et al., \"Spatially Controllable Image Synthesis with Internal Representation Collaging,\" arXiv preprint arXiv:1811.10153, 2018. [57] Z. Teed and J. Deng, \"RAFT: Recurrent All-Pairs Field Transforms for Optical Flow,\" in Proc. ECCV, 2020. [58] A. Tewari et al., \"Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images,\" in Proc. CVPR, 2022. [59] A. Tewari et al., \"StyleRig: Rigging StyleGAN for 3D Control over Portrait Images,\" in Proc. CVPR, 2020. [60] N. Tritrong et al., \"Repurposing GANs for One-Shot Semantic Part Segmentation,\" in Proc. CVPR, pp. 4475–4485, 2021. [61] J. Wang et al., \"Improving GAN Equilibrium by Raising Spatial Awareness,\" in Proc. CVPR, pp. 11285–11293, 2022. [62] S.-Y. Wang, D. Bau, and J.-Y. Zhu, \"Rewriting Geometric Rules of a GAN,\" ACM Trans. Graph., 2022. [63] Y. Xu et al., \"3D-aware Image Synthesis via Learning Structural and Textural Representations,\" in Proc. CVPR, 2022. [64] F. Yu et al., \"LSUN: Construction of a Large-Scale Image Dataset Using Deep Learning with Humans in the Loop,\" arXiv preprint arXiv:1506.03365, 2015. [65] R. Zhang et al., \"The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,\" in Proc. CVPR, 2018. [66] Y. Zhang et al., \"DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort,\" in Proc. CVPR, 2021. [67] J. Zhu et al., \"LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis,\" arXiv preprint arXiv:2301.04604, 2023. [68] J.-Y. Zhu et al., \"Generative Visual Manipulation on the Natural Image Manifold,\" in Proc. ECCV, 2016 [69] X. Pan, Y. Xu, X. Dai, E. T. Wang, J. Zhu, B. Dai, and C. Theobalt, “Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold,” arXiv preprint arXiv:2305.10973, 2023.

Copyright

Copyright © 2025 Y. Mallikarjun, Dr. V. Uma Rani, Sunitha Vanamala. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.