CICD_Nexus_PyPi/README.md

Multi-Nexus Python Package Manager with CI/CD Integration

Overview

This project is a robust and flexible tool designed for managing Python package dependencies across multiple Nexus repositories. It integrates with CI/CD pipelines to automate the process of fetching, uploading, and managing Python packages, ensuring a streamlined and secure workflow for Python projects. The tool includes functionality for removing unused packages, checking for known vulnerabilities, and handling dependencies across various repositories.

Features

1. Multi-Nexus Integration:

   • Capable of interacting with multiple Nexus repositories.
   • Easily expandable to handle more than the currently supported two Nexuses.

2. CI/CD Pipeline Integration:

   • Nexus_Cleanup
     • Clone additional repositories required for the project.
     • Aggregate and analyze all requirements.txt files to identify necessary dependencies.
     • Compare local dependencies with those available on Nexus and remove any unused packages.
   • Nexus_Vulnerability
     • Check for known vulnerabilities in the packages uploaded to Nexus.
   • Nexus_Upload
     • This is normally part of extra repositories. Example file extra_repo.yml
     • Clone CICD repository into the extra repository
     • Conduct a comparison of Nexus contents versus local requirements.
     • Check for vulnerabilities in missing packages before downloading and uploading them to Nexus.

3. Package Management:

   • Fetch Packages from Nexus: Retrieves all Python packages and their versions from multiple Nexus repositories.
   • Upload Wheel Files: Automates the upload of .whl files to the Nexus repositories using the twine tool.
   • Delete Unused Packages: Removes packages that are no longer needed from the Nexus repositories.

4. Vulnerability Checking:

   • Utilizes the safety tool to check for known vulnerabilities in the packages.

5. Automated Package Handling:

   • Downloads missing package versions as .whl files and uploads them to Nexus using the REST API.

Technical Details

• Modular Design: The project is divided into two main classes (Nexus and Project) to handle Nexus-related operations and project-specific dependency management, respectively.
• Authentication: Uses environment variables to securely manage Nexus credentials.
• Web Scraping: Employs BeautifulSoup for parsing HTML content to retrieve package and version information.
• Command Execution: Utilizes subprocess for executing shell commands, such as twine upload and safety check.
• Error Handling: Implements robust error handling to manage exceptions and ensure smooth execution.
• Configuration Flexibility: The tool can be configured to handle any number of Nexus repositories.

Use Cases

• CI/CD Pipelines: Integrate with continuous integration and deployment pipelines to automate dependency management and package uploads.
• Dependency Audits: Regularly check for and address vulnerabilities in project dependencies.
• Repository Cleanup: Maintain clean and efficient Nexus repositories by removing unused packages.

Technologies Used

• Programming Language:

  • Python

• Libraries and Frameworks:

  • requests: For making HTTP requests to Nexus and PyPI.
  • BeautifulSoup: For parsing HTML content.
  • subprocess: For executing shell commands.
  • json: For handling JSON data.

• Tools:

  • twine: For uploading Python packages to Nexus.
  • safety: For checking package vulnerabilities.

• Environment Management:

  • os: For handling environment variables and file operations.
  • Pathlib: For path manipulations and directory operations.

• Continuous Integration/Continuous Deployment (CI/CD):

  • YAML: For defining CI/CD pipeline stages.
  • Nexus Repository Manager: For storing and managing Python packages.
  • Git: For cloning repositories and managing version control.

• Others:

  • PyPI: As the source for fetching the latest package versions.

License

This project is licensed under the terms of the Proprietary License.