karlji
/
CICD_Nexus_PyPi
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Multi-Nexus Python Package Manager with CI/CD Integration.
2 Commits 1 Branch 0 Tags 35 KiB
Python 100%
Go to file
karlji ba539a1890 Upload files to "/" 2024-08-09 16:46:00 +00:00
.gitlab-ci.yml Upload files to "/" 2024-08-09 16:45:38 +00:00
LICENSE Upload files to "/" 2024-08-09 16:45:38 +00:00
README.md Upload files to "/" 2024-08-09 16:46:00 +00:00
cicd_parser.py Upload files to "/" 2024-08-09 16:45:38 +00:00
cicd_requirements.txt Upload files to "/" 2024-08-09 16:45:38 +00:00
extra_repo.yml Upload files to "/" 2024-08-09 16:45:38 +00:00
module_fetch.py Upload files to "/" 2024-08-09 16:46:00 +00:00

README.md

Multi-Nexus Python Package Manager with CI/CD Integration

Overview

This project is a robust and flexible tool designed for managing Python package dependencies across multiple Nexus repositories. It integrates with CI/CD pipelines to automate the process of fetching, uploading, and managing Python packages, ensuring a streamlined and secure workflow for Python projects. The tool includes functionality for removing unused packages, checking for known vulnerabilities, and handling dependencies across various repositories.

Features

  1. Multi-Nexus Integration:

    • Capable of interacting with multiple Nexus repositories.
    • Easily expandable to handle more than the currently supported two Nexuses.

  2. CI/CD Pipeline Integration:

    • Nexus_Cleanup
      • Clone additional repositories required for the project.
      • Aggregate and analyze all requirements.txt files to identify necessary dependencies.
      • Compare local dependencies with those available on Nexus and remove any unused packages.
    • Nexus_Vulnerability
      • Check for known vulnerabilities in the packages uploaded to Nexus.
    • Nexus_Upload
      • This is normally part of extra repositories. Example file extra_repo.yml
      • Clone CICD repository into the extra repository
      • Conduct a comparison of Nexus contents versus local requirements.
      • Check for vulnerabilities in missing packages before downloading and uploading them to Nexus.

  3. Package Management:

    • Fetch Packages from Nexus: Retrieves all Python packages and their versions from multiple Nexus repositories.
    • Upload Wheel Files: Automates the upload of .whl files to the Nexus repositories using the twine tool.
    • Delete Unused Packages: Removes packages that are no longer needed from the Nexus repositories.

  4. Vulnerability Checking:

    • Utilizes the safety tool to check for known vulnerabilities in the packages.

  5. Automated Package Handling:

    • Downloads missing package versions as .whl files and uploads them to Nexus using the REST API.

Technical Details

  • Modular Design: The project is divided into two main classes (Nexus and Project) to handle Nexus-related operations and project-specific dependency management, respectively.
  • Authentication: Uses environment variables to securely manage Nexus credentials.
  • Web Scraping: Employs BeautifulSoup for parsing HTML content to retrieve package and version information.
  • Command Execution: Utilizes subprocess for executing shell commands, such as twine upload and safety check.
  • Error Handling: Implements robust error handling to manage exceptions and ensure smooth execution.
  • Configuration Flexibility: The tool can be configured to handle any number of Nexus repositories.

Use Cases

  • CI/CD Pipelines: Integrate with continuous integration and deployment pipelines to automate dependency management and package uploads.
  • Dependency Audits: Regularly check for and address vulnerabilities in project dependencies.
  • Repository Cleanup: Maintain clean and efficient Nexus repositories by removing unused packages.

Technologies Used

  • Programming Language:

    • Python

  • Libraries and Frameworks:

    • requests: For making HTTP requests to Nexus and PyPI.
    • BeautifulSoup: For parsing HTML content.
    • subprocess: For executing shell commands.
    • json: For handling JSON data.

  • Tools:

    • twine: For uploading Python packages to Nexus.
    • safety: For checking package vulnerabilities.

  • Environment Management:

    • os: For handling environment variables and file operations.
    • Pathlib: For path manipulations and directory operations.

  • Continuous Integration/Continuous Deployment (CI/CD):

    • YAML: For defining CI/CD pipeline stages.
    • Nexus Repository Manager: For storing and managing Python packages.
    • Git: For cloning repositories and managing version control.

  • Others:

    • PyPI: As the source for fetching the latest package versions.

License

This project is licensed under the terms of the Proprietary License.