80 lines
4.1 KiB
Markdown
80 lines
4.1 KiB
Markdown
# Multi-Nexus Python Package Manager with CI/CD Integration
|
|
|
|
## Overview
|
|
This project is a robust and flexible tool designed for managing Python package dependencies across multiple Nexus repositories. It integrates with CI/CD pipelines to automate the process of fetching, uploading, and managing Python packages, ensuring a streamlined and secure workflow for Python projects. The tool includes functionality for removing unused packages, checking for known vulnerabilities, and handling dependencies across various repositories.
|
|
|
|
## Features
|
|
|
|
1. **Multi-Nexus Integration:**
|
|
- Capable of interacting with multiple Nexus repositories.
|
|
- Easily expandable to handle more than the currently supported two Nexuses.
|
|
|
|
2. **CI/CD Pipeline Integration:**
|
|
- **Nexus_Cleanup**
|
|
- Clone additional repositories required for the project.
|
|
- Aggregate and analyze all `requirements.txt` files to identify necessary dependencies.
|
|
- Compare local dependencies with those available on Nexus and remove any unused packages.
|
|
- **Nexus_Vulnerability**
|
|
- Check for known vulnerabilities in the packages uploaded to Nexus.
|
|
- **Nexus_Upload**
|
|
- This is normally part of extra repositories. Example file extra_repo.yml
|
|
- Clone CICD repository into the extra repository
|
|
- Conduct a comparison of Nexus contents versus local requirements.
|
|
- Check for vulnerabilities in missing packages before downloading and uploading them to Nexus.
|
|
|
|
3. **Package Management:**
|
|
- **Fetch Packages from Nexus:** Retrieves all Python packages and their versions from multiple Nexus repositories.
|
|
- **Upload Wheel Files:** Automates the upload of .whl files to the Nexus repositories using the `twine` tool.
|
|
- **Delete Unused Packages:** Removes packages that are no longer needed from the Nexus repositories.
|
|
|
|
4. **Vulnerability Checking:**
|
|
- Utilizes the `safety` tool to check for known vulnerabilities in the packages.
|
|
|
|
5. **Automated Package Handling:**
|
|
- Downloads missing package versions as .whl files and uploads them to Nexus using the REST API.
|
|
|
|
## Technical Details
|
|
- **Modular Design:** The project is divided into two main classes (`Nexus` and `Project`) to handle Nexus-related operations and project-specific dependency management, respectively.
|
|
- **Authentication:** Uses environment variables to securely manage Nexus credentials.
|
|
- **Web Scraping:** Employs BeautifulSoup for parsing HTML content to retrieve package and version information.
|
|
- **Command Execution:** Utilizes `subprocess` for executing shell commands, such as `twine upload` and `safety check`.
|
|
- **Error Handling:** Implements robust error handling to manage exceptions and ensure smooth execution.
|
|
- **Configuration Flexibility:** The tool can be configured to handle any number of Nexus repositories.
|
|
|
|
## Use Cases
|
|
- **CI/CD Pipelines:** Integrate with continuous integration and deployment pipelines to automate dependency management and package uploads.
|
|
- **Dependency Audits:** Regularly check for and address vulnerabilities in project dependencies.
|
|
- **Repository Cleanup:** Maintain clean and efficient Nexus repositories by removing unused packages.
|
|
|
|
## Technologies Used
|
|
|
|
- **Programming Language:**
|
|
- Python
|
|
|
|
- **Libraries and Frameworks:**
|
|
- `requests`: For making HTTP requests to Nexus and PyPI.
|
|
- `BeautifulSoup`: For parsing HTML content.
|
|
- `subprocess`: For executing shell commands.
|
|
- `json`: For handling JSON data.
|
|
|
|
- **Tools:**
|
|
- `twine`: For uploading Python packages to Nexus.
|
|
- `safety`: For checking package vulnerabilities.
|
|
|
|
- **Environment Management:**
|
|
- `os`: For handling environment variables and file operations.
|
|
- `Pathlib`: For path manipulations and directory operations.
|
|
|
|
- **Continuous Integration/Continuous Deployment (CI/CD):**
|
|
- YAML: For defining CI/CD pipeline stages.
|
|
- Nexus Repository Manager: For storing and managing Python packages.
|
|
- Git: For cloning repositories and managing version control.
|
|
|
|
- **Others:**
|
|
- PyPI: As the source for fetching the latest package versions.
|
|
|
|
|
|
## License
|
|
This project is licensed under the terms of the [Proprietary License](LICENSE).
|
|
|