Improved Environment Management for CICD and integration

The proposed feature aims to significantly enhance the developer experience by providing a consolidated view of CI and integration environments, reducing the time developers spend searching for critical information and expediting debugging and resolution processes. Automated management and fair distribution of runner workloads will optimise resource usage and ensure a stable CI/CD pipeline. By collecting and analysing metrics on resource usage and pipeline failures, teams will gain actionable insights, leading to continuous improvement of CI/CD processes. Encouraging the setup of persistent or better-isolated testing environments will result in more reliable and reproducible test results, thereby improving software quality. Ultimately, identifying and eliminating bottlenecks will streamline the integration process, reducing the overall time to integrate for new features and updates, thereby increasing efficiency and productivity across the board.

• Improved View for CI and Integration Environments:
  • Developers can access a unified dashboard that displays all relevant information about their CI and integration tests, including:
    • Logs
    • Resource usage
    • Pod status
    • Links to Test results
  • Navigation within the dashboards for above is intuitive and well documented, allowing easy location of specific information.
• Automated Management of CI and Integration Environments:
  • All CI and integration environments are automatically annotated/tagged upon creation.
  • Runner workloads are visible to investigate how to distribute fairly across environments.
  • Runner and k8s resources are visible to investigate how to share fairly between deployments.
  • Metrics on resource usage are generated and shared with teams and business management. They should be able to grouped by repository, how many times a job has run
  • A system is in place to automatically collect and report on resource usage, network issues, and pipeline failures if they happen 
• Improved Pipeline Configuration:
  • Gitlab configuration is reviewed and optimised to improve the stopping of pipeline runs. Repositories are configured automatically
  • Persistent or better-isolated environments for testing are set up and maintained as a self serving system for teams
  • Data on pipeline failures (including resource usage, network issues, and successful re-runs) is collected and analysed for insights. They should be able to grouped by repository, how many times a job has run
• Bottleneck Identification and Elimination:
  • Bottlenecks related to integration environment usage (and pipeline machinery in general) and unnecessary time spent are identified and addressed.
  • Pipeline machinery and Gitlab metrics are utilised to streamline faster integration processes.

• New Grafana Gitlab Dashboards are generated https://monitoring.skao.int/d/gitlab_ci_jobs/gitlab-ci-jobs?orgId=1 , https://monitoring.skao.int/d/a87fb0d919ec0ea5f6543124e16c42a5/kubernetes-compute-resources-namespace-workloads?orgId=1 
• New Namespace Management dashbaords are generated with a new namespace manager (that we can extend to cover more complex scenarios such as clean CI namespaces only after a pod starts failing for 10 minutes etc.) https://monitoring.skao.int/d/e374e7bb-e223-4398-aaa8-f15845755fd6/namespace-manager-namespaces?orgId=1 
• New metrics for environment management are generated with accompanying dashboards https://monitoring.skao.int/d/edvfag7wkt7nke/namespace-manager-overall?orgId=1 
• Gitlab related settings are reviewed and automated (FF is turned off for now): https://developer.skao.int/en/latest/tools/ci-cd/gitlab-settings.html 
• Deployed Headlamp
• Pipeline Machinery Metrics are implemented and deployed: https://monitoring.skao.int/dashboards/f/ddvpmaqhx3klcb/