Production-like DevOps Infrastructure Lab
I designed and implemented an end-to-end DevOps environment for a Python-based Monte Carlo Value-at-Risk application. The project combines Linux administration, Ansible automation, GitLab CI/CD, Docker, K3s, PostgreSQL, security hardening, backup and restore, and infrastructure monitoring.
One application, full operational lifecycle.
The goal was not to build a large application, but to build and operate a credible technical environment around it: repeatable provisioning, secure deployment, persistent data, backup, monitoring, and real troubleshooting.
Infrastructure
Multiple Linux servers with clearly separated responsibilities for GitLab, Kubernetes control plane, worker workloads, database, and monitoring.
Automation
Host configuration is managed with Ansible roles and playbooks instead of undocumented manual server changes.
Application Delivery
Git push triggers automated testing, image build, registry push and a controlled Kubernetes deployment.
End-to-end deployment flow
The application is a batch workload. Each successful delivery creates an immutable container image identified by the Git commit SHA and runs it as a Kubernetes Job.
Control Plane
A dedicated Rocky Linux server runs the K3s control plane and Kubernetes API. CI access is restricted by firewall rules and Kubernetes RBAC.
Worker
A separate Ubuntu worker executes application workloads. The batch application runs as a Kubernetes Job and terminates after the calculation is complete.
Infrastructure as repeatable configuration
Ansible manages the Linux hosts; Kubernetes manages the application workloads. This separation keeps OS-level configuration and application orchestration clear.
Base configuration
Common packages, timezone and consistent Linux server baseline.
SSH hardening
Direct root login and password authentication are disabled. Administration uses a dedicated Ansible user, SSH keys and sudo.
Firewall policy
Infrastructure ports are source-restricted. For example, the Kubernetes API and monitoring exporters are not opened globally.
K3s installation
Separate roles configure the control-plane server and worker agent. The node token is read at runtime rather than hardcoded in the repository.
PostgreSQL and backup
Database service, users, TLS, SCRAM authentication, access rules, scheduled backups and retention are managed through Ansible.
Monitoring
Node Exporter and PostgreSQL Exporter feed Prometheus. Grafana provides the operational dashboard.
From commit to running workload
The delivery pipeline uses three clear stages. A failed test prevents an image from being built and deployed.
1. Test
Python dependencies are installed in a clean environment and automated tests are executed with pytest.
2. Build
Docker builds a non-root application image. The image is tagged with the Git commit SHA and pushed to the GitLab Container Registry.
3. Deploy
GitLab CI uses a dedicated Kubernetes ServiceAccount with restricted RBAC permissions to create and monitor the batch Job.
A real deployment failure, traced end to end.
The project includes operational work, not only successful installation. One deployment incident became a useful troubleshooting case.
ImagePullBackOff caused by registry authentication
A deploy stage timed out because the Kubernetes Pod entered
ImagePullBackOff. I used kubectl describe to confirm the
image-pull failure. Because the container had never started, there were no
application logs. I then inspected the K3s agent logs and found an HTTP
401 Unauthorized response from the GitLab Registry authentication
endpoint. After verifying the imagePullSecret, I rotated the read-only registry
token, rebuilt the Kubernetes secret, tested the exact image pull independently,
and reran the deployment successfully.
Database Operations
PostgreSQL uses TLS and SCRAM-SHA-256 authentication. Backups are scheduled with a systemd timer, retention is enforced, and restore capability was tested.
Monitoring
Prometheus scrapes Linux node and PostgreSQL metrics. Grafana visualizes node status, CPU, memory, disk usage, database availability and active connections.