Case studyInfrastructure & reliability
Two data centers that run as one system
Two physical sites now operate as one shared set of services, so losing a single component no longer turns straight into an outage for users.
- Role
- System Administrator & DevOps Engineer
- Period
- 2024 — present
- Status
- Ongoing
- Used
Context and problem
The company infrastructure had grown without a consistent pattern: some services ran in one facility, some in another, and storage was never designed to back itself up. The result was that one failed device could become a service outage, and adding capacity mostly added new risk. This work reorganised both sites into one whole that can lose a component without stopping the service.
What made it hard
- The two physical sites ran differently enough that nobody had a single view of service health.
- Data had no distributed copy. If a storage device failed, recovery depended on backups rather than on the system still running.
- The internal name service was a single point. When one unit struggled, every service that calls another service queued up behind it.
- More than a thousand user accounts depended on a web server fleet that had to stay up while other components were repaired.
Constraints that could not move
- Both physical sites stay in use. Consolidating everything into one location was not an option.
- Running services could not be stopped for the redesign.
- Existing hardware had to be used, including rack installations handled by a third party.
- No additional headcount, while day-to-day operations continued alongside the work.
My role and responsibilities
- Designed and built two mutually supporting virtualisation clusters, each paired with distributed storage in its own location.
- Set up distributed object storage for internal use and a set of clustered databases.
- Replaced the internal name service with one primary unit and two secondary replicas.
- Arranged inbound traffic paths and perimeter protection, including certificate handling and request filtering.
- Handled tier-three escalation, including physical hardware installation inside the data centers.
How I went about it
- 01
Aligning the pattern before adding capacity
Both sites were built to the same pattern so one way of working applies everywhere, and nobody has to memorise two procedures for the same job.
- 02
Spreading data across many devices at once
Storage was designed so data lives on several units and stays readable when one is lost, rather than being concentrated on the fastest device.
- 03
Removing the single point in name resolution
One primary unit works alongside two secondary replicas, so a single failure does not stop internal name resolution.
- 04
Separating the network edge from the core
Inbound traffic passes through edge protection before touching internal services, and traffic between sites travels over encrypted tunnels.
- 05
Testing by taking one component away
Reliability was proven by actually powering down or removing a component while the service was running, not by assuming the design holds.
Two physical sites are a fact that cannot be worked around, so the question changed: not how to avoid running in two places, but how to make them cover for each other when one is in trouble. The previous design treated each site as an island. What happened here was giving both the same pattern, so a failed component stopped being a problem for every user.
Aligning the pattern before adding capacity
Adding hardware first would have multiplied the number of distinct ways of working rather than reducing them. So both sites were built to the same layout first: clear roles for each group of machines, the same maintenance approach, and the same naming. Capacity came after that, and adding it no longer forced anyone to learn a new procedure.
Spreading data so one failure stays small
Storage is what decides reliability. Data sits on several devices at once rather than on the fastest one, so losing a unit does not stop the service and repairs happen while the system keeps running. That choice costs capacity, and it was accepted because unused capacity is cheaper than a service that stops.
Keeping name resolution alive
The service that translates internal names is a small thing that disrupts almost everything else when it dies. It was rebuilt as one primary with two secondary replicas, so name resolution keeps working even when one unit is gone. The same thinking applies to the inbound traffic path and edge protection: centralised, but not dependent on a single point.
Results in short
56 TB of raw storage capacity is available across two clusters in two data centers. 12 production web servers serve more than 1,000 active user accounts with zero critical downtime, and the internal name service runs as one primary with two secondary replicas. Beyond the numbers, handling tier-three escalation means on-site incidents no longer pile up on one person.
Outcomes
56 TB of raw storage capacity is available across two clusters in two data centers.
Installed, usable capacity rather than a procurement plan.
verified
12 production web servers serve more than 1,000 active user accounts with zero critical downtime.
Some run on bare metal, some on virtual machines.
verified
The internal name service runs as one primary unit with two secondary replicas.
verified
With tier-three escalation handled properly, on-site incidents get resolved at the site instead of piling up on one person.
Technical detail
This part only matters if you want to judge the technical side. Everything above is enough to understand the outcome.
Open the implementation detail3 decisions
Two high-availability Proxmox VE clusters, each paired with a Ceph distributed storage cluster in its own location, totalling 56 TB of raw capacity. MinIO provides S3-compatible object storage, alongside clustered PostgreSQL databases. Internal name resolution runs PowerDNS with one primary and two replicas. Internal traffic goes through HAProxy and Nginx Proxy Manager, the edge is secured with Cloudflare, cross-site access runs over WireGuard, and mail runs on Mailcow with a WHM/Exim SMTP relay.
- Building two separate distributed storage clusters instead of one stretched across both sites.
- A single cluster is easier to view on one screen, but every write then depends on the quality of the inter-site link and adds latency for users. What was given up is ease of management and one unified view of capacity.
- Replicating data inside the distributed storage even though it consumes capacity.
- Internal replication turns the loss of one storage unit into a non-event for users, but part of the raw capacity is permanently spent on copies and can never hold data.
- Running virtualisation and storage on software over standard hardware rather than a single vendor managed platform.
- This keeps control of the data and avoids depending on one supplier, at the cost of owning upgrades, monitoring, and recovery ourselves.
The least visible part of this work was physical installation and escalation handling inside the data centers. It was also the part that taught the most about where the design fails, because problems show up directly instead of through a dashboard. What I would do differently is write a single-component recovery procedure earlier, because writing it after an incident is always slower.
What I learned
- Reliability is decided by how many components may be lost, not by how fast one component performs.
- Two sites built to the same pattern are easier to maintain than two sites each holding its own best solution.
- Tier-three escalation shows where the design is weak in the real world, and that only becomes visible if you handle incidents yourself.