Hardening a Live AI Platform After the Migration
Four concurrent workstreams across cost, performance, observability and security, run on a rule that performance is measured before it is tuned and that a whole class of exposure is removed rather than defended.
- Cloud Optimization
- FinOps
- Observability
- Cloud Security
- Zero
- Permanently exposed administrative serversThere is no bastion host in this architecture: not a hardened one, not a locked-down one, not one behind a second factor. The category was removed rather than defended, and administrative access runs over VPN into a fully private network. Removing a class of exposure is a single decision; defending it is a permanent obligation.
- Four
- Workstreams, each with its own definition of doneCost, performance, observability, and security and governance, run concurrently across development, UAT and production rather than sequenced, because each one produced evidence the others needed.
- Daily
- Automated cost reporting by account, service and environmentReplacing a monthly total noticed after the fact. The first deliverable of the cost workstream was the visibility that funded the rest of it, because further optimization could then be argued from real numbers rather than intuition.
Overview
A live, multi-service AI platform was running on AWS in a European region, serving production traffic across three application services, and a migration had just completed cleanly. By any steady-state standard the environment was also wrong. Compute capacity was static and generous. Storage had no lifecycle rules. The production database ran continuously on on-demand pricing. Container images and objects accumulated with nothing retiring them. Monitoring was whatever had been sufficient to survive a cutover.
None of that was a mistake, and the distinction matters more than it first sounds. During a live traffic migration the correct instinct is to over-provision, because what you are protecting is continuity. A conservative capacity decision costs money. An aggressive one costs users.
So the debt was not a defect. It was the bill for having done the migration properly, and it came due afterwards.
The Context
The same European AI SaaS platform as the migration this follows, in its first steady state on the new platform: three application services across development, UAT and production, with the cutover behind it and the tuning still ahead.
The client, the region and the product services are not named. The cloud platform and its services are, because that describes what the work involved rather than who it was for.
The Challenge
Optimization and hardening work has a structural problem: nothing is broken. The platform is live, users are served, and every item on the list is an improvement to something that currently functions. That makes it perpetually deferrable, and in most organizations it is deferred until either the bill or an auditor forces the issue.
The risks of leaving it are real but quiet, which is exactly why they lose to louder work.
- Cost grows silently until somebody notices the bill, by which point the cause is months old and spread across several services
- Performance problems get masked rather than fixed, because a larger instance is the fast answer to a slow query
- Incidents arrive from users first, when alerting is not tuned to how the platform actually behaves
- Security debt carries over, because reproducing the previous model inherits its gaps along with its architecture
- There is no audit trail, which is exposure for change management and incident investigation at the same time
What Was at Stake
A migration is the one moment when changing a security model costs nothing extra, and that window closes. Left alone, an environment sized for cutover safety keeps charging cutover prices indefinitely, monitoring tuned for survival keeps reporting problems after customers do, and infrastructure changes without a recorded plan and apply trail stay unevidenced for exactly as long as nobody asks. None of those produce an incident on a particular day, which is why they are the work that does not get done.
Our Role
Kainskep owned the post-migration optimization and hardening phase across four concurrent workstreams, each with its own definition of done: cost, performance, observability, and security and governance. The engagement covered the live environment across development, UAT and production, and included establishing the evidence trail that makes the work reviewable rather than merely reported.
The Approach
One rule did most of the work, and it is the opposite of the profitable answer for any supplier whose commercial model is tied to infrastructure spend.
Measure before tuning, in a fixed sequence
When the application is slow, look at the container layer first and establish whether the bottleneck is application CPU or memory. If it is not, move to database performance insights, examine load and latency, identify the specific SQL statements responsible, optimize the query, and measure again. Only workloads that genuinely need more capacity are given more capacity.
Treat elasticity as a separate problem from efficiency
Autoscaling and moving short-lived, event-driven work onto Lambda sit around the measurement sequence rather than inside it. They handle elasticity, which should not be used to disguise a query that is doing too much work.
Remove classes of exposure rather than manage populations of them
No bastion host, with administrative access over VPN into a private network. Team-based permission sets through IAM Identity Center rather than individually assigned permissions, and device compliance extending the boundary to the endpoint, so access depends both on who the user is and on whether their device is managed.
Configure environments differently on purpose
Development, UAT and production differ in compute model, log retention, monitoring depth and evidence requirements. Treating them identically is simpler, and it is how estates drift into paying production prices for a development environment nobody uses at night.
The Solution
Four workstreams, run concurrently.
Cost
ECR and S3 lifecycle policies retiring stale images and ageing objects automatically. ECS on EC2 with Spot capacity for interruption-tolerant workloads. Reserved Instance pricing for the always-on production database. Threshold-based budget alerts, and a daily automated cost report broken down by account, service and environment.
Performance
Performance Insights driving the slow query investigation sequence. Custom RDS parameters tuned to the application’s actual workload rather than left at defaults. ECS autoscaling tied to CloudWatch alarms, replacing fixed task counts. Short-lived, event-driven work moved off always-on tasks and onto Lambda.
Observability
Custom CloudWatch dashboards bringing ECS, load balancer, RDS and infrastructure metrics into one operational view rather than four. Alerting tuned to separate normal traffic growth from genuine instability: repeated task restarts, rising 5xx rates, sudden database CPU spikes and climbing connection counts raise alarms rather than sitting in a log.
Security and governance
Security Hub scoring the environment continuously against published benchmarks, Inspector for vulnerability assessment and Detective for investigation. WAF rules written for the application’s own traffic patterns rather than a default configuration, with security response headers validated against real production responses including error responses. Terraform plan and apply evidence captured for every infrastructure change.
What We Delivered
Cost visibility and controls
Daily automated reporting by account, service and environment, threshold budget alerts, lifecycle policies and a capacity model matched to workload rather than to cutover safety.
Measurement-driven performance fixes
Query-level changes traced to identified bottlenecks, tuned database parameters, and traffic-responsive autoscaling in place of fixed capacity.
A unified operational view
Dashboards across compute, load balancer and database, with alerting tuned to distinguish growth from instability, and log retention scaled per environment.
A private-by-default access model
No bastion host, VPN-only administrative access, team-based permission sets, and device compliance extending the boundary to the endpoint.
Continuous posture assessment
The environment scored continuously against a published benchmark rather than assessed once, with vulnerability assessment and investigation tooling alongside it.
An audit-ready change trail
Terraform plan and apply evidence for every infrastructure change, usable for change management and incident review from the same record.
Outcomes
Cost is reported daily and budgeted against by account, service and environment rather than surfacing as a monthly total after the fact. Performance fixes reach root cause instead of masking it behind larger infrastructure. Detection moved ahead of users, with tuned proactive alerting replacing troubleshooting triggered by customer reports. Security posture became measurable, assessed continuously against a published benchmark and tracked over time rather than established once. The change trail became audit-ready, and administrative exposure was removed rather than hardened.
The through line is that each of these replaced a judgement with an observation. Spend, performance, stability and posture all moved from things the team believed to things the team could show.
No savings figure is published. The cost work produced one, and it is not printed here because a percentage without its baseline, its period and its workload mix is not evidence of anything.
- Spend visible daily rather than monthly
- Every performance change traced to an identified bottleneck
- Alerting ahead of customer reports
- Posture scored continuously against a published benchmark
What This Page Does Not Cover
Deliberately excluded, in the interest of accuracy rather than completeness: savings figures from the cost work; a finding-by-finding remediation log; WAF rule syntax, database parameter values or autoscaling thresholds; audit dates or results; and any decommissioning timeline for legacy access paths.
The posture score is also not printed. It is assessed against a published benchmark and it moves when the environment changes, which makes it useful in a conversation where it can be explained and misleading as a headline number.
The client, the region and the product services are withheld at the client’s request.
Related Capabilities
The disciplines this engagement drew on, and where each is described in full.
Cloud Infrastructure
Capacity models matched to workload, lifecycle policy design and reserved pricing, tied to visible daily cost reporting.
Cloud infrastructure engineeringDevOps & Automation
Observability platform design, alerting tuned to real behavior, and an infrastructure change trail that supports incident review.
Delivery and operations engineeringCloud Security & Compliance
Private-by-default architecture, continuous posture scoring, identity and access governance, and evidence capture for audit.
Cloud security and compliance readinessData Engineering & Machine Learning
Measurement-driven database performance work: query and parameter tuning rather than reflexive instance resizing.
Data engineering and machine learning