Case Study | Ask Media Group

How Ask Media Group Tackled Skyrocketing Container Costs with Embedded Kubernetes Engineers

The DevOps team at Ask Media Group is responsible for ensuring the company’s technological foundation and infrastructure are running as smoothly and efficiently as possible. This includes managing observability platforms, Kubernetes and cloud environments, and cost and security governance, among other things.

Recently, AMG experienced a 250% increase in its Kubernetes cluster size, which resulted in skyrocketing costs. The company utilizes Amazon Web Services (AWS) Elastic Kubernetes Services (EKS), a managed container service which runs and scales Kubernetes applications.

According to David Smith, Director of Technical Operations, “Our Dev teams were deploying new applications into production weekly, and scaling up applications at a rapid pace. Costs were going through the roof, and we couldn’t quickly identify the best way to manage the cluster growth or figure out how to efficiently mitigate the problem. We needed a solution that would quickly lower costs, provide better observability, and improve the platform management of our Kubernetes environment.”

FINDING THE RIGHT SOLUTION

David began searching for a solution and was referred to SuperOrbital, a Kubernetes engineering and training company. “We looked at several companies, but SuperOrbital really stood out. They quickly understood the problems we faced, had deep business and technological expertise, and a partnership-based approach.”

SuperOrbital embedded two of its engineers to work shoulder to shoulder with AMG’s engineering team. “Multiple engineers allowed us to work in parallel. By working this way, we were able to split the pie and pinpoint issues more quickly,” explained David. “It was extremely efficient, and my team had the added benefit of learning about new tools and techniques at the same time.”

VITAL INSIGHTS INCREASE RESILIENCE AND STABILITY FOR LONG-TERM GROWTH

AMG utilizes Amazon’s EC2 (Elastic Cloud Compute) Spot instances as the node infrastructure for their container platform , which enables the company to reserve compute capacity at a reduced price. EC2 allows for elastic growth and scalability of workloads. However, one of the drawbacks is that the instances can be volatile and subject to interruption if AWS determines the resources are needed elsewhere.

"Within the first two weeks of working with SuperOrbital, we started seeing our costs come down significantly and had an understanding of why the Kubernetes cluster was growing the way it was."

“Our number one problem was that our cluster was growing, and we didn’t know why. We were committed to Spot for a number of reasons but needed help leveraging the infrastructure in a way that was as stable and resilient as possible,” said David.

The project started with a quick, but thorough discovery validation phase. By the second week, SuperOrbital began implementing structural changes, which made it possible for AMG to run more workloads on AWS Spot Instances, as well as adding much needed resilience and stability.

“SuperOrbital walked us through how to use Spot Instances in a structured, programmatic way, which gives us better insights into our infrastructure, dependability, and platform health as a whole,” said David. “Instead of targeting one Spot instance type in an availability zone that could disappear, or be unavailable, they identified four separate instance types to improve availability and create the resiliency and stability we needed.” Having multiple instance types that are similar to each other, created much needed redundancy—if there was no Spot availability for one type, the second, third, or fourth could be utilized.

NEW TOOLS AND SOLUTIONS OFFER GREATER TRANSPARENCY AND FLEXIBILITY

David likened Kubernetes clusters to a cargo ship with containers. They didn’t know if their containers were loosely or tightly packed or fully or partially utilized. Either way, containers still take space on a ship, which means you can’t ship more things.

“We had limited tools to manage our clusters at this level of detail. SuperOrbital helped us implement new tools and open-source solutions native to the AWS environment. This helped us better manage our clusters and see how they’re being run,” David explained. “Where previously our options and visibility were limited, we now have the insights to make informed decisions, and this was a huge benefit to what they provided.”

SuperOrbital enacted configurations and toolings, and built dashboards with metrics to ensure the cluster was healthy. These provide for greater transparency and enhanced flexibility to safely launch new products and services in the future.

INSIGHTS, ACCOUNTABILITY, AND GOVERNANCE RESULT IN REDUCED CONTAINER COSTS

SuperOrbital helped the company reduce costs in two key areas. More resilience and control with Spot Instances reduced volatility and risk, which in turn allowed AMG to get the best rate possible. Secondly, they recommended a third-party solution to help better manage costs in the future.

"...SuperOrbital really stood out. They quickly understood the problems we faced, and they had deep business and technological expertise and a partnership-based approach."

“Within the first two weeks of working with SuperOrbital, we started seeing our costs come down significantly and had an understanding of why the Kubernetes cluster was growing the way it was,” said David. “We learned that our technology team had launched new product initiatives that made the clusters run at a larger size. We decided this was manageable because there was a valid reason for the increase.”

The ability to identify if a cluster has failed in an environment and determine the cost to a company’s infrastructure, is critical. “SuperOrbital helped us achieve our immediate cost reduction objective and pointed us in the right direction to find solutions to help manage our container costs long term,” said David. “They got us on the road to governance. We now know when and where to turn the ship, and have increased accountability and created a feedback loop.”

Infrastructure can be a fairly complex environment, and David was impressed with SuperOrbital’s experience, approach, and results. “They were very articulate, handson, and easily bridged the technical/non-technical gap,” said David. “The pairedprogramming approach with our engineers was a big win. It helped our internal engineers build confidence and accelerated SuperOrbital’s engineers, allowing them to quickly understand our environment. Instead of the traditional solutions provider approach of paying for experts to go catch the proverbial fish, they taught us how to fish, all while delivering the solutions we needed.”

David is excited about the next phase and about taking a more proactive vs. reactive approach. Every six months, SuperOrbital will be providing architectural consulting on best practices and industry insights to ensure everything continues to run smoothly. “I feel that we have a sustained partnership.

Our costs were going through the roof in Q1 and Q2, and by the end of Q2, we put the fire out with minimal impact. SuperOrbital helped us mitigate any negative effects and quickly achieve success, which benefits not only my team but our entire organization.”