I’d suggest to just set up automatons to fix those things automatically. Lets say 80% CPU for 5 minutes it too high. Ok, add an auto-scale rule at 65% CPU for 3 minutes to add an extra node to the cluster to load balance the CPU load
Sure, if it were a normal service and not a distributed database that requires days to scale. Days. It's not, "add one node" and we're good. It's Add Node - Migrate Data - Add Node - Migrate Data... And in many cases, we have explicit instructions NOT to scale the customer because they won't be able to afford the larger cluster.
Also, would you auto-scale for a 5 minute blip that goes away in that time and doesn't consistently recur? I certainly wouldn't. The customer might not be able to pay for the size we put them on.
Our customers can simultaneously demand that we respond to all alerts AND not to scale their cluster. Who's fuckin' idea this was, I've no clue.
Like it sounds like you’re saying the issues are caused by systems not being robust and lack of automation… If they’re this scared of outages and breaking SLA, they should work on having less outages, or having fall-backs when they occur.
No. That's reading far more into my statement than I hoped. The reliability is indeed there - it's VERY unlikely our managed database goes down to a technology issue in our control. If it does, it's usually an operator error thing. However, if it were down to just operator error alerts and things actually impacting the end users, my job would be a dream!
Automation is somewhat there, but there's a few stakeholders that insist on human validated steps. So, while I have an ansible playbook for most issues, operating that playbook takes hours.
But it could get pretty difficult to get management to do this kinda things from random suggestions from some SRE. I’d probably talk with the team-lead about this, and other people in your team, cause you’re probably not the only one with these issues. And then have a meeting with the entire dev/SRE team and management to point out it’s not sustainable the way it’s going, and with suggestions to improve it
Sure, if it were technical. But this is largely not a technical issue, as you had assumed. The issue is that there is someone, with power, who gets to say that we must follow unreasonable customer requests to the letter. Even if those requests run counter to our sustainability.
They want to be notified of anything that could potentially slow down their system. So any anomaly. The catch being is that they constantly change patterns because they introduce new workloads weekly - which wouldn't be a problem if they could better communicate their forecasts. And that's just one of a few dozen customers - again all with unique cluster configuration and needs.
Yeah, it sucks. The first year was pretty great and we had a fully integrated and unified managed services team where we were getting some great automation done. Then they split the team in half in order to focus on a different flavor of our product (with an entirely new backend) and left folks who were newer (myself included) with maintaining the old product. We were even told that we should be doing minimal maintenance on the thing as the new product would be the new norm. Then once upper management remembered how contracts work, they decided we needed to support 3 new platforms without growing the team. All while onboarding new customers and growing the environment count. We're now in operational overload after some turnover that was backfilled with offshore support that has a very minimal presence.
I have tried championing this, but I don't expect an ableist, masculinity shaming person like you to understand a call for social pointers on how to "manage up".
"Man Up" - good lord, way to be an ass.