AWS to Bare Metal Two Years Later: Answering Your Questions About Leaving AWS

The AWS mandatory training I did in the past was 100% marketing of their own solutions, and tests are even designed to make you memorize their entire product line.

The first two levels are not designed for engineers: they're designed for "internal salespeople". Even Product Managers were taking the certification, so they would be able to recommend AWS products to their teams.

I wish I could come up with some kind of formalization of this issue. I think it has something to do with communication explosions across multiple people.

MARS isn't strictly needed for most things. Some features that requires it are ORM (EF) proxies and lazy loading. If you need MARS, there are third party "accelerators" that workaround this madness.

"MARS Acceleration significantly improves the performance of connections that use the Multiple Active Result Sets (MARS) connection option."

https://documentation.nitrosphere.com/resources/release-note...

I'd like to +1 here - it's an understated risk if you've got datacenter-scale workloads. But! You can host a lot of compute on a couple racks nowadays, so IMHO it's a problem only if you're too successful and get complacent. In the datacenter, creative destruction is a must and crucially finance must be made to understand this, or they'll give you budget targets which can only mean ossification.

In a large company I worked the Ops team that had the keys to AWS was taking literal months to push things to the cloud, causing problems with bonuses and promotions. Security measures were not in place so there were cyberattacks. Passwords of critical services lapsed because they were not paying attention.

At some point it got so bad that the entire team was demoted, lost privileges, and contractors had to jump in. The CTO was almost fired.

It took months to recover and even to get to an acceptable state, because nothing was really documented.

Looking back at doing various hiring decisions at various levels of organizations, this is probably the single biggest mistake I've done multiple times, hiring specific people using specific technology because we were specifically using that.

You'll end up with a team unwilling to change, because "you hired me for this, even if it's best for the business with something else, this is what I do".

Once I and the organizations shifted our mindset to hiring people who are more flexible, even if they have expertise in one or two specific technologies, they won't put their head in the sand whenever changes come up, and everything became a lot easier.

In my experience, the ops folks were absolutely thrilled with the arrival of the cloud because with a trivial amount of training and a couple of certifications they had a pathway to get paid as much, if not more, than devs, especially if they rebranded as “devops engineers” instead of “ops guys”.

The only pushback against the cloud, other than some of us engineers who actually were among the first to jump on the cloud, still really loved it, but also recognized that it wasn’t the best fit for all uses and carried significant risks, were people worried about data safety.

The latter concern has largely turned out to not be a real one yet, but a decade and a half later people are finally realizing that actually there are many areas where the cloud may not be the best fit.

If you hire people that are not responsive to your needs, then, sure, that is a problem that will be a problem irrespective of what their pet stack is.

This is key.

Most people never scale to a size where they hit that limit, and in most organisations where that happens, someone else have to deal with it, and so most developers are totally unaware of just how fictional the "infinite scalability" actually is.

Yet it gets touted as a critical advantage.

At the same time, most developers have never ever tried to manage modern server harware, and seem think it is somehwat like managing the hardware they're using at home.

"The right services" is I think doing a lot of work here. Which services specifically are you thinking of?

- S3? sure, 100x, 1000x, whatever, it doesn't care about your scale at all (your bill is another matter).

- Lambdas? On their own sure you can scale arbitrarily, but they don't really do anything unless they're connected to other stuff both upstream and downstream. Can those services manage 100x the load?

- Managed K8s? Managed DBs? EC2 instances? Really anything where you need to think about networking? Nope, you are not scaling this 100x without a LOT of planning and prep work.

Its a lot worse than this in terms of AWS cost for apps that often barely any people use. They're often incorrectly provisioned and the AWS bill ends up in the hundreds of thousands or millions and could have been a few thousand in bare metal on Hetzner with a competent sysadmnin team. No, its not harder to administer bare metal. No, its not less reliable. No, its not substantially harder to scale for most companies to do bare metal(large fortune 50 excluded).

I've never seen a cloud setup where that was true.

For starters: Most cloud providers will impose limits on you that often means going 100x would involve pleading with account managers to have limits lifted and/or scrounding a new, previously untested, combination of instance sizes.

But secondly, you'll tend to run into unknown bottlenecks long before that.

And so, in fact, if that is a thing you actually want to be able to do, you need to actually test it.

But it's also generally not a real problem. I more often come across the opposite: Customers who've gotten hit with a crazy bill because of a problem rather than real use.

But it's also easy enough to set up a hybrid setup that will spin up cloud instances if/when you have a genuine need to be able to scale up faster than you can provision new bare metal instances. You'll typically run an orchestrator and run everything in containers on a bare metal setup too, so typically it only requires having an auto-scaling group scaled down to 0, and warm it up if load nears critical level on your bare metal environment, and then flip a switch in your load balancer to start directing traffic there. It's not a complicated thing to do.

Now, incidentally, your bare metal setup is even cheaper because you can get away with a higher load factor when you can scale into cloud to take spikes.

> And generally speaking if your problem is at a scale where baremetal is trivial to implement, its likely we're only taking about a few hundred dollars a month being 'wasted' in AWS. Which is nothing to most companies, especially when they'd have to consider developer/devops time.

Generally speaking, I only relatively rarely work on systems that cost less than in the tens of thousands per month and up, and what I consistently see with my customers is that the higher the cost, the bigger the bare-metal advantage tends to be as it allows you to readily amortise initial setup costs of more streamlined/advanced setups. The few places where cloud wins on cost is the very smallest systems, typically <$5k/month.

It's perfectly valid to not want to put engineering effort into it at the "wrong time" when delivering features will give you a higher return, but it came across as a lack of interest in paying attention to cost at all.

I saw a lot of that attitude from the tech side when I was looking at this. A lot of the time the CFO or CEO would be appallled, because they were actually paying attention to burn rates, but where often getting stonewalled by the tech side who'd often just insist all the costs were necessary - even while they often didn't know what they were spending or on what.

Because it was mostly fine at first, but later we had some close calls when there were changes that needed to be made on the servers. By the time we managed to mess up our hand managed incremental restart process, we had several layers of cache and so accidentally wiping one didn’t murder our backend, but did throw enough alerts to cause a P2. And because we were doing manual bucketing of caches instead of consistent hashing we hit the OOMKiller a couple times while dialing in.

But at this point it was difficult to move back to managed.

This feels closest to digital ocean’s business model.

It’s really disturbing how the human factor controls decision making in corporations.

For my peace of mind, I chose a sane path - if the company as an entity decides to do AWS, I will do my best to meet its goals. I’ve got all Professional and Specialty certs. It’s the human nature. No purpose in tilting at windmills.