Leaving Serverless Led to Performance Improvement and a Simplified Architecture

Why's that? Serverless is just the generic name for CGI-like technologies, and CGI is exactly how classical web application were typically deployed historically, until Rails became such a large beast that it was too slow to continue using CGI, and thus running your application as a server to work around that problem in Rails pushed it to become the norm across the industry — at least until serverless became cool again.

Making your application the server is what is more complex with more moving parts. CGI was so much simpler, albeit with the performance tradeoff.

Perhaps certain implementations make things needlessly complex, but it is not clear why you think serverless must fundamentally be that way.

I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics.

0: https://gist.github.com/hellerbarde/2843375

"Serverless was fighting us" vs "We didn't understand serverless tradeoffs" - one is a learning experience, the other is misdirected criticism.

Im genuinely curious because this is not singling out your team or org, this is a very common occurrence among modern engineering teams, and I've often found myself on the losing end of such arguments. So I am all ears to hear at least one such team telling what goes on in their mind when they make terrible architecture decisions and if they learned anything philosophical that would prevent a repeat.

Edit: found it. Cool! https://rove.dev/

Setting up the required roles and permissions was also a nightmare. The deployment round trip time was also awful.

The 2 good experiences I had with AWS was when we had a super smart devops guy who set up the whole docker pipeline on top of actual instances, so we could deploy our docker compose straight to a server in under 1 minute (this wasn't a scaled app), and had everything working.

Lambda is also pretty cool, you can just zip everything up and do a deploy from aws cli without much scripting and pretty straightforward IaC.

https://github.com/1Strategy/fargate-cloudformation-example/...

DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly.

One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL.

EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)

Say you're in city A where you use transit provider 1 and city B where you use transit provider 2. If a user is in city B and their ISP is only connected to transit provider 1, BGP says deliver your traffic to city A, because then traffic doesn't leave transit provider 1 until it hits your network. So for every transit network you use, you really want to connect to it at all your PoPs, and you probably want to connect to as many transit networks as feasible. If you're already doing multihoming at many sites, it's something to consider; if not, it's probably a whole lot of headache.

GeoDNS as others suggested is a good option. Plenty of providers out there, it's not perfect, but it's alright.

Less so for web browsers, but you can also direct users to specific servers. Sample performance for each /24 and /48 and send users to the best server based on the statistics, use IP location as a fallback source of info. Etc. Not great for simple websites, more useful for things with interaction and to reduce the time it takes for tcp slow start (and similar) to reach the available bandwidth.

Azure/AWS/GCP all have solutions for this and does not require you to use their services. There are probably other DNS providers that can do it as well.

Cloudflare can also do this as well but it's probably more expensive than DNS.

They were a much nicer, if overpriced, load balancing alternative to the Cisco Content Switch we were using, though.

Just use Docker, there are plenty of services where deployment is simply - “hand your container to us and we run it”.

Even the most complicated popular ways to deploy Docker are simpler than deploying to a VM and a lot less error prone.

And to add, this rarely indicates anything about the depth and/or breadth of the 'used to' experience.

A lot of the strongest individual contributors I see want to stay in that track and use that experience to make positive and sensible change, while the ones that move into the management tracks don't always have such motivations. There's no gatekeeping intended here, just an observation that the ones that are intrinsically motivated by the detailed technical work naturally build that knowledge base through time spent hands-on in those areas and are best able to make more impactful systemic decisions.

People in senior tech leadership also are not often exposed to the direct results of their decisions too (if they even stay in the company for long enough to see the outcome of longer-term decisions, which itself is rare).

While it's not impossible to find the folk that do have breadth of experience and depth of knowledge but are comfortable and want to be in higher-level decision making places, it's frustratingly rare. And in a lot of cases, the really good ones that speak truth to power end up in situations where 'Their last day was yesterday, we wish them all the best in their future career endeavours.' It's hardly surprising that it's a game that the most capable technical folks just don't want to play, even if they're the ones that should be playing it.

This all could just be anecdata from a dysfunctional org, of course...

It also forces you to address all the non-existent problems first, the ones you just wish you had like all the larger companies that genuinely have to deal with thousands of file upload per second.

And don't forget all the new infrastructure you added to do the job of just receiving the file in your app server and putting it into the place it was going to go anyway but via separate components that all always seem to end up with individual repositories, separate deployment pipelines, and that can't be effectively tested in isolation without going into their target environment.

And all the additional monitoring you need on each of the individual components that were added, particularly on those helpful background workers to make sure they're actually getting triggered (you won't know they're failing if they never got called in the first place due to misconfiguration).

And you're now likely locked into your upload system being directly coupled to your cloud vendor. Oh wait, you used Minio to provide a backend-agnostic intermediate layer? Great, that's another layer that needs managing.

Is a content delivery network better suited to handling concurrent file uploads from millions of concurrent users than your app server? I'd honestly hope so, that's what it's designed for. Was it necessary? I'd like to see the numbers first.

At the end of the day, every system design decision is a trade off and almost always involves some kind of additional complexity for some benefit. It might be worth the cost, but a lot of these system designs don't need this many moving parts to achieve the same results and this only serves to add complexity without solving a direct problem.

If you're actually that company, good for you and genuinely congratulations on the business success. The problem is that companies that don't currently and may never need that are being sold system designs that, while technically more than capable, are over-designed for the problem they're solving.

GP said this is an app from the 2000s.

For S3 you do need to generate a presigned URL, so you would have to add this logic there somewhere instead of "just having a generic HTTP upload endpoint".

Unless the solution is "don't have the problem in the first place" the cloud limitations are just getting in the way here.

For uploads under 50 MB you could also skip the multipart upload and take a naive approach without taking a significant hit.