Internal Rfcs Saved Us Months of Wasted Work

We usually use this before tickets are even created. In many cases, an RFC can lead to an epic-level ticket that includes multiple user stories.

In other cases, it can be very tactical. I think you are right, probably it can be expressed directly in the form of a ticket. The discussion well happen in the comments to the ticket in this case.

[delayed]

Ticketing systems inevitably get Goodharted. Everyone starts in good faith, then the manager starts using the number of tickets closed, touched, etc as a proxy for work being done, then the agents replace number of tickets with things actually accomplished.

> That said, a ticketing system should ostensibly offer this same effect

If the ticketing system were private to the engineers, it probably would serve the same purpose. But inevitably ticketing systems have wider visibility, and become a mechanism for signalling progress to management/product/sales... at which point engineers are actively incentivised not to put actual data in the ticketing system

>> The most common objection is that writing proposals is “a waste of time” compared to writing code.

> The extra time spent writing is actually spent thinking.

Common theme for decades is "we can save a few days of planning with just a few weeks of programming".

But then there's the darker realization that sometimes the people you are working for are incapable of reasoning about planning artefacts or understanding how the system will look or operate simply from a document. So you need to present the system in small iterative chunks and repeatedly re-align expectations with reality: Agile.

And sometimes you genuinely need to do exploratory work which doesn't fit into a planning framework - actual research!

>> The most common objection is that writing proposals is “a waste of time” compared to writing code.

> The extra time spent writing is actually spent thinking.

Until someone decides that using ChatGPT to write your RFC is a good idea. Then you get something that looks great, but the person behind the prompt actually understands less.

It's not a foregone conclusion. I've had great success writing BDD-style literate tests (not using a framework, just comments that follow a particular grammar). They've allowed verbally negotiating the approximate expected specification, and dealing with the realities of what comes up once you start writing code.

So it's possible to leave it to developers, but expect a high quality spec along with the code.

That's like me trying to get my son to wash his hair and him responding by saying "We have shampoo in the shower."

I am right and my son is right, but his hair is still not washed.

I became a more effective manager and a better father when I learned how to talk to him better.

It's not the lack of communication. IMO, it's the lack of team culture. Keeping documentation up to date is something only the team could do. And it can't be solved by using Confluence/Wiki/mailing lists tools.

In our org, an RFC precedes a tech spec. The RFC literally is the "let's formally talk about this before we nail down a specification". For smaller specs, annotated comments can serve this purpose. Before this process, what we had found was no one was paying attention to tech discussions in our eng slack channels. Having an RFC gave us an inflection point where we could point back and say, "an official discussion happened, we decided to move forward with a spec".

> "The single biggest problem in communication is the illusion that it has taken place."

Love this! Corollary: when you have too many meetings, that’s easy to notice. When you don’t have enough meetings, that’s harder to notice.

I’m in the process of carefully adding meetings and process to our small team of 6 (we had a PM from a large company drop in a few years ago and haphazardly add a bunch of process, and it didn’t really help).

We’re fully remote and have a daily huddle and, on average, 1 hour of meetings a week. It turns out this isn’t enough. So far, each bit of communication we’ve added has resulted in better outcomes and higher morale because we feel more like a team.

In my organization we have RFCs, PRDs, ADRs etc. and I would say that the process is fairly broken. That said, I think what you mention is an important but not the only failure mode of a proposals process.

In some cases I have seen, people use RFCs to steamroll decisions when they are the only stakeholders. Here the waste comes from the fact that the proposal becomes just a bureaucratic step or a way to diffuse responsibility.

In the case you mention (which I have seen many times) I would say the general issue is that the goals and the constraints are not qualified sufficiently. If that's the case, then there are only 2 cases: there is an objective way to measure if an objection or comment makes sense or not, or it is subjective. If it's objective, then consensus is easy to reach, if it's subjective, it needs to be acknowledged and usually the decisions falls on those who are responsible for the outcome (e.g., the team who needs to maintain or build the thing).

Of course, the debate can move to constraints, goals or even ways to measure, but these are generally more straightforward.

> This leads to authors having to re-explain their thinking in detail, covering points that they’d omitted for brevity or because they are obvious to those with a good understanding of the problem.

IMHO there should be no gaps like that. If you already thought hard and long about these problems, why not just write all your knowledge down, so other people can benefit from that? Also, this makes your work a lot more accessible to those who came after you. Of course, you have to assume a certain baseline of knowledge. But if you already have a good understanding of the problem, why not formulate that as part of the RFC?

There is no perfect process. I think after a long time working in software I now understand that it works like this: you need one person to work by themselves really really quickly to ship and create the foundations. After that, they need to remain in place to make sure the design continues to make sense as more people add code to the monolith. Having a specification process is a must in complex environments, and the downside is that it adds friction (and some people can abuse the back and forth to prevent you from landing your design changes), so you need that process to happen as late as possible. But if you're running in production in sensible environments then you will need to make that process happen earlier than desired.

Folks with big titles will always write comments that sound smart and thoughtful but in reality hinder the process. For example:

- This architecture binds us to AWS. Have we estimated the engineering effort to remain cloud-agnostic in case we need to move to Azure next year?

- I see we're using Postgres. Have we considered how we’ll handle horizontal sharding if our user base grows by 1000x in Q4?

- This synchronous API call introduces tight coupling. Shouldn't this be an event-driven architecture to handle back-pressure?

All sound like things that are easy to ask, sound prudent to management, but are impossibly expensive to answer or implement for a feature that just needs to ship.

> This leads to authors having to re-explain their thinking in detail, covering points that they’d omitted for brevity or because they are obvious to those with a good understanding of the problem.

There's nothing wrong with this. Being able to explain your thinking in detail to someone that doesn't necessarily understand the problem is a pretty good exercise to make sure you yourself fully understand the problem _and your thinking._ Of course, this can't turn in to a lecture on basic things people should know or have looked up before commenting.

I feel this tension, but I think something has gone awry if people feel like every comment has to be addressed to everyone’s satisfaction. Comments are not commitments, and commenters don’t have an equal stake in the decision—they certainly don’t own the decision, and it’s okay to disagree and commit.

It’s also probably worth being explicit that there is a cost to inaction that can exceed the costs of building the wrong thing. A year or two ago I was lead on a project where we didn’t know the answer to a big ambiguous problem—we just didn’t have a good way to get the information necessary to make the right decision—different people had different ideas about what we should do. So we identified the smallest thing we could build that would be useful such that we could get more information from real world use, knowing full well we might have to rebuild something else from the ground up. And we did! But we were able to get the confidence we needed to build the right thing later! And we got there much faster than if we had tried to deliberate and speculate about what that thing would be.

Also, in that saga, one engineer in particular was really adamant that we address his particular set of concerns. He was unable to disagree and commit—he was kind of religious about all concerns being addressed before moving forward with anything. He had a very difficult time understanding that the RFC process existed to meet business goals—the business did not exist to slot into his ideal RFC process. He is no longer with the company.

Like many things in dev it sounds sooo good on the surface, but is a minefield in practice (Brandolini's law + The Iron Law of Bureaucracy for starters).

I find it funny that what you said summarize the publishing process in academia very well. Only except that it's much, much worse.

> This leads to authors having to re-explain their thinking in detail

What I like about RFCs (or similar documents) is the ways they work to prevent this. Recently I was involved in planning an initiative without a document like this, and we had to keep explaining and re-explaining the motivation for our decisions to stakeholders and higher-ups. With a document (assuming everyone reads the document before giving feedback), most questions get pre-empted; the ones that don't only need to be addressed once, because the answers end up in the version of the doc which you show to the next person.

Certainly I think it's worth being selective about who you're soliciting comments from, to avoid a too-many-cooks situation, but rare is the project that doesn't need approval from someone. Presenting a big fat document gives a sense for the amount of thought that has gone into the design, which quells the kind of off-the-cuff "why not X?" comment you might get in response to a boxes-and-arrows chart and a high-level summary.

This is my experience as well, the RFCs turn out are being made for internal processes and policy making. And also I have the feeling that people are making it like Request for Approval for technical changes, so we have to take significantly more time on the RFCs than the real work.

You don't really want company wide consensus on every point in the RFC. That should be reserved between the author(s) and their immediate teams. Setting/allowing the expectation that any comment made will be reviewed and responded to until everyone is in agreement about it is going to make things miserable for everyone and improve nothing.

I.e. that culture of pragmatism needs to be defined as part of the process, not an unspoken promise between coworkers.

Yeah I think it all boils down to culture. Tools like RFC (and anything else) can help propel a good culture forward. But you can't fix a broken culture with a tool.

My take is that an RFC should be very early in the engineering process, like as part of a proof of concept phase, and should not block progress towards completing a design proposal. The design proposal should list any legitimate alternatives to overall or component designs discussed during RFC along with the reasoning for not using them in a "designs not chosen" appendix. This at least gives your engineering leadership an opportunity to evaluate the general design ideas before anyone is prepared to die on the hill of those ideas.

Architecture / Design review happens post proof of concept but still before any significant development work and major action items are blockers to beginning development. Further discussion about designs not chosen can happen here, especially when a flaw is uncovered that would be addressed by one of those not chosen.