[Strong Opinion]
As a principal engineer, you are the product manager.
[/Strong Opinion]
I don’t mean that you replace the PM. But think about what a good PM does. Customers ask for features, and the PM listens to enough of them to find the product sitting underneath. Inside the company, nobody does that for the problems engineers have. Except you.
Engineers ask for things all the time. “Can you create an SQS queue for me?” “How do I set up alerts for my service?” “How do I know who changed this row?” Each one is a real problem, and each one can be solved on its own, in a ticket, in an afternoon.
But if you listen to enough of them, you start hearing the problem between the problems. The thing those requests have in common that none of the people asking can see, because they’re each looking at their own.
That’s where the products are. The ones I’m going to talk about here, nobody asked for. People only ever asked for pieces of them.
Listening for it
This is why, when I work with a PM, I ask them to “just give me the problem”.
A loosely defined problem is plenty. “Agents can’t tell who changed a policy” is great. What I don’t want is the solution already decided, because a solution hides the problem underneath it. If I get a ticket that says “add an updated_by column to the policies table”, I can do that in an afternoon. I’ll also never connect it to the three other teams who are about to ask for the same thing in a different shape.
I wrote about a version of this in When PRDs Accidentally Invent Architecture, where a simple “send an email when something fails” turned out to be a reporting layer nobody planned to build.
It’s not a new idea for me either. Back in 2024 we started a data initiative where teams went out to “hunt for problems” with their business partners, and I wrote the guide for running those conversations. The section I cared about most was this one:
We are engineers. We are problem solvers. That said, I would highly encourage you to avoid solutioning during the meeting. Just listen to the problems and try to find the root causes of them. […] You will also find a lot more problems and you will be able to identify patterns more efficiently.
Patterns is the important word there. You can’t see a pattern in one conversation. You have to talk to a lot of people (that same guide had a section literally titled “Do I Really Have To Talk To A Bunch Of People?”, and the answer was yes) and you have to resist fixing each thing the moment you hear it.
The problem between the problems
A few weeks ago a team needed to know who changed a record. Pretty normal requirement, something breaks in production and the first question anyone asks is “who did this?”
The Slack thread went where these threads usually go. Lots of smart people with good answers: make the data model immutable, use a ledger, add a history table, use Postgres triggers to fill the history table so nobody has to remember to write to it. Someone posted a screenshot of a field history table from another system (field, old_value, new_value, created_by_id).
All of that is correct. And all of it answers the problem that was asked.
I jumped in with this:
There are multiple things here. History is immutable and history is kept. One does not have to dictate the other. The other is the logging of users on actions.
Not the most eloquent thing I’ve written, but there were two problems in that thread.
The first one is how you keep the history of a record. That’s domain specific. Some teams need full event sourcing, some don’t need anything, and the team should be the one deciding.
The second one is who did it. That one wasn’t really this team’s problem. Every service that writes to Postgres needs it. Every team that doesn’t have it is going to end up asking in Slack sooner or later, and every team that builds it on their own builds it slightly differently. Different column names, different coverage, and no way to follow the user from the BFF they logged into all the way to the service that actually wrote the row.
Nobody asked for a provenance library. They asked how to build a history table, for one service. The library is what was sitting between that question and the next five versions of it.
Safari Provenance
The thread was on October 21st. The first PR for @hippo/safari-provenance went in on October 26th.
Adopting it in a service looks like this:
@Auditable
@Table({ underscored: true })
export class Portfolio extends Model {
@Column(DataType.STRING)
declare name: string;
}
@Module({
imports: [AuditModule.forRoot({ actorId: 'portfolio-lab-service' })],
})
export class AppModule {}
Plus a migration with a single addAuditColumns(queryInterface, 'portfolios') call in it.
After that, every create and update stamps created_by_actor, created_by_initiator and created_by_request_id (and the same for updated_by). Actor is the service that triggered the write, initiator is the human behind the request, and the request ID gets you to the exact call that caused it.
The actorId in forRoot is only a fallback. When the call comes through a BFF, the BFF’s name travels down in the request context and wins. That’s why the diagram below shows portfolio-lab-bff as the actor. The service’s own name only shows up for direct calls, like a cron or a CLI script.
User -> BFF -> Service -> DB
| |
| +-- created_by_actor = "portfolio-lab-bff"
| +-- created_by_initiator = "user@example.com"
| +-- created_by_request_id = "req-abc-123"
|
+-- puts actor + initiator in baggage
None of this needed new infrastructure. safari-context already carries baggage across every service boundary in x-context-* headers, nginx already sets the caller, and the auth guards already put the user on the request. The pieces were all there, nobody had connected them at the database layer yet. It takes about five minutes to adopt, and it’s running in two BFFs and two services right now.
I also didn’t build a history table library, or anything close to event sourcing. Teams that need that still make that call themselves. They just don’t have to figure out “who” as part of it anymore. Deciding what you’re not going to build is a big part of the product work. Maybe the biggest part.
The first version didn’t have request_id. We added it three days later because the first people using the library were debugging something and that’s what they actually needed. You ship it and then you keep listening.
The service directory
Provenance is the most recent one. The bigger one is the service-directory.
I built the first version of it at Globality in 2018, and made the first commit on the Hippo version in June 2021. Today it’s the source of truth for every product we have at Hippo. Every service, worker, runner, bucket and library is a JSON entry in that repo, around 400 of them. Close to 3,000 commits and over 200 contributors.
I’ve written about how it works in The Hippo PaaS and about why it lasted in Constraints Over Flexibility, so I won’t go through all of it again.
Nobody asked for this one either. What people asked for was “can you create an SQS queue for me?”, or “how do I set up alerts for my service?”, or my favorite, “who owns this thing that’s paging me at 2am?”.
On their own those are three unrelated tickets. One is about infrastructure, one is about monitoring, one is about ownership. You can close them one at a time for years, and I’ve seen teams do exactly that.
Put them next to each other and they’re the same problem. There was no single place that knew what a service is, what it needs, and who it belongs to. Every one of those tickets was somebody hand-assembling a piece of that answer.
So that’s what we built. You declare what you’re building and the platform generates the rest. Even the 2am question got answered: it’s a responsible_teams field, and the alerts route off of that same field.
Back to 2018
Since I mentioned Globality, here’s another one from the same time.
In 2018 I was there, running DevOps for around 80 microservices on CircleCI. Every one of them had its own CI config file sitting in its own repo. The request I kept getting was “I need CI configuration for my service”, or the version I heard even more, “where is the latest configuration?”
Nobody asked for a system that generates CI configurations. They asked for a file. Copy the newest one you can find, change the name, done.
But the problem between those requests was obvious once you’d heard it enough times. Nobody knew which config was the latest because there wasn’t one. There were 80 of them, slowly drifting apart, and every time we wanted to add something to the build process somebody had to edit 80 files in 80 repos.
So we built globality-build. Each project got a small build.json that said what type of project it was, and the library generated the CircleCI config from templates. When the templates changed, we patched every repo through the GitHub API. When we upgraded Node, it took two minutes to have every build on the new version.
I wrote about it at the time in Generating CI Configurations at Scale. Reading it now, it’s the same shape as the service directory. A small declaration of what you’re building, and generate everything else. Eight years later I’m still building that same shape, just for different problems.
What I’d tell someone
Look, there’s nothing magic here.
Don’t solve the first version of a problem you hear. Write it down, and wait to hear it again from someone else. The second time, it’s not a question anymore, it’s a missing product. The third time, you’re late.
When someone hands you a solution, work backwards until you find the problem they were trying to solve. Their solution is useful information, but it hides exactly the thing you need to see.
Build on what you already have. Provenance took five days because safari-context was already there, and the header propagation, the nginx caller, the auth guards were all things teams already trusted in production. Nobody had to adopt a new transport or a new deployment. If your thing needs new infrastructure before anyone can use it, or a design review to adopt it, people will roll their own. I would too.
I’ll be upfront about one thing. This works at Hippo because the platform was already there to build on, and because over the years I’ve earned enough trust that when I pick something up, nobody asks whose team it belongs to. If you don’t have that yet, that’s the first product. Build the small thing, make it boring and reliable, and the next one gets easier to start.
There’s a failure mode on the other side of this, and I’ve seen it plenty. The principal engineer who sees a platform in every Slack thread. Builds a framework for a problem two teams had once, ships it, and moves on to the next one. Now there’s a library nobody maintains and three teams are pinned to a version from last year. That’s not product work. A product has users, and if you can’t name them before you start, or you’re not going to be around to answer their questions after, don’t build it. And even when it is a real product, ask whether you’re the one who should own it right now. Every library I pick up is time I’m not spending on something else.
Nobody is going to write the ticket for these. The people asking only see their own problem, and that’s fine, that’s their job. Seeing what’s between them is yours.