How to build a webhook: guidelines and best practices

If your application generates data that interests your customers (i.e., you’re doing something right), you will get requests for webhooks at some point. But there’s not a ton of standard guidance for how to build them, especially on the security side. 

This post will walk you through the following:

• The basics of how to build a webhook and send it from your app
• How to create webhooks with authentication and security in mind
• How to provide a smooth developer experience to your customers

But before we start, there are a few things you need to know.

How to build a webhook

Webhooks are reverse APIs, so they need non-standard infrastructure. Typically, when your app generates data, customers can access it via an API, authenticate with an API key, and make requests as needed. But with webhooks, your customers want proactive notifications of events happening in your app.

Here’s the key difference: While a traditional API is built to receive and respond to requests, webhooks are designed to actively send out data to other systems whenever internal triggers fire. That requires you to persist information on where you’re supposed to be sending data to and the status of those endpoints.

In practice, the process ends up looking like this:

• A service that makes POST requests to arbitrary endpoints [backend].
• A database to store endpoints (and associated metadata) that your webhook sends data to [backend].
• A form to intake endpoint information from developers who want to subscribe to the webhook [frontend].

As with anything, creating a webhook can get as complicated (a Kafka topic with a webhook consumer) or as simple (Lambda) as you want it to be. More on that later.

Before we dive in, let’s clarify the terminology since things can get confusing:

• Webhook provider: Your app sends out POST requests based on event triggers.
• Webhook consumer: Other developers and their apps that are receiving your webhooks at their endpoints.

Some platforms, like Slack, use "outgoing" and "incoming" to describe this relationship, but the concept remains the same.

Handling authentication: Signing webhooks and encrypting payloads

Authenticating webhooks is slightly trickier than standard APIs because you’re sending data to an endpoint without receiving anything back. There are a lot of ways that can go wrong, like spoofing the endpoint, infiltrating the network, and more. That’s just from your end — the consumer also needs to verify that the data coming into their webhook endpoint (that accepts webhook events) is actually from your app and hasn’t been spoofed/corrupted in transit.

How to handle implementation with authentication

Here’s how to implement webhooks with authentication and user experience in mind:

1. Verify ownership of the endpoint

The first thing you need to do is verify that the developer signing up to subscribe to your webhook actually owns the endpoint they’re giving to you. Standard practice is to send a test event to the endpoint and ask the developer to verify they’ve received it — either by returning a 200 or by including a “challenge” that the endpoint needs to echo back. 

For example, Dropbox verifies webhook endpoints by sending a GET request with a “challenge” param (a random string) encoded in the URL, which your endpoint is required to echo back as a response.

2. Decide on a security approach for webhook payloads

Verify that the endpoint doesn’t solve the whole problem, that endpoints can still be spoofed, that networks are uncertain, etc. There are basically two ways to secure the actual data being transmitted, which we’ll cover next.

How to secure data in transit with authentication

1. Don’t send sensitive data through webhooks

This is the easiest way to avoid problems and is the approach that Dropbox takes. When an event happens in their system (e.g., a document gets updated), they’ll send out a webhook that says something along the lines of “the document with an ID of 1234 has been updated by user 1234.” 

This information is completely useless by itself, so you’ll need to follow up by making requests to the API that translates those IDs into whatever information you need to take action on the webhook’s information. But it also means that if a third party gets ahold of the webhook’s payload, they can’t do anything with it.

2. Sign and protect webhook payloads

The other more labor-intensive way to handle auth is to actually… handle auth. You need to approach this in two ways: authenticating yourself as the sender and authenticating the consumer (endpoint) to which you’re sending data.

Signing your webhook

‍To verify to your webhook consumers that you indeed are who you say you are (and the data you’re sending via your webhook is legit), you can sign your webhook payload with a secret key. It’s easiest to do this symmetrically, but you can also use public/private encryption if you want. 

Stripe signs their webhook payloads with a symmetrical secret key in the request header and gives users access to that key in their dashboard so they can verify the signature at their endpoint. ‍WorkOS also includes a unique signature with every webhook request it sends, which you can verify using the secret you generated when setting up your endpoint.

Protecting your webhook payload

Once you’ve verified yourself with your consuming endpoints, you’ll want to think about how to verify the consuming endpoints themselves and that the sensitive data you’re sending in your webhook payload isn’t susceptible to hacking. There are two ways to approach this:‍

1. Encrypt the entire payload: This is fairly uncommon among major webhook providers (Dropbox, Stripe, Twilio, etc.) and requires some extra work on both your and your consumers’ end, but it ensures pretty tight security.‍
2. Certificate pinning‍: This is the most common way to handle payload security. You can only send data over HTTPS (this should be obvious by now) and require your consumer to provide the specific certificate they’re using. For example, Twilio won’t send webhook data to HTTPS endpoints with self-signed certificates.

You’ve probably realized by now that there’s an impossible tradeoff here between developer experience and security. Sending no useful information in webhooks minimizes security risk but requires much more work for the consumer. 

Including information in the webhook payload makes for a smooth developer experience but is hard to do perfectly securely. That, and the fact that security is part of the developer experience, means you’ll need to weigh the risks and choose what’s best for your application.

Sending events: Error handling, ordering, and duplicates

Your webhook system will not be a perfect message queue, and you shouldn’t try to make it one — even companies like Stripe guarantee almost no integrity around ordering, number of events sent, and other ergonomics that you’d expect as a consumer. The general rule — and expectation from your consuming developers — is that you’ll send events at least once, but that’s about it.‍

Here’s how to manage some of the most common issues.

Error handling and retry logic

When you send your POST requests to the endpoints in your database, some of them will inevitably fail (DNS issues, incorrect routing, etc.). You’ll want to retry to some degree, but not constantly and not forever. Here are some general best practices:

• Use exponential backoff to slowly increase the time between your retries. To avoid laughably large wait periods, set a maximum backoff time (usually once a day) via truncated exponential backoff (this is, coincidentally, how GCP handles their Pub/Sub topics).
• If an endpoint hasn’t been responding for a while, mark it as “broken” in your endpoints database and stop sending requests to it. Once you’ve marked an endpoint as broken, send an email to the developer notifying them that you’ve been unable to reach the endpoint and that they need to fix it.
• Ideally, all of your consuming endpoints should be returning 200s to all of your POST requests. If they’re not, use that to determine whether a request was successful or needs retrying.

Handling event ordering‍ and duplicates

Webhook providers typically do not guarantee that events will make it to consuming endpoints in order. See, for example, Stripe:

Webhook providers typically also do not guarantee how many events they’ll send via webhooks, so consumers will need to make their endpoints idempotent to some degree. See, for example, Dropbox:

There’s an overlap between events being out of order, concurrency, and duplicates (as you can see in the above screenshot). In general, you can spend time improving your webhook system to try and avoid some of these issues, but it’s pretty rare to see in the wild.

Development tips: Public URLs, sample events, and logging

As you start working on building a webhooks system, there are a couple of things you can set up early that will make things smoother (beyond your fancy Vim setup).‍

Here’s what to keep in mind.

Testing with live URLs‍

Sending POST requests locally — especially as you’re debugging auth — won’t work since a local dev server isn’t available on the public internet, making it impossible to receive webhooks from the provider service. 

To work around this, set up a public URL as a test endpoint. You can either set up a simple server or just use something like ngrok to tunnel to your localhost.‍

Building a sample events library‍

It’s worth investing time upfront to build a library of sample events you’d want to send out via webhooks. Otherwise, you’ll get stuck needing to trigger things in your app or worse, via external providers like Okta (if you want to send out a webhook when a user authenticates, etc.).

Setting up a database

As mentioned above, you’ll need some sort of database to store all of the endpoints you’re sending webhooks out to. The schema should look something like:

• Endpoint URL
• UserID / email
• Last event sent time
• Is broken

There’s really no good reason to use anything other than a simple relational database for this to start, as it’s unlikely this table will scale to anything that will give you problems.‍

Implementing logging‍

Log each webhook that gets sent out along with the payload, timestamp, and endpoint for debugging and compliance purposes down the road. ‍

Separating events from webhooks‍

This is more of a high-level architectural note. Still, it’s worth noting that there should be a layer of separation between what’s happening in your business systems ("an event") and the actions that you take based on that event (like sending out a webhook). If something goes wrong with your webhooks, you don’t want that to impact other pieces of your application.

How to create webhooks: Stripe’s approach

Stripe provides webhooks whenever an event happens (customer created, card charged, etc.). You can add your endpoint(s) via the UI (below) or Stripe’s webhooks API (yes, an API for configuring Stripe webhooks).

If you’re a consumer, you can accept all of these webhooks on one endpoint or set up multiple endpoints (one for each event). In the latter case, the API is probably more useful. Here’s what a sample POST request to add a new webhook endpoint looks like (from Stripe’s docs):

As we covered above, Stripe does not guarantee the ordering of events, and events may show up in duplicates as well. Their system expects your endpoint to return a 2xx when the webhook gets sent out. If it doesn’t, they’ll retry in increasingly sparse increments until they eventually mark your endpoint as broken and email you about it.

Predictably, the documentation is excellent, especially these best practices.

Here’s another example of a simple Cloudflare Worker for processing a webhook event sent by the WorkOS API.

Scaling your webhook implementation

We’ve outlined a few of the best practices for implementing your webhook system. Still, as your service gets more popular and more and more users consume your webhooks, you’ll likely need to find ways to scale and deliver more and more events without additional latency.

To handle increased demand:

• Consider using streaming event-based databases like Kafka (well, technically a pub/sub system) or AWS Kinesis with multiple worker processes to send the actual webhook. 
• If you’re on the receiving end and need to scale your webhook ingestion, you can use the same approach you would use for standard web traffic — by using a load balancer or reverse proxy in front of your web servers. 
• If you eventually grow past that, consider a non-webhook-based event streaming solution like AMQP or Tibco.

Next Steps

In the meantime, take a look at the WorkOS docs to see how we implement common payloads and event types to make your app enterprise-ready. While you’re at it, check out the WorkOS Events API. It addresses most of the webhook issues we’ve discussed — events will always arrive in order, and you never have to worry about overloading your servers since you can process as little or as many events as you can.

Explore WorkOS events.