How Orb Drives Billing Transparency with Sentry

Headquartered in San Francisco, Orb is a usage-based billing platform built for complex pricing models. Orb provides highly scalable data infrastructure for real-time usage tracking, a flexible billing engine for any pricing model, and a native invoicing experience with transparency at the forefront.

Orb’s platform is mission-critical for revenue, so accuracy and performance both play a vital role in its success. Lack of real-time metrics and intelligent alerting could easily cause delays in identifying and resolving issues, directly impacting Orb’s customers and their users. Additionally, as a high-performance startup, the team needed a solution that stripped away as much manual work for developers, so they can get back to building an industry-leading service.

A Customer-First Billing Experience

Orb is innovating on billing with real-time visibility into consumption, helping teams move from a manual and error-prone process to a richer, more transparent experience. Integration with Orb provides companies a real-time view into their customers’ usage across all the metrics that they charge on, which are then embedded into their own dashboards (much like Sentry’s own usage dashboards that show month to date consumption!).

One of the most significant benefits of Sentry at Orb is its ability to effortlessly integrate with any part of the stack. Orb uses Sentry’s automatic instrumentation to capture errors and uncaught exceptions across a range of different environments, including Flask, Elastic Container Service, AWS Lambda, and Next.js. With exception handling seamlessly available anywhere that business logic can run, Orb’s engineering team can focus on domain-specific problems rather than boilerplate and instrumentation.

Identifying and Prioritizing Issues

Orb’s team is focused on maintaining customer trust; in an area like billing, even the smallest errors involving payment orchestration or financial reporting can be costly. At any fast growing company, managing risk is paramount; rather than avoiding 100% of issues and grinding velocity to a halt, engineers at Orb look to Sentry to quickly identify and prioritize the most critical ones.

With Sentry’s release management toolset, Orb can quickly identify issues specific to new releases, and understand exactly when regressions are introduced and resolved. Sentry’s automatic instrumentation includes rich metadata, allowing engineers and the current oncall to quickly understand the priority of an issue:

• Environment-based urgency: Each issue in Sentry is tagged with an environment. Issues in the production environment lead to direct pages to the oncall engineer with an SLA, whereas issues in development environments are routed to an asynchronous queue.
• Services and transaction tags help determine surface area: All issues and warnings are automatically tagged with the relevant Sentry service or transaction name, so engineers can quickly identify which service might be in trouble. For example, it’s much more important to fix issues affecting the external API and potentially causing 5XX errors than those affecting a retryable analytics job.
• Using tenant tagging to identify patterns: Each business’ billing complexity is a reflection of their unique go-to-market motion. In some cases, this characteristically reflects in their use of Orb as a platform and any issues they may trigger. In Orb’s Sentry instance, nearly every issue in Sentry is tagged with an organization_id and often a meaningful resource_id. This facilitates understanding how issues may be related to one another, rather than treating them in isolation.
• Occurrence and tag visualization to measure impact: Orb makes heavy use of Sentry tags and their count to help determine how cross-cutting an issue is. For instance, an issue with thousands of occurrences but a single asynchronous job_id tells a very different story than one with hundreds of failing jobs.

Intelligent Alert Management

It’s important that engineers are alerted of important issues immediately, while also ensuring that unnecessary noise does not lead to incident fatigue. Orb uses Sentry’s custom alerting framework and tooling integrations to accomplish this in a streamlined way:

• Every new or re-opened issue in Sentry creates a task in Asana, Orb’s project management solution. This integration uses the fields available in the Sentry issue to populate the task’s custom fields, indicating environment, number of instances, stacktrace, and a direct link to the Sentry dashboard.
• For production environments, Sentry’s threshold-based alerts trigger PagerDuty incidents, notifying the current oncall that an issue requires their immediate attention and resolution.

Orb also makes heavy use of Sentry’s capture_message functionality to understand the context in which specific scenarios are hit. For example, a function might capture a non-critical message when an unexpected or unparseable webhook is seen from a vendor. Although it’s most likely not a cause for concern, these warnings allow engineers to form a much better understanding of the instance, including the stacktrace and local variables. Sentry shines at this task, requiring no special setup and building a much more informative picture rather than a single counter or custom metric.

Looking Ahead

At Orb, correctness and platform stability are paramount. That being said, the team is also exploring using Sentry for its performance and profiling capabilities to identify bottlenecks and optimize resource utilization. This continuous improvement allows Orb to provide a seamless billing experience to its customers.

Empowering Customers with Real Time Alerting

Orb allows customers to set up alerts in its platform for specific subscription thresholds, so customer success managers and end-users can receive notifications when spend limits are reached. This proactive approach puts the customers in charge of their spending, fostering a sense of ownership and control.

Delivering on these alerting capabilities is no easy feat, and requires careful analysis of the workload at hand. Although querying over an events dataset may be fast with the appropriate datastore, Orb aims to use Sentry’s profiling solution to dig deeper into potential code hotspots that can be low hanging fruit for optimization.

With Sentry, Orb’s billing platform has been a game-changer in delivering transparent and customer-focused billing experiences. Leveraging real-time metrics, intelligent alerting, and seamless instrumentation, they’ve been able to improve code quality, streamline incident management, and enhance planning capabilities, allowing them to continue to innovate and redefine the way businesses approach billing transparency and control.