Welcome back to another Sentry Snack of the Week. As promised from last week,
this week we're joining Adam as he describes a performance improvement that
was made by changing the default behavior of Flagr.
How's it going everyone. My name is Adam and I am one of the Engineering Managers
here at Sentry. And today, I'm goin to tell you a quick story about caching,
Python objects, and JSON blobs, oh my! Like most companies, we use feature flags
to show which features to show to customers. Initially, we used a home grown solution
to do this, but after a few years it started to show its age and we decided to switch
to an open source solution called Flagr.
While implementing this, we started to notice a few different performance regressions.
And since we build a performance monitoring tool, we decided to use it and really
dig into what was going on.
Now, on any given page we may have hundreds or thousands of flags being checked,
And what we do is we take all of these flags, pass them to Flagr, and send in some
extra context around the user and the organization that they are a part of.
Flagr then sends us back a giant JSON blob with all of the details for each
individual flag. Basically, whether or not the user can view that flag. The problem
was that Flagr's deserialization code was turning that entire JSON blob into a
Python object. This deserialization is actually really expensive to do. So when we
removed it and just access the JSON blob itself we noticed a reduction in time to
parse all of these flags by about 500 milliseconds.
The next problem we saw was around caching. Because previously the deserialization
was taking so much time, we didn't actually know that this was a problem until we
fixed that. Calling Flagr now is faster than checking and storing in the cache. And
something that previously took 900 milliseconds was now only taking 200 milliseconds.
Sometimes with performance monitoring, especially at scale, it's less about finding
that one specific performance issue, and more about peeling back the layers until you
uncover a number of different things. Things that you wouldn't expect to be slow,
like caching, Can sometimes bite you in the butt.
And don't forget to like, subscribe and follow us on YouTube.
You don't want to miss anymore of these Sentry Snacks of the Week.
