Okay, hello, my name is Aber van Givensch, I'm a side reliability engineer at SoundCloud.
We ask yourself, what is a side reliability engineer exactly?
What?
Elbow?
Okay.
The side reliability engineer is actually just a fancy term for a backend engineer with
a special focus on high availability, monitoring and scaling and automation and stuff like
this.
And the talk I want to give you today is about measuring, measuring for velocity.
And it's a story about one of our main products.
And we improved the performance a lot doing a measure first approach.
And I will dig into the technical details, but I will also talk about organizational
issues, how to find the right team to solve problems like this actually.
But first, some numbers about SoundCloud.
Yeah, first, I have to say what it is actually, we are a social sound sharing platform.
You've seen it before, where everyone can just upload sound, whether it's a music track
or just a recording from birds in the park or something.
And currently, yeah, we are 30 million creators and listeners on this platform and growing
really fast.
And our platform consists of a variety of products and services.
And we have a desktop web page, mobile web page, Android app, iOS app, everything works
through our REST API, which is also used by lots of partners, which integrate with us.
And our Alexa index is currently around 220.
Usually it's, yeah, we're continuously climbing.
Yeah, so, but today I want to focus on a different product and it's about the SoundCloud Player
Widget.
It's just an embed, you can put in almost every internet site.
Interesting.
Okay.
And here you see it in the CNN web page integrated.
We have it also integrated with Facebook, we get integration with Twitter and Tumblr.
And yeah, down there you see one of my favorite podcasts, the Bugle audio newspaper for our
visual world.
You could check it out on SoundCloud.
So but first, what was actually the problem?
Why do I stand here?
So like around a year ago, we had a product out there, which was built by a really cool
HTML5 JS team.
And we have had almost no insights, whether it's performing good, or if it's online or
offline, or yeah, we haven't had any metrics about it.
And actually, our community team pointed us often to some problems, somewhere because
they saw some tweets, or the widget isn't rendering, or it's too slow, stuff like this.
So we sat together and thought, yeah, how should we tackle this problem?
And we thought before we do any enhancements or improvements, we actually should measure
to work against metrics, instead of just guessing, yeah, obvious.
So it was a measure first approach.
If you want to measure first, you need to find the right tool, or first clarify how
you want to measure.
Actually, there are two different approaches to it.
Synthetic monitoring on the left, and real user monitoring on the right side, real user
monitoring, yeah, you know, maybe from everyone knows it from Google Analytics, for example.
Synthetic monitoring is a bit less known.
It uses, for example, PhantomJS, or a scripted browser, or a browser emulation to fetch
a website and render it, and yeah, collect a bunch of metrics about it.
So let's look first at synthetic monitoring.
Really cool thing about it, it's highly repeatable, because the test pad doesn't change, so context
doesn't change.
But one big disadvantage is it certainly doesn't show the real user experience, we're
just emulating something, right?
But this together already makes it really easy to create benchmarks.
On the other hand, real user monitoring shows the user experience, so we actually really
know how fast our widget was rendering for a group of users.
But especially for widgets and other embeds, we have a big problem with real user monitoring,
because as users change their behavior, this can also affect the metrics.
For example, imagine a track embedded in our widget, and if now users start to comment
a lot of it, we have to fetch more metadata with the next trend, next time we render it.
And if you have, I mean, the widget is really a global thing, we use it, it's emitted around
the globe everywhere, almost.
And if you now have some users in Asia that doesn't do commenting, and we have a really
slim track there, and you have in Europe, which is close to our data center, tracks
with lots of data, metadata, and users interacting a lot of it, it's hard to reason whether Europe
is faster or slower than Asia, for example.
Much worse is the impact of the embedding side, actually.
So more or less the same example, if you have a really empty page just embedding a widget
somewhere in Australia, and we have a super packed page in Europe with lots of other embeds,
it's hard to find out which one renders faster and reason about the outcome.
So this all makes benchmarking super difficult, or difficult enough to use synthetic monitoring,
so we decided to use it.
And there are different levels of doing synthetic monitoring.
You could just do HTTP ping, which just measures the round trip.
You can render a full page, including fetching all assets, or you could do workflow transactions,
the Selenium style of testing, where you, for example, render a widget, press play,
and wait until the sound starts.
But as we want to move fast, we want to keep it simple.
And we decided to just do a page load, and that should be enough.
Maybe I'm fast or you have questions, so just interrupt me and ask something, if this is
the case.
So we wanted to just render a page.
For this, we need to have a tool.
You could either write it yourself, but we don't have the time for it.
And there are some companies out there which have products for synthetic monitoring in
place, and we evaluated a lot of them.
And it's actually not funny.
Some of those UIs were so crappy that no one wanted to use it.
I had to email the support, and it went back and forced around 10 emails to find a certain
option in their UI.
Others need native Windows apps to record tests.
So that doesn't work at all.
So we, in the end, decided on Catchpoint, which is not that known, I think.
And it's a really small company.
But we had or we have a direct contact to their CEO, which helped a lot with debugging
and further problems and questions later.
So let's start with benchmarks.
If you want to make a benchmark, you have to define what is fast, actually.
And we didn't want to come up with a random goal we defined, so let's make it 10% faster.
Maybe it's obvious, so it doesn't make sense to just define a random goal.
And there are some benchmark results out there, which are a good orientation, but they don't
help a lot, because at least they're a supercouple to a certain test.
And if you can't repeat the test, the benchmark result is meaningless.
And for a widget, it's always a bit trickier to compare benchmarks at all.
So we thought a bit about this problem.
And we came up with the idea to benchmark a competitor's widget, which is pretty similar
to ours.
And actually, it's a real competitor in a business sense compared to our SoundCloud
widget.
Imagine a SoundCloud, a sound block where you see how well you have a YouTube embed
and a SoundCloud embed.
The embed that wins the render time gets the first attraction by the user and is more likely
to be clicked.
So we created a YouTube benchmark and let it run for some time and looked at the first
results.
And they looked like this.
In Europe, we were pretty much the same, which is super awesome.
In the US, we were slightly worse.
And outside the US, we were substantially worse.
And this is super easy to explain, because this is roughly how our data center structure
looks like.
We just have one in Amsterdam.
And the wider your way, the slower it gets, the simpler it is.
So now we have benchmarks, and we wanted to report it to our engineers.
And yeah, because the UI was reasonable usable of catch point, we could just give access
to the engineers.
So it wasn't a problem.
But we also have a lot of non-engineers now company, managers, boards, the community team.
And now that we had some benchmark results, we wanted to show it to everyone and see how
we can improve it.
And this is a chart, really high-level chart we came up with, the so-called donor chart
where we have market share of a certain area related to the performance.
For example, in Northern America, we have around 40% of the visitors coming from there.
And we are slightly worse.
So now we know that we need to improve, but we had still some problem in our way.
And that was actually the team structure.
It had reasons, there were reasons why we ran in this situation.
We had front-end teams and back-end teams, and often tasks were related, and they actually
split into two different teams.
And that resulted in this.
So they blocked each other.
And it was super easy to explain.
This was our team structure back in the days.
We had a design team and app teams that created or that implemented all the functional requirements.
And we had operations team and infrastructure team, which solved all the infrastructure
or provisioning problems and tasks.
And for performance improvements, it's super crucial to just work together with the rest
of the team.
I mean, I speak from my point of view because I'm a back-end engineer.
And I was in the infrastructure team.
And it was great to just go into the app team, build a cross-functional team, and to work
close with these guys together and share our knowledge to just improve it.
So this is how our team structure looked afterwards.
And this already was a huge improvement.
So yeah, we did some renaming, operators, operations and system engineers, infrastructure
team into side reliability engineers.
But that's just names.
But let's look at metrics.
We now had our tests on our benchmarks running for a reasonable amount of time, a few weeks.
And we saw this.
This is the response time.
And actually, it shows just change if we either change the code base or we change something
in our stack.
But we saw this jumpy line.
And yeah, it was hard to make sense of it.
We dug deeper and looked at this graph.
It shows the number of items that get loaded.
And for some tests down there, there were just four items loaded.
I mean, it could be a CSS or JS or something.
And sometimes 16.
So yeah.
I stared at this graph a long time and debugged it and tried to find out what could the reason
behind it be.
So it turned out that it was a local storage problem of the emulated browser.
Our partner catchpoint, our service, actually didn't know that they had local storage in
place.
And our HTML5 kick ass front end team used it for caching.
So this broke the cache isolation, actually.
So our notes where the test ran before, we had already a few items in place.
And on others, we didn't.
So yeah.
This was a problem.
But it got fixed.
And in the course of debugging this, we also took screenshots and stumbled about another
problem.
And it looked like this, some screenshots.
So in the middle of the rendering time, the emulated browser thought, OK, done.
The timing, let's collect the timing, two seconds or something finished.
And the problem is this, as we build our widget as a JavaScript front end app, we are just
fetching a DOM which loads the behavior and the presentation.
And then starts to query our API and then renders our widget completely.
And on document ready, for example, there wasn't just nothing to see on this page.
To gain back the control, we thought, OK, let's not do a full page load and wait for
all assets to be there.
But just use the metric for document complete, which already shows a lot.
But the lesson learned for us was to have metrics in place is a great thing.
But to have meaningful metrics and be able to make sense of it is just more valuable.
So let's look closer to our performance improvements and the whole problem of making a widget faster.
So as I mentioned earlier, we are often embedded besides other widgets from different companies.
Here you see a music block together.
We are embedded together with a widget of Twitter, of YouTube, and I think a few other things.
And yeah, when do the widgets are, when does the rendering of our widgets actually start?
From an embedding side point of view, it looks like this.
First, the DOM gets fetched, the behavior and the presentation gets loaded.
And when the page fires document ready, the embed starts.
So the only one that starts this poll position from the poll position is actually the embedding
side.
And all embeds are already at a huge disadvantage.
And what makes things worse, they are competing for connections and resources and CPU time
and so on.
So it's super crucial that embed renders much faster than usual HTML page.
In an embed, it looks similar.
We have a DOM, we have a behavior and presentation, send me asynchronously load metadata through
our API and render the page.
The thing is we can't influence the outer page.
So we need to focus on our widget and make at least as little connection as possible
and speed up everything as best as we can.
So what can we do?
There are four parts involved.
It's the DNS time, the connect time, the time to first byte and the time to last byte.
To make an overview, here you see all four together with a limiting factor.
For example, DNS, the limiting factor is geography.
The problem actually is latency and a possible solution or mitigation would be any cast,
for example.
Connect time is the same and you want to reuse connections.
And the actual problem for time to first byte as a backend engineer is you or I am
the problem in this case.
So actually this is where most of the backend engineers focus on, to speed up their code,
to do caching and so on.
But there are much more means where you can improve the site performance.
And we actually just started to collect best practices, which you can gather from a lot
of awesome tools and then start to make a to-do list and go through it.
To collect the to-do list, there are great tools out there.
For example, page speed and why slow, we just started to use it for our widget and it instantly
showed what problems we have or what opportunities to speed it up.
So for example, I want to go through all four steps again.
For DNS, the problem or the bottleneck was geography and the bottleneck or problem is
latency.
We just vended it.
So I haven't had much to do with DNS at all, which is a great thing because we have a good
DNS provider.
So it just worked and in all benchmarks, we had pretty good timings.
That was good.
But some of those tools showed that we use other external libraries that are located,
for example, one we had in the US and they had their server in the US.
And if you render them from Europe, you had really bad DNS timings because they didn't
care about their DNS that much.
So one solution to speed up our DNS connection time was to just remove or to improve our
critical pass time, to just remove this library from the critical pass and do it afterwards.
So the next time is geography and latency.
As I mentioned, we want to reuse connections as much as possible.
And why do we want to do this?
I want to stress this point a lot, actually, and distance matters absolutely.
So here you see a TCP handshake.
And it goes three times back and forth and it gets even worse for SSL handshakes.
So let's have a look at the scale we are talking about.
Network traffic has a speed of light.
It has electrons flying through the wire.
And in one nanosecond, we can fly the distance of 30 centimeters shown here.
And this is the scale where we measure time spent and enough functions or in the CPU between
and the CPU round trips and stuff like this.
And when we talk about connections, we talk about 100 milliseconds.
So it's always hard to imagine how they compare to each other.
And actually, to say 100 milliseconds are a great connection time, it's a bit bad, actually,
because if you look at what 100 milliseconds is in distance, it's this.
You can go from one to New York, but the other way around the globe.
So actually, in 100 milliseconds, we should be able to initiate a connection already,
right?
And if you just can halve it and save 50 milliseconds, you can actually measure this in distance.
So this is 15,000 kilometers.
And those 15,000 kilometers are better spent in your local network or in your CPU for sure.
So when we put our widget, so we terminated it directly in the CDN to make all SSL handshakes
and connection times as close as possible to the end user, we saw it immediately in
the graphs.
On the left side, we measured at the same time the widget without the CDN.
On the right side, using a CDN.
Let's look at the first part.
Time to first part.
This is the response time document.
So the question is, those two?
This is a web page response and response time.
So this is more a document complete.
And this is a calculated time when the page is rendered completely, but which has the
error that the scripted browser actually doesn't really know when a JavaScript client-side
app is rendered completely.
So it's an approximation because it's just a really huge amount of tests.
But it's actually better to look at the line on the bottom.
So more questions?
Let's look at the first part.
Time to first part.
It's actually the server and you.
So as a backend engineer, you're absolutely in your area.
One simple thing, which can be done, depends on your project, is to change the build process
of your client-side app to put, for example, shards in the URLs so that they can be cached
as long as possible for years.
And we changed the design that we only have one entry point, which then references all
our assets.
If an asset changes, then we need to fetch it new.
But otherwise, it just lives in the CDN.
If a customer user comes from Sao Paulo, chances are that the asset is already in Sao Paulo.
Another thing is we started to put our API in the CDN to introduce some caching.
For our widget, we have some metadata that doesn't change that often, and we have a REST
API.
For example, comments on a certain track, we can cache for five minutes or so, especially
if a user isn't logged in, so he hasn't the means to change it.
We can just change it, this improved a lot again.
And time to last, but, yeah, it's usually compression.
Compression can mean two different things from application developer point of view.
It maybe means that you use a better compression for JPEG or strip out all comments, all of
your CSS bundle everything together.
For a backend engineer, it means one thing, actually, gzip.
And we looked at those tools, and we found out that our zip is open.
It was a strange interplay with varnish, or nginx varnish and nginx, where the first request
was compressed, and subsequent requests weren't anymore.
And it took me maybe five days or more to find it out and to reproduce it using TCP dump,
everything, read source code of all the tools, but eventually we found it out.
So let's look back at our chart.
It was the situation before, after those changes, we now had this chart again.
So this was already a really cool improvement.
So what's next?
The things we wanted to do as far as further improvements are to introduce a fallback data
center.
Currently, we just have a centralized system, one data center on Amsterdam.
If it's down, we have a problem.
And we start to set up another one and use it as fallback.
Balance-wise, we want to put the whole rest API in the CDN to improve all connections
time, all SSL headchains, and use it not only for caching.
And consistency-wise, there are some interesting approaches out there where you could load
balance between different CDNs.
Sedexis is an example for it.
It's a service which uses DNS and measures different CDNs and tries for a certain area
on the globe to find the fastest one.
Or if a CDN goes down, it can switch over.
If you use any elsewhere provider, where you are limited by them, or do you use...
No, no.
We're not self-hosted.
We use, for our storage of our checks, we use Amazon S3.
And yeah.
I mean, we had outages there, but you can split.
So this actually worked pretty good.
But you know, random time-wise, it doesn't have any influence.
I'll have very quickly regarding the performance of the API in the CDN, what exactly do you
use for that?
Do you need an API request to cache it?
This is actually what I wanted to show a few slides earlier.
This is...
Distance matters.
Yeah.
I mean, if you have a CDN pop in Sao Paulo, and you're from Sao Paulo, and you need the
HDI to CCP connection and the SSL connection, it doesn't make sense to go over the ocean
to Amsterdam.
And yeah, just waste time.
This is the point.
And if you...
Even without any caching, it makes a lot of sense to just use a CDN for it, because CDN
can hold the SSL connection to your data center open, and if a new visitor comes, just
initiates the connection with the local pop and gets this data directly through the open
connection.
Yeah.
Exactly.
Yeah.
So, yeah.
We want to completely terminate API.soundcloud.com in a CDN.
So to wrap it up, our course was, and our learnings were, measure first.
It's always great to implement stuff and program against metrics to see, I mean, especially
that you did something really cool today, because the response time dropped.
Keep it simple.
It's super easy to get lost in the complexity of stuff like this, and team autonomy.
So build teams that can work together that have all requirements and are operable unblocked
from other teams.
But yeah, this is the end of my talk, but yeah, I hope you have questions.
Ask me anything.
How many people are you interested in?
We are currently 180 people.
180?
Yeah, quickly growing.
And all local...
No, no.
We've actually ran out of office space in Berlin, so that now we have three offices there,
which is an unfortunate situation, but there's a new building that gets built where we can
move in.
And we also have the office in Sofia, where our fingerprinting team works and lives.
We have one in London, and we have one in San Francisco.
Yeah.
Just mentioned fingerprinting.
Yeah.
All the music labels want to make sure that you can block tracks, and I mean, it shouldn't
be used for illegal stuff.
And if you had to upload the music track or something, it should be recognized automatically
and taken down.
Are you using existing technologies for that?
We are moving from existing software to our own one, because...
I'm not that deep into this fingerprinting stuff, but there were some problems, some
shortcomings.
Oh, it was external services we used before, and they were pretty expensive, I think.
So it makes sense after you reach a certain amount of traffic to just build your own tool.
Yeah.
When you were talking about compression, is there any consolation in which compression
does not make sense?
Like, in terms of the need to use angle of service on the client, that's often time spent?
Or is it always played because the lines are much slower than the compression?
Yeah, I mean, if you have really small HTTP requests and responses, then maybe it doesn't
make sense, but this is micro-optimization.
So we just want to have compression all the time.
And actually, by moving in the CDN, the CDN can handle this.
Oh, yeah.
Do you not distinguish between situations in which it might be better to block?
No.
This would be already a more complex solution.
And I guess to answer requests differently, depending on their size, would spend a lot
of time already, or some time where you can just decompress it.
Everything is just made for a gzip compression thing to handle it.
What I meant is that you have to compress it like a jpeg, for example.
Sorry.
I was speaking about text.
We don't compress jpegs and all other media formats.
That doesn't make any sense, because the compression rate would be awful.
No, we didn't.
We just saw some benchmarks, for example, for Myanmar, where requests take like 20 seconds
or so.
So there needs to be some big firewall in place, which filters all kinds of stuff.
But yeah, I have no idea.
More questions?
You said you focus on synthetic monitoring for your test.
What's your monitoring strategy right now?
Do you still rely on synthetic monitoring?
Did you add some real user monitoring?
I mean, we always use both.
But as I said, it's really hard to reason about the results of real user monitoring,
especially for a widget.
For our main page, it's much simpler.
But as we don't know where it is embedded and how many widgets are embedded on a certain
page, we can't make sense of it.
Synthetic monitoring has some great properties, so it's highly repeatable.
You can always point it to one widget, which doesn't change.
So now we use Google Analytics, which is too expensive after a certain rate.
Then we introduce AT Internet, and currently we are building our own event logout system,
which runs in Amazon, to finally use our own logging infrastructure.
So, any more questions?
Thanks.
That means do it.