Synthetic vs. Field testing

When working with site performance, the first thing you need to learn is the difference between synthetic tests (lab) and Field tests. Both methods have their benefits and drawbacks (especially if used alone), but we strongly recommend that you familiarize yourself with the methods, and that you consider using both methods when testing and monitoring your website.

Synthetic tests

Synthetic testing basically means that an automated, headless browser visits your website. Synthetic tests give you a lot of benefits:

  • You get highly detailed Lighthouse reports, including 4 different scores, a lot of metrics, Lighthouse recommendations, breakdown of CPU usage, network usage, etc.
  • You get a full network request waterfall, showing the details of every single network call.
  • You typically get both a desktop and a mobile test, where the mobile tests use artificial network and CPU throttling in order to simulate a mobile device.
  • You don't need to install anything on your website, and there are no requirements for your traffic level.
  • You can run these tests against both your own websites and your competitors.

There are also a number of drawbacks:

  • The way your website performs in an artificial browser running in a cloud data center is not necessarily the same way your website behaves in real-world scenarios. In reality browsers may have faster or slower connections, faster or slower devices, a warm cache, etc. So optimizing for a perfect Lighthouse score doesn't necessarily correlate with a perfectly fast website.
  • You typically only get 1 daily data point which means that the statistical significance of a change in performance isn't very high.

Field Testing

Understanding how your website actually performs in the wild requires that you get data from real traffic. There are two ways to get this data: Chrome UX Report (CrUX) and installing PageVitals' Field Testing script. Let's go through both.

Chrome UX Report (CrUX)

Chrome browsers around the world send anonymized performance data to Google for users that are logged in to their Google account and have enabled auto-sync on Chrome.

Websites that get a certain minimum amount of traffic from this data source are included in the Chrome UX Report (CrUX) which is freely and publicly available (requires you to have a Google Cloud account and generate an API key).

The data consists of 75th percentile metrics for Largest Contentful Paint (LCP), First Contentful Paint (FCP), Time-to-first-byte (TTFB), First Input Delay (FID), and the currently experimental Interaction to Next Paint (INP). The data also includes density histogram data showing the percentage of traffic that is in the "Good", "Needs improvement" and "Poor" buckets.

75th percentile can be explained like this. 75% of all your audience will experience the given load speed, or faster. It's a more useful metric than the average or median, because you make sure more people get a good experience, and you remove the outliers that skew your data.

In February 2023 Google also launched the CrUX Historical API which makes historical data 6 months back available.

To sum up, CruX data has these benefits:

  • No need to install anything on your website
  • The dataset is the same Google uses as a ranking signal
  • You can check your own and your competitors' data

But unfortunately it has a few drawbacks:

  • The data is an average of the last 28 days, so you won't see a problem before it's too late
  • Your website requires quite a lot of traffic before being included in the report (Google won't disclose exactly how much traffic is needed)

PageVitals Field Testing

As a final option, you can use PageVitals' Field Testing script to get the performance of your website in real-world browsers. Let's look at the benefits:

  • You can get performance data from all traffic, or just a subset of them
  • No traffic level requirements
  • No cookies are used and no personal data is sent (even the IP address is thrown away immediately). No user behavior or navigation paths are tracked. We call it being privacy-fanatic. This makes most privacy concerns disappear, and typically no consent is required (although you'll have to double check with your local law)
  • The script is tiny (around 8KB), non-blocking, async, and uses minimal CPU and network resources. After all, performance is our lifeblood so we're using all the best practices available to minimize the overhead of the script.

And of course there are some drawbacks:

  • Field testing requires you to install a script, either via your tag manager or directly in your HTML (read our Field Testing installation guide)
  • The script, like all scripts, adds an small overhead on your website. In our case we've done everything we can to minimize this overhead.