pixely

With Google's Core Web Vitals starting to impact search results from May 2021 you may now be trying to work out which tools you can use to understand and improve your performance to help you meet the Core Web Vital targets.

A number of different performance monitoring tools have quickly incorporated the Core Web Vital metrics, but you'll find the results vary across the tools depending on how they work especially between tools which run synthetic "lab" tests against your site compared to those which monitor the experiences seen by real users "in the field". See synthetic vs RUM performance monitoring for more details on the difference between the two and why both forms of testing are useful.

This post gives a comparison of a number of popular performance testing and monitoring tools including some of the key limitations of each so that you can make a more informed choice over which tools to use to understand your page speed performance.

Anything which encourages people to pay attention to performance and drives improvements to user experience is a good thing, but it is worth noting that Core Web Vitals does have some limitations. The three Core Web Vital metrics of Largest Contentful Paint, First Input Delay and Cumulative layout shift give you an insight into user experience, but they don't give you the complete picture. Both the tooling and the metrics also skew towards a view of performance looking exclusively at Chrome. It's always worth considering what your performance is like in a wider range of browsers, not all users are on Chrome and their experiences may vary quite a lot. Use Core Web Vitals as a starting point, don't rely on the exclusively.

• Lighthouse
  • Chrome Dev Tools
  • PageSpeed Insights
  • Lighthouse CI
  • Lighthouse CI + Lighthouse Server
• CrUX – Chrome User Experience Reports
  • Search Console
• WebPageTest
• SpeedCurve
• Calibre
• Core Web Vitals Chrome Extension

Lighthouse

Synthetic

Lighthouse is a hugely popular tool for running performance audits against pages on your site. You select the pages you want to test against with Lighthouse usually then simulating fixed network and device conditions, making it a synthetic form of testing.

The best thing about Lighthouse is that it gives you a detailed and very readable report not just with the performance metrics but also with recommendations on how to improve them along with links for more information. It's a great introduction to performance and, because of this, has become very popular.

You can run Lighthouse in a number of different ways including built into Chrome or integrated into a range of other performance tools. It's important to consider the how the way you run Lighthouse impacts the result it will give you.

As a synthetic tool, Lighthouse can't track the First Input Delay metric. Instead, Lighthouse provides the Total Blocking Time metric which is a good synthetic alternative.

Chrome Dev Tools

Built into Chrome you can run Lighthouse against any page by opening up the developer tools and finding the "Lighthouse" tab (View → Developer → Developer Tools → Lighthouse → Generate Report).

This makes it quite straightforward to use, especially during local development. Whilst Lighthouse will simulate fixed network and device conditions it can give quite variable results depending on what else your computer is doing – if you've got other tabs open or Chrome extensions active these can all impact your results.

For the best results with Lighthouse on Chrome Dev Tools I typically use an incognito window and take the average of multiple test runs (minimum of 3).

• 👍 Built into Chrome
• 👍 Straightforward to use
• 👍 Human-friendly results
• 👎 Highly variable results

PageSpeed Insights

PageSpeed Insights lets you request a Lighthouse audit from a webpage but have that audit run on Google's servers, removing some of the variability you can encounter with Lighthouse with Chrome Dev Tools. It'll do so simulating a mid-tier mobile device on a mobile connection.

The limitation of PageSpeed Insights is that it doesn't let you dig further under the hood – what you see in the report is what you get whereas some other tools let you go further with the data.

One of the unique selling points of PageSpeed Insights is that alongside the synthetic data it also mixes in real world data from real users based on Google's Chrome User Experience Report data. This is the same data source Google uses when assessing your Core Web Vitals. For this to work there needs to be enough traffic visiting that page for Google to build up a big enough data set to report with. If there isn't enough data, PageSpeed Insights will also try and use the data for the site as a whole, rather than just that page.

• 👍 Run an audit from a webpage
• 👍 More consistent results
• 👍 Includes real-user data alongside synthetic data
• 👎 Limited depth of synthetic data, can't go beyond the Lighthouse data

Lighthouse CI

If you want a deeper integration you can run also run Lighthouse from the command line, allowing you to automate Lighthouse into your continuous integration pipelines.

This is a great way of catching performance issues early before they impact your users. Depending on your CI setup, you may find you get quite a lot of variability in your test results but it can run the checks multiple times to take the averages.

Automation does require a lot more work to get things setup, and typically requires you deciding upon the URLs you want to monitor as well as setting fixed a budget for the metrics to allow you to alert when you are no longer meeting your targets.

• 👍 Fast feedback on performance changes
• 👍 Automated
• 👎 More complex setup
• 👎 Variable results

Lighthouse CI + Lighthouse Server

You can also take Lighthouse CI to the next level by integrating it with Lighthouse CI Server (LHCI Server). It requires hosting a LHCI Server instance, but allows you to do things like track your performance over time or compare two branches instead of generating one-off "throwaway" audits.

• 👍 Monitor performance over time
• 👍 Compare branches
• 👎 More setup required

CrUX – Chrome User Experience Reports

Real User Monitoring

Chrome User Experience Reports (CrUX) is a report made available by Google based on real user data coming from Chrome users (depending on privacy settings). You can query this data for any public site with enough traffic to make it into the dataset, making it useful for competitor analysis.

This real user data is the same data Google is using when accessing if a site is meeting the Core Web Vital targets, so keeping an eye on these reports is the best way of knowing if you're meeting Core Web Vital targets accordingly to Google.

The granularity of data is fairly limited. The reports only show you the performance for the overall site and not for specific pages, and only covers a fairly small set of metrics making it quite hard to turn this report into actionable changes you can make to your site to improve performance.

It's also important to remember that CrUX reports only show you the performance as seen by a subset of Chrome users – so it may not be representative of the experience of most of your users.

• 👍 Core Web Vitals as seen by Google
• 👍 Free real user data
• 👍 Competitor analysis
• 👎 Very limited data
• 👎 Data isn't available for all sites
• 👎 Only applies to Chrome users

Google Search Console

Google Search Console also includes a Core Web Vitals report which pulls the same CrUX data into the Search Console UI.

The advantage of using Search Console is it allows you to dive a bit deeper into the data. Rather than just giving you the average for the whole domain you can now start diving into problematic groups of pages to find the areas which need the most attention.

• 👍 Linked to your site search data
• 👍 Highlights the groupings of pages used by Google
• 👎 Same data limitations as the full Chrome User Experience Report

WebPageTest

Synthetic

WebPageTest is an incredibly useful tool for performance audits, letting you run detailed synthetic tests using a range of different locations, browsers and even real devices.

The amount of flexibility WebPageTest offers can be overwhelming, but is really powerful. Some of my most used options include running multiple tests, showing first and repeat views, scripting user journeys and allowing you to simulate blocking requests. The combination of running multiple tests and on dedicated test hardware help reduce variance making WebPageTest one of the best choices for benchmarking your performance.

WebPageTest is a bit like an advanced version of PageSpeed Insights. As well as the Lighthouse report you also get plenty of other data allowing you to dig in deeper – from the full waterfall diagram through to filmstrip images and videos.

When it comes to Core Web Vitals, WebPageTest also includes plenty of markers in the reports showing when Core Web Vital metrics happened.

• 👍 Lots of flexible configuration
• 👍 Runs on real devices
• 👍 Well suited to benchmarking performance
• 👍 In-depth data
• 👎 Scripting can be fiddly

SpeedCurve

Synthetic and Real User Monitoring

SpeedCurve let's you combine both synthetic and real user monitoring.

For synthetic, it runs WebPageTest with Lighthouse across a range of devices and locations. The magic that SpeedCurve brings is the ability to run these on a regular schedule (e.g. a couple of times a day) and/or when you deploy your code and then present this data in a series of really powerful dashboards. These dashboards turn your audits from one-off tests to tracking your long-term performance trends showing how your site performance changes over time.

SpeedCurve LUX is their real user monitoring product which lets you understand the performance seen by your users by adding their LUX snippet to your site. This goes further than the Chrome User Experience reports, letting you see more detail and filter down to specific pages to really understand which pages are doing well, and which need a bit more attention.

If you're looking at Cumulative Layout Shift it's worth noting that there are differences between how SpeedCurve and Google track CLS.

Having both synthetic and real user data opens up some interesting additional reporting within SpeedCurve, for example by aggregating your Lighthouse recommendations to show which will have the biggest impact across your synthetic and real user page views. SpeedCurve also has a specific vitals dashboard joining together the synthetic and real user data to give you a comprehensive view on how well you're doing.

Unfortunately this does come at a cost that means not everyone will be able to use it – I don't use it on my personal sites but use it heavily at work.

• 👍 Combines synthetic and real user data
• 👍 Shows performance changes over time
• 👍 Powerful, but intuitive, dashboards
• 👍 Dedicated web vital dashboards
• 👍 Set a performance budget with alerting
• 👎 Cumulative Layout Shift measurement differs to Google's approach
• 👎 Costly

Calibre

Synthetic

Calibre is another paid solution, this time focusing purely on synthetic performance testing showing your performance changes over time and letting you set budgets against key metrics.

Each test runs Lighthouse, allowing Calibre to report on the synthetic Core Web Vital metrics of Largest Contentful Paint, Total Blocking Time and Cumulative Layout Shift. The UI is light, clean and easy to navigate taking just minutes to set up and start collecting data.

There are a number of similarities between Calibre and the synthetic functionality of SpeedCurve. Whilst both cover similar functionality they have focused their efforts in different places. SpeedCurve puts more emphasis on reporting, with a wider range of dashboards available whereas Calibre has more of a focus on automation with a CLI tool, Node.js API and Pull Request reviews on GitHub making it a very developer-friendly option.

• 👍 Shows performance changes over time
• 👍 Automation tooling giving fast feedback on performance changes
• 👍 Set a performance budget with alerting
• 👎 Not as many dashboard options as SpeedCurve
• 👎 Can get costly

Web Vitals Chrome Extension

Synthetic(ish!)

The Web Vitals Chrome Extension gives you a very quick, high-level view of the Core Web Vitals as seen in your browser as you navigate the web. It'll run against all web pages you visit, with a small badge available to show the status which opens up to show the scores for each metric.

This doesn't throttle or control your device or network at all – so it is really representative of just your own device and network and not those of your users but it can give you a nice early alert as your browse your site of slow performing pages which you can then dive into deeper with other, more detailed, tools.

• 👍 Quick, automatic, summary of Core Web Vitals on any page
• 👎 Likely not representative of experience seen by all users

Summary

There's a great range of performance tools available out there which include Core Web Vitals, giving you a nice level of choice when it comes to understanding if you are on track with meeting the Core Web Vital targets.

If possible, I'd recommend using a service like SpeedCurve so that you can track your Core Web Vitals over time and gain that insight from real users. For doing one-off audits, I tend to then use WebPageTest. If you can't pay for a service then I would recommend using Lighthouse CI Server so that you can track metrics over time. It won't be quite as accurate as a paid service, but it remains better than not having any historical data – it's what I use for this site.

Chrome User Experience reports may give only a limited view into your site performance that won't really help you identify how to improve but it does help you understand performance as seen by Google and therefore understand if you are on target for the Core Web Vitals.

Ultimately, every performance tool comes with pros and cons. Start small, and try using Lighthouse to get an initial understanding before trying other tooling.