The Principle Analysis of Frontend Monitoring SDK

A complete frontend monitoring platform consists of three parts: data collection and reporting, data processing and storage, and data visualization.

This article focuses on the first component - data collection and reporting. Below is an outline of the topics we'll cover:

Since theoretical knowledge alone can be difficult to grasp, I've created a simple monitoring SDK that implements these technical concepts. You can use it to create simple demos and gain a better understanding. Reading this article while experimenting with the SDK will provide the best learning experience.

Collect Performance Data

The Chrome developer team has proposed a series of metrics to monitor page performance:

• FP (First Paint) - Time from when the page starts loading until the first pixel is painted on the screen (essentially the white screen time)
• FCP (First Contentful Paint) - Time from page load start until any part of page content is rendered
• LCP (Largest Contentful Paint) - Time from page load start until the largest text block or image element completes rendering
• CLS (Cumulative Layout Shift) - Cumulative score of all unexpected layout shifts occurring between page load start and when the page's lifecycle state becomes hidden

We can obtain these four performance metrics through PerformanceObserver (they can also be retrieved via performance.getEntriesByName(), but this method doesn't provide real-time notifications when events occur). PerformanceObserver is a performance monitoring interface used to observe performance measurement events.

FP

Implementation code:

const entryHandler = (list) => {        
    for (const entry of list.getEntries()) {
        if (entry.name === 'first-paint') {
            observer.disconnect()
        }
        console.log(entry)
    }
}

const observer = new PerformanceObserver(entryHandler)
// The buffered property indicates whether to observe cached data, 
// allowing observation even if the monitoring code is added after the event occurs
observer.observe({ type: 'paint', buffered: true })

The FP measurement output:

{
    duration: 0,
    entryType: "paint",
    name: "first-paint",
    startTime: 359, // FP time
}

The startTime value represents the paint timing we need.

FCP

FCP (First Contentful Paint) - Time from page load start until any part of page content is rendered. The "content" in this metric refers to text, images (including background images), elements, and non-white elements.

To provide a good user experience, the FCP score should be kept under 1.8 seconds.

The measurement code:

const entryHandler = (list) => {        
    for (const entry of list.getEntries()) {
        if (entry.name === 'first-contentful-paint') {
            observer.disconnect()
        }

        console.log(entry)
    }
}

const observer = new PerformanceObserver(entryHandler)
observer.observe({ type: 'paint', buffered: true })

We can get the value of FCP via the above code:

{
    duration: 0,
    entryType: "paint",
    name: "first-contentful-paint",
    startTime: 459, // fcp 时间
}

The startTime value is the painting time we need.

LCP

LCP (Largest Contentful Paint) - Time from page load start until the largest text block or image element completes rendering. The LCP metric reports the relative render time of the largest visible image or text block in the viewport, measured from when the page first begins loading.

A good LCP score should be kept under 2.5 seconds.

The measurement code:

const entryHandler = (list) => {
    if (observer) {
        observer.disconnect()
    }

    for (const entry of list.getEntries()) {
        console.log(entry)
    }
}

const observer = new PerformanceObserver(entryHandler)
observer.observe({ type: 'largest-contentful-paint', buffered: true })

We can get the value of LCP via the above code:

{
    duration: 0,
    element: p,
    entryType: "largest-contentful-paint",
    id: "",
    loadTime: 0,
    name: "",
    renderTime: 1021.299,
    size: 37932,
    startTime: 1021.299,
    url: "",
}

The startTime value is the painting time we need. And element refers to the element being painted during LCP.

The difference between FCP and LCP is: FCP event occurs when any content is painted, while LCP event occurs when the largest content finishes rendering.

LCP considers these elements:

• elements
• elements inside
• elements (using poster images)
• Elements with background images loaded via the url() function (not using CSS gradients)
• Block-level elements containing text nodes or other inline-level text elements

CLS

CLS (Cumulative Layout Shift) - Cumulative score of all unexpected layout shifts occurring between page load start and when the page's lifecycle state becomes hidden.

The layout shift score is calculated as follows:

layout shift score = impact score × distance score

The impact score measures how unstable elements affect the visible area between two frames. The distance score is calculated by taking the greatest distance any unstable element has moved (either horizontally or vertically) and dividing it by the viewport's largest dimension (width or height, whichever is greater).

CLS is the sum of all layout shift scores.

A layout shift occurs when a DOM element changes position between two rendered frames, as shown below:

In the above diagram, the rectangle moves from the top-left to the right side, counting as one layout shift. In CLS terminology, there's a concept called "session window": one or more individual layout shifts occurring in rapid succession, with less than 1 second between each shift and a maximum window duration of 5 seconds.

For example, in the second session window shown above, there are four layout shifts. Each shift must occur less than 1 second after the previous one, and the time between the first and last shifts must not exceed 5 seconds to qualify as a session window. If these conditions aren't met, it's considered a new session window. This specification comes from extensive experimentation and research by the Chrome team, as detailed in Evolving the CLS metric.

CLS has three calculation methods:

1. Cumulative
2. Average of all session windows
3. Maximum of all session windows

Cumulative

This method adds up all layout shift scores from page load start. However, this approach disadvantages long-lived pages - the longer a page is open, the higher the CLS score becomes.

Average of All Session Windows

This method calculates based on session windows rather than individual layout shifts, taking the average of all session window scores. However, this approach has limitations.

As shown above, if the first session window has a high CLS score and the second has a low score, averaging them masks the actual page behavior. The average doesn't reflect that the page had more shifts early on and fewer later.

Maximum of All Session Windows

This is currently the optimal calculation method, using the highest session window score to reflect the worst-case scenario for layout shifts. For more details, see Evolving the CLS metric.

Below is the implementation code for the third calculation method:

let sessionValue = 0
let sessionEntries = []
const cls = {
    subType: 'layout-shift',
    name: 'layout-shift',
    type: 'performance',
    pageURL: getPageURL(),
    value: 0,
}

const entryHandler = (list) => {
    for (const entry of list.getEntries()) {
        // Only count layout shifts without recent user input.
        if (!entry.hadRecentInput) {
            const firstSessionEntry = sessionEntries[0]
            const lastSessionEntry = sessionEntries[sessionEntries.length - 1]

            // If the entry occurred less than 1 second after the previous entry and
            // less than 5 seconds after the first entry in the session, include the
            // entry in the current session. Otherwise, start a new session.
            if (
                sessionValue
                && entry.startTime - lastSessionEntry.startTime < 1000
                && entry.startTime - firstSessionEntry.startTime < 5000
            ) {
                sessionValue += entry.value
                sessionEntries.push(formatCLSEntry(entry))
            } else {
                sessionValue = entry.value
                sessionEntries = [formatCLSEntry(entry)]
            }

            // If the current session value is larger than the current CLS value,
            // update CLS and the entries contributing to it.
            if (sessionValue > cls.value) {
                cls.value = sessionValue
                cls.entries = sessionEntries
                cls.startTime = performance.now()
                lazyReportCache(deepCopy(cls))
            }
        }
    }
}

const observer = new PerformanceObserver(entryHandler)
observer.observe({ type: 'layout-shift', buffered: true })

A single layout shift measurement contains the following data:

{
  duration: 0,
  entryType: "layout-shift",
  hadRecentInput: false,
  lastInputTime: 0,
  name: "",
  sources: (2) [LayoutShiftAttribution, LayoutShiftAttribution],
  startTime: 1176.199999999255,
  value: 0.000005752046026677329,
}

The value field represents the layout shift score.

DOMContentLoaded and Load Events

The DOMContentLoaded event is triggered when the HTML is fully loaded and parsed, without waiting for CSS, images, and iframes to load.

The load event is triggered when the entire page and all dependent resources such as stylesheets and images have finished loading.

Although these performance metrics are older, they still provide valuable insights into page behavior. Monitoring them remains necessary.

import { lazyReportCache } from '../utils/report'

['load', 'DOMContentLoaded'].forEach(type => onEvent(type))

function onEvent(type) {
    function callback() {
        lazyReportCache({
            type: 'performance',
            subType: type.toLocaleLowerCase(),
            startTime: performance.now(),
        })

        window.removeEventListener(type, callback, true)
    }

    window.addEventListener(type, callback, true)
}

First Screen Rendering Time

In most cases, the first screen rendering time can be obtained through the load event. However, there are exceptions, such as asynchronously loaded images and DOM elements.

In such cases, we cannot obtain the first screen rendering time through the load event. Instead, we need to use MutationObserver to get the first screen rendering time. MutationObserver triggers events when the properties of the DOM elements it's monitoring change.

The process of calculating first screen rendering time:

1. Use MutationObserver to monitor the document object, triggering events whenever DOM element properties change.
2. Check if the DOM element is in the first screen. If it is, call performance.now() in the requestAnimationFrame() callback function to get the current time as its rendering time.
3. Compare the rendering time of the last DOM element with the loading time of all images in the first screen, and use the maximum value as the first screen rendering time.

Monitoring DOM

const next = window.requestAnimationFrame ? requestAnimationFrame : setTimeout
const ignoreDOMList = ['STYLE', 'SCRIPT', 'LINK']

observer = new MutationObserver(mutationList => {
    const entry = {
        children: [],
    }

    for (const mutation of mutationList) {
        if (mutation.addedNodes.length && isInScreen(mutation.target)) {
             // ...
        }
    }

    if (entry.children.length) {
        entries.push(entry)
        next(() => {
            entry.startTime = performance.now()
        })
    }
})

observer.observe(document, {
    childList: true,
    subtree: true,
})

The code above monitors DOM changes while filtering out style, script, and link tags.

Checking if Element is in First Screen

A page may have a lot of content, but users can only see one screen at a time. Therefore, when calculating first screen rendering time, we need to limit the scope to content visible in the current screen.

const viewportWidth = window.innerWidth
const viewportHeight = window.innerHeight

// Check if DOM element is in screen
function isInScreen(dom) {
    const rectInfo = dom.getBoundingClientRect()
    if (
        rectInfo.left >= 0 
        && rectInfo.left < viewportWidth
        && rectInfo.top >= 0
        && rectInfo.top < viewportHeight
    ) {
        return true
    }
}

Using requestAnimationFrame() to Get DOM Rendering Time

When DOM changes trigger the MutationObserver event, it only means the DOM content can be read, not that it has been painted to the screen.

As shown in the image above, when the MutationObserver event is triggered, we can read that document.body already has content, but the left side of the screen hasn't painted anything yet. Therefore, we need to call requestAnimationFrame() to get the current time as the DOM rendering time after the browser has successfully painted.

Comparing with All Image Loading Times in First Screen

function getRenderTime() {
    let startTime = 0
    entries.forEach(entry => {
        if (entry.startTime > startTime) {
            startTime = entry.startTime
        }
    })

    // Need to compare with all image loading times in current page, take the maximum
    // Image request time must be less than startTime, response end time must be greater than startTime
    performance.getEntriesByType('resource').forEach(item => {
        if (
            item.initiatorType === 'img'
            && item.fetchStart < startTime 
            && item.responseEnd > startTime
        ) {
            startTime = item.responseEnd
        }
    })

    return startTime
}

Optimization

The current code still needs optimization, with two main points to consider:

1. When should we report the rendering time?
2. How to handle asynchronously added DOM elements?

For the first point, we must report the rendering time after DOM changes stop, which typically happens after the load event triggers. Therefore, we can report at this point.

For the second point, we can report after the LCP event triggers. Whether DOM elements are loaded synchronously or asynchronously, they need to be painted, so we can monitor the LCP event and only allow reporting after it triggers.

Combining these two approaches, we get the following code:

let isOnLoaded = false
executeAfterLoad(() => {
    isOnLoaded = true
})

let timer
let observer
function checkDOMChange() {
    clearTimeout(timer)
    timer = setTimeout(() => {
        // Calculate first screen rendering time after load and LCP events trigger and DOM tree stops changing
        if (isOnLoaded && isLCPDone()) {
            observer && observer.disconnect()
            lazyReportCache({
                type: 'performance',
                subType: 'first-screen-paint',
                startTime: getRenderTime(),
                pageURL: getPageURL(),
            })

            entries = null
        } else {
            checkDOMChange()
        }
    }, 500)
}

The checkDOMChange() function is called each time the MutationObserver event triggers and needs to be debounced.

API Request Timing

To monitor API request timing, we need to intercept both XMLHttpRequest and fetch requests.

Monitoring XMLHttpRequest

originalProto.open = function newOpen(...args) {
    this.url = args[1]
    this.method = args[0]
    originalOpen.apply(this, args)
}

originalProto.send = function newSend(...args) {
    this.startTime = Date.now()

    const onLoadend = () => {
        this.endTime = Date.now()
        this.duration = this.endTime - this.startTime

        const { status, duration, startTime, endTime, url, method } = this
        const reportData = {
            status,
            duration,
            startTime,
            endTime,
            url,
            method: (method || 'GET').toUpperCase(),
            success: status >= 200 && status < 300,
            subType: 'xhr',
            type: 'performance',
        }

        lazyReportCache(reportData)

        this.removeEventListener('loadend', onLoadend, true)
    }

    this.addEventListener('loadend', onLoadend, true)
    originalSend.apply(this, args)
}