How does website monitoring work?

Website owners of that are considering purchase of website monitoring services often ask how such services work. Below we explain what exactly website monitoring is, the types of monitoring and how it is performed depending on what exactly it monitors.

Definition of website monitoring

Website monitoring is the process of testing and verifying that end-users can interact with a website or web application as expected. Website monitoring is often used by businesses to ensure website uptime, performance, and functionality is as expected. [Source: Wikipedia]

Website monitoring functions

Availability and proper functioning of websites are monitored by means of various monitoring functions. The most important among them are:

• availability monitoring — detecting total website failures,
• content monitoring — ensuring that the website displays what it is supposed to,
• page speed monitoring — loading time tracking and detection of slowdowns,
• monitoring of functions and processes — testing the right operation of service functions,
• monitoring other features of websites.

How website monitoring works

Website monitoring is a service provided by an external SaaS (Software as a Service) application, which uses servers located in several different locations. The main reason for using a global network is to exclude any false alarms that may be caused by local network problems, that may concern a single node.

The monitoring application continuously (even every 1 minute) performs specific tests — pretending to be a user to a greater or lesser extent — and if it detects deviations from the standard, it sends out alerts and records the event for later analysis.

Availability monitoring

The basic availability monitoring involves frequent connection to the server on which the monitored website operates and checking the response of this server when trying to open a specific URL.

A website is considered accessible if the server returns the response “200 OK”, potentially preceded by a redirection code (e.g. “301” or “302”). Any other response, as well as the lack of response within a certain period of time (“CONNECTION TIMEOUT”) will cause recording and reporting of a failure — of course after confirming the problem from several locations.

When monitoring availability, only the HTTP header is downloaded, which causes minimal server load and is invisible for traffic tracking packets such as Google Analytics.

Content monitoring

Correct server response confirms that a connection has been established, but it doesn’t mean that the website is loading in the expected form. That is why it is worth monitoring the content of the website as well — and more precisely, to verify the presence of a defined fragment of text or code on the website.

Such monitoring downloads more than just the HTTP header — but it is still not much, because it is just the HTML code of the page, which is a bit of text.

Page speed monitoring

When monitoring availability, the server response time can also be measured. However, the measure that has a real impact on the user experience of the page is the total time of downloading and displaying the page.

In order to correctly measure the time of page downloading, the monitoring application needs to download all elements embedded in the page (styles, images, scripts) and to render the page — i.e. display it by executing scripts. If the time of loading a given page exceeds a defined limit, the monitoring application registers a failure and sends alerts.

This type of monitoring is no longer “invisible” to Google Analytics due to the aforementioned scripting.

Monitoring of functions and processes

The most advanced function of website monitoring is scenario-based transaction testing. It allows to detect faults at various stages of crucial processes — e.g. purchasing.

It consists in regular execution of defined steps (filling in the form field, clicking the button, checking the presence of content, etc.) and at the same time “pretending” to be a real user of a typical browser.
If a problem occurs at any step of the scenario, a screenshot is recorded, and the owner receives an alert.

Such monitoring loads the server and link to the same extent as an actual user — and generated visits are registered by Google Analytics and others (unless views with appropriate filters are used).

Monitoring of other website features

In addition to availability, speed and functionality, other features of the website can (and should) be monitored:

• expiration of the domain name
• validity and expiration of SSL certificate
• presence of a domain on blacklists
• search engine robots’ blocks.

Synthetic versus Real User Monitoring

All of the above described functions consist of the so-called Synthetic Monitoring, which means monitoring a website from the outside. It has many advantages — including the fact that it is completely unaffected by the technology on which the monitored website is based. It doesn’t matter whether it is a Linux or Windows server or whether it was written in PHP or ASP.NET.

The opposite of synthetic monitoring — and, from the point of view of benefits, rather a supplement — is Real User Monitoring. “RUM” involves monitoring actual visits of users and catching irregularities they have encountered. One could say that RUM has more to do with application error tracking packages — such as Sentry — than external monitoring.

Real User Monitoring — also known as “passive monitoring” — as opposed to synthetic monitoring, requires a script (usually JavaScript) to be installed inside the monitored website. This results in another consequence — the load of the monitoring application depends on the traffic in the service. For this reason, RUM prices depend on how many visits the service receives per month.

Other types of monitoring

Website owners use other monitoring services as well, and these are sometimes confused with website monitoring:

• traffic monitoring — the previously mentioned tracking of user behavior on the website;
• monitoring of changes — detecting any alterations in the content of websites, used mainly to observe competition;
• keyword rank monitoring — tracking the position of websites in Google search results for specific queries;
• monitoring of mentions — searching for occurrences of specific keywords (most often brands and product or service terms) online — on websites, blogs, forums and social networks.

How other types of monitoring work

Traffic monitoring

Google Analytics, Yandex Metrica, Matomo — packages analyzing users’ traffic and behavior work similarly to RUM — based on the JavaScript code fragment installed on the monitored website.
Each user’s visit is recorded alongside many features and then made available in the form of reports allowing to draw conclusions based on aggregated data.

Monitoring of changes

Changes are mainly monitored on the competitors’ websites, so it has to be external monitoring. It involves constant comparison of the current version of a given page with the previous one. It is conducted in two ways:

• code/text-based
• visual — based on the comparison of graphic screenshots of a page or a selected part of it.

In each of these methods it is possible to determine the percentage of the change to be reported.

Keyword rank monitoring

Automated observation of the results of SEO activities is more difficult than one might expect. Firstly, Google is blocking repeated requests from the same source in large numbers. This forces the use of special VPNs.
Secondly, the far-reaching personalization of Google search results makes it possible to monitor only certain averages.
Nevertheless, monitoring involves entering the phrase in Google and checking on which position in the results the monitored domain will appear.

Monitoring of mentions

This monitoring — which is undoubtedly very useful in marketing — consumes the most resources and consequently has the highest cost. To capture the records effectively, you have to constantly scan the entire Internet, creating your own giant and updated index, just as Google does.

Summary

Staying up-to-date with a website requires constant monitoring of its various features — both inside the website and in its surroundings. Monitoring services are so developed that they can effectively pretend to be users and detect any irregularities, alerting the website owner before real users spot anything.

Originally published at https://www.supermonitoring.com on September 30, 2020.