Falling short

Why product performance doesn’t always match manufacturer claims

In an attempt to broach the often-controversial topic of product underperformance, a new white paper, “Broken Trust: Energy efficiency and the refrigeration industry,” is focusing attention on product performance within the European refrigeration sector. The white paper has revealed research from Eurovent Certification, citing a potential 31% gap between a product’s claimed and expected performance. It’s a worrying statistic, which sews seeds doubt on the validity of stated product performance.

The implications are serious. In a sector where even a small underperformance can have a large negative impact on the long term energy consumption, carbon footprint and running costs of systems, the prospect of a 31% shortfall will fill many with dread. So how do deviations in data occur and how do you know what data to trust?

The margin for error

For those manufacturers who have their products independently tested, and their claims impartially scrutinised through third-party laboratories or certification schemes, the risk of inaccurate data is minimal. However, not all manufacturers go through this process, and many self-assess their own product performance. While this is a long-accepted practice in the refrigeration and HVAC industry, for those who self-assess, there is a potential margin for error. 

1) Testing can vary between manufacturers

Self-assessment testing facilities, test equipment calibration, processes and procedures can differ between manufacturers. Indeed, even for products produced by the same company, but across different manufacturing sites, (sometimes in different countries), there is no guarantee that those products are tested identically. 

Crucially, this also means that test failure criteria and procedures may differ also. Therefore, the range of characteristics tested, and the manner of their testing can vary, as can the representation of test results. For refrigeration and HVAC decision makers, this creates an unlevel playing field, making it infinitely harder to decipher data and compare products.

2) Self-assessment lacks impartiality

Self-assessment means the manufacturer:

• Has full control of the testing process, including all aspects of laboratory tests
• Selects the products to be tested
• Has a vested interest in favourable results.

It is a conflict of interest which can lead to bias, (even if completely unintentional).

3) Performance testing is complex

Depending on product type, there may be a high range of variables impacting on performance. Without a comprehensive testing procedure, gaps can arise in performance data. In addition, different refrigerants need different testing processes and correction factors applied to achieve accurate results. This increases the chance of inaccurate data.

4) Laboratories may not have ISO 17025 compliance

While it is not mandatory for laboratories to be assessed according to ISO 17025, compliance with ISO 17025 is highly beneficial. It gives laboratories international recognition of their competence and ability to produce valid results. Therefore ISO 17025 adds in an extra layer of protection against inaccurate results.

While not an exhaustive list, these aspects can build to make self-assessed products vulnerable to discrepancies between actual and declared performance data. 

The value in independent testing

The use of independent laboratories and certification bodies greatly reduces bias in the testing routine and can address the issues of self-assessment. Third-party laboratories are often at the forefront of product testing, using the latest technologies and test methodologies to calculate actual performance. The best hold ISO 17025 and follow standardised testing practices, so that products from a range of manufacturers are tested equally.

In the report, “Value of the Testing, Inspection and Certification Sector,” by Europe Economics, the key benefits of independent testing and product certification for refrigeration and HVAC professionals are:

• Products and services are safer, true to their advertised claims, and consistently reliable and high quality.
• It is easier to compare products and services, hence reducing search costs. 
• Testing, inspection and certification enhances competition among products and services, which may result in lower prices (for a given quality) and a greater variety of products.

For manufacturers, there are many benefits too including:

• Higher levels of regulatory compliance.
• Enhanced quality control.
• Enhanced trust and confidence in the product or service, leading to higher market demand.
• Enabling market entry and market access.

Accurate performance data allows for informed decision-making, ensuring that refrigeration systems meet the specific needs of the operation, deliver promised efficiency, and maintain the safety and quality of the products they are intended to protect.

Product certification = guaranteed performance

While independent laboratory testing is vital in proving product performance, only product certification offers a full guarantee that products will perform as advertised. For example, products joining a Eurovent Certified Performance programme will undergo independent laboratory tests along with additional levels of evaluation, including:

• Factory audits to ensure production line quality 
• Software checks and / or software certification*
• Validation of all submitted data (including marketing materials) for technical credibility and continuity
• Products are also put under a surveillance process to ensure they continue to comply with declared data. 

The Eurovent certification process keeps evaluation protocols uniform across all products, countries and continents to ensure the certification process is fair and equal for manufacturers. To do this, Eurovent Certification: 

• Tests all products within a programme according to the same criteria
• Expresses results in the same unit of measurement, regardless of the country where the products are manufactured or marketed
• Ensures all products within the programme go through exactly the same certification, auditing and surveillance process
• Ensures products with components manufactured across multiple sites (and countries), all reach Eurovent certification standards.

Crucially impartiality standards are kept high as manufacturers and laboratories have no direct contact, keeping the process clear of any bias. Additionally, products to be tested can be taken from the production line, factory stock or point of sale, ensuring the product tested is exactly the same as those leaving the production line.

*Dependent on programme 

Conclusion

The expected performance of self-assessed products can fall short of manufacturer claims. This is for many reasons, including inherent bias within the testing regime; variations in procedures, methodologies and test equipment; complexities in product testing; and the competency of the testing laboratory.  

To ensure products display accurate performance information, decision makers should choose brands with independently verified data, either via third-party laboratories or certification bodies. For those seeking guaranteed product performance, certified products offer extra protection. Not only have they undergone a comprehensive testing and evaluation process, which includes independent laboratory tests, factory audits and software checks, but certification offers enhanced impartiality as the certification body selects the units to be tested and there is no contact between the manufacturer and laboratory. Also, products remain under Eurovent Certification’s scrutiny, ensuring continued compliance.  

View certified products