Assessing Materiality in Water Quality Auditing: Our Top Ten Tips

Top Ten Tips Article Lead Photo

One of the things that always needs to be established, when assessing conformance against criteria, is the materiality of a finding against that criterion, that is, the impact on the objectives of the criterion or the risk objectives of the utility[1]. Our regulators rely on auditors to help quantify materiality – but we have found that auditors can have differences of opinion on materiality. This difference of opinion can create uncertainty for both the regulator and the auditee as well as creating difficulties in terms of comparing results between audits. Here we look at what materiality means for drinking water quality and provide some of our top tips for reviewing and assessing materiality.

What does ‘material’ actually mean?

Material non-conformances are a significant issue for utilities, often bringing with them substantial board and even political ramifications (where the utility is a public enterprise and non-conformances are reported to the responsible minister). Being involved in many statutory audits between us, we have seen first-hand the to-ing and fro-ing in terms of what constitutes a material non-conformance. There are several instruments (standards and acts) which cover actual and potential material outcomes – it’s therefore important for an auditor to take into account whether something has materialized or, based on the evidence, whether it could materialise, whether it is the proverbial ‘ticking time bomb’, which needs to be addressed. Definitions of materiality include:

  • Instance(s) of non-compliance that are significant, individually or collectively, in the context of the entity’s compliance with compliance requirements, and that might influence relevant decisions of intended users or affect the assurance practitioner’s conclusion.” Standard ASAE 3100 (in relation to compliance engagements)[2]
  • The determination of performance materiality is not a simple mechanical calculation and involves the exercise of professional judgement. It is affected by the auditor’s understanding of the entity, updated during the performance of the risk assessment procedures; and the nature and extent of misstatements identified in previous audits and thereby the auditor’s expectations in relation to misstatements in the current period. External Reporting Board (XRB) NZ ISA (NZ) 320 (financial lens: in relation to materiality in planning and performing an audit)

Although some jurisdictions are providing examples of what they consider to be material, as well as the level of materiality, it is the experience of an auditor in assessing materiality within a particular industry, that is key. So, what is material for water quality?

What is ‘material’ for water quality?

It is the auditor’s duty to ensure they have enough evidence to assess materiality. For instance, document control uncertainties may be seen by some auditors as a non-compliance(t), non-material finding, however it is known that lack of document control has exacerbated water quality and other risk endpoint outcomes – including in one case, loss of lives. Therefore, it is important that the auditor not only knows their industry but that they also know the context of the finding, have information from other industries where similar things have occurred (the importance of continuing professional development), understand the non-compliance in the overall context of the system, understand the frequency of the non-compliance, and that they always refer back to the criterion being assessed. Only when armed with all of this information, can auditors make a considered judgement call on the materiality of a finding. 

Our top tips for assessing and considering materiality

In summary, here are our top ten tips for assessing and considering materiality: 

  1. Criterion: the essence of the audit criterion is essential to understanding how material a non-compliance against it, might be. You should make sure you fully understand what is being asked by the audit criterion and understand if you need to seek further guidance from other documents to help satisfy yourself of the outcome of the non-compliance.
  2. Context: context is everything. A document control non-compliance may be material if it is a deficiency noted in an emergency management plan (such as a non-current contact list) or a flow diagram (such as not having reviewed the flow diagram prior to undertaking a risk assessment). However, a small number of typographical errors is unlikely to be material and is more likely to just be noted as an observation.
  3. Consistency: it pays to consider creating an audit outcomes library so you can cross-reference audit outcomes between audits and between similar scenarios. Aiming for consistency in assessing materiality creates more certainty for the auditee, the regulator (if a regulatory audit) and other auditors and helps to create more of a level playing field.
  4. Frequency: the number of non-compliances identified can also be a factor in determining materiality. The more non-compliances identified, the more material the finding, as the frequency speaks to a lack of overall attention to managing risk and therefore, an increased likelihood of not meeting objectives.  
  5. System understanding: due to the complexity of modern utilities, it is important that the auditor reviews all supporting departments and contractors, as well as the water quality team, to identify instances of materiality which may impact on achievement of objectives. Examples may include where a contractor is undertaking works on a utility’s behalf, but has little awareness of their role in protecting drinking water quality or a planning department approving a development in a drinking water catchment without due consideration of the development’s impacts on the source water.
  6. Auditor experience and knowledge: in our experience in auditing water quality management systems, we have found that a profound knowledge and good depth of experience of the subject is required to understand the materiality. A subject matter expert may also be needed on your team to help provide in-depth knowledge, of a particular process or provide a different perspective (including from other jurisdictions where necessary). As noted in NZ ISA (NZ) 320, assessing materiality is not just a mechanical calculation, it requires the auditor to exercise professional judgement.
  7. Regulator requirements: the operating context of an audit will also dictate how (or even if) materiality should be assessed. For instance, some audits are more about hard pass/fail rather than an assessment of materiality. However, our view is that the materiality of a finding should be assessed as it provides valuable feedback to the auditee and helps them improve their systems. 
  8. Evidence-base: a serious material audit breach is highly likely to have significant financial, compliance, or reputational outcomes for the auditee. You will need to keep good records to substantiate how you assessed the materiality. We have found these records to be invaluable when later challenged on the decision. The auditee might not like your finding but facts cannot be argued with!
  9. Be firm but fair: while the outcome of the material finding may have significant ramifications and the auditee may pressure you to change the outcome, as noted above, you should stand by your decision, and remember that you have a duty to act professionally, accurately and in an unbiased manner.
  10.  Professional development: it is your responsibility to keep up to date with industry and any other changes, which may have an impact on the way materiality needs to be assessed.

Need help with your next water quality audit? With many years of experience, Dr Annette Davison of Risk Edge and D2K Information and Matt Parkinson of Wai Comply can help you prepare for your next audit, helping you to understand and address your material risk gaps, before your next regulator scrutiny. Want to streamline your water quality auditing and improve assessment of audit results? Learn more here.

Assessing Materiality in Water Quality Auditing: Our Top Ten Tips

Annette Davison, Director and Principal Risk Analyst, Risk Edge Pty Ltd; Co-founder and R&D Manager, D2K Information Pty Ltd, Australia

Matt Parkinson, Senior Compliance Advisor, Wai Comply, New Zealand

#waterqualityaudit #riskmanagement #materiality #ISO31000

One of the things that always needs to be established, when assessing conformance against criteria, is the materiality of a finding against that criterion, that is, the impact on the objectives of the criterion or the risk objectives of the utility. Our regulators rely on auditors to help quantify materiality – but we have found that auditors can have differences of opinion on materiality. This difference of opinion can create uncertainty for both the regulator and the auditee as well as creating difficulties in terms of comparing results between audits. Here we look at what materiality means for drinking water quality and provide some of our top tips for reviewing and assessing materiality.

What does ‘material’ actually mean?

Material non-conformances are a significant issue for utilities, often bringing with them substantial board and even political ramifications (where the utility is a public enterprise and non-conformances are reported to the responsible minister). Being involved in many statutory audits between us, we have seen first-hand the to-ing and fro-ing in terms of what constitutes a material non-conformance. There are several instruments (standards and acts) which cover actual and potential material outcomes – it’s therefore important for an auditor to take into account whether something has materialized or, based on the evidence, whether it could materialise, whether it is the proverbial ‘ticking time bomb’, which needs to be addressed. Definitions of materiality include:

  • Instance(s) of non-compliance that are significant, individually or collectively, in the context of the entity’s compliance with compliance requirements, and that might influence relevant decisions of intended users or affect the assurance practitioner’s conclusion.” Standard ASAE 3100 (in relation to compliance engagements)
  • The determination of performance materiality is not a simple mechanical calculation and involves the exercise of professional judgement. It is affected by the auditor’s understanding of the entity, updated during the performance of the risk assessment procedures; and the nature and extent of misstatements identified in previous audits and thereby the auditor’s expectations in relation to misstatements in the current period. External Reporting Board (XRB) NZ ISA (NZ) 320 (financial lens: in relation to materiality in planning and performing an audit)

Although some jurisdictions are providing examples of what they consider to be material, as well as the level of materiality, it is the experience of an auditor in assessing materiality within a particular industry, that is key. So, what is material for water quality?

What is ‘material’ for water quality?

It is the auditor’s duty to ensure they have enough evidence to assess materiality. For instance, document control uncertainties may be seen by some auditors as a non-compliance(t), non-material finding, however it is known that lack of document control has exacerbated water quality and other risk endpoint outcomes – including in one case[3], loss of lives. Therefore, it is important that the auditor not only knows their industry but that they also know the context of the finding, have information from other industries where similar things have occurred (the importance of continuing professional development), understand the non-compliance in the overall context of the system, understand the frequency of the non-compliance, and that they always refer back to the criterion being assessed. Only when armed with all of this information, can auditors make a considered judgement call on the materiality of a finding. 

Our top tips for assessing and considering materiality

In summary, here are our top ten tips for assessing and considering materiality: 

  1. Criterion: the essence of the audit criterion is essential to understanding how material a non-compliance against it, might be. You should make sure you fully understand what is being asked by the audit criterion and understand if you need to seek further guidance from other documents to help satisfy yourself of the outcome of the non-compliance.
  2. Context: context is everything. A document control non-compliance may be material if it is a deficiency noted in an emergency management plan (such as a non-current contact list) or a flow diagram (such as not having reviewed the flow diagram prior to undertaking a risk assessment). However, a small number of typographical errors is unlikely to be material and is more likely to just be noted as an observation.
  3. Consistency: it pays to consider creating an audit outcomes library so you can cross-reference audit outcomes between audits and between similar scenarios. Aiming for consistency in assessing materiality creates more certainty for the auditee, the regulator (if a regulatory audit) and other auditors and helps to create more of a level playing field.
  4. Frequency: the number of non-compliances identified can also be a factor in determining materiality. The more non-compliances identified, the more material the finding, as the frequency speaks to a lack of overall attention to managing risk and therefore, an increased likelihood of not meeting objectives.  
  5. System understanding: due to the complexity of modern utilities, it is important that the auditor reviews all supporting departments and contractors, as well as the water quality team, to identify instances of materiality which may impact on achievement of objectives. Examples may include where a contractor is undertaking works on a utility’s behalf, but has little awareness of their role in protecting drinking water quality or a planning department approving a development in a drinking water catchment without due consideration of the development’s impacts on the source water.
  6. Auditor experience and knowledge: in our experience in auditing water quality management systems, we have found that a profound knowledge and good depth of experience of the subject is required to understand the materiality. A subject matter expert may also be needed on your team to help provide in-depth knowledge, of a particular process or provide a different perspective (including from other jurisdictions where necessary). As noted in NZ ISA (NZ) 320, assessing materiality is not just a mechanical calculation, it requires the auditor to exercise professional judgement.
  7. Regulator requirements: the operating context of an audit will also dictate how (or even if) materiality should be assessed. For instance, some audits are more about hard pass/fail rather than an assessment of materiality. However, our view is that the materiality of a finding should be assessed as it provides valuable feedback to the auditee and helps them improve their systems. 
  8. Evidence-base: a serious material audit breach is highly likely to have significant financial, compliance, or reputational outcomes for the auditee. You will need to keep good records to substantiate how you assessed the materiality. We have found these records to be invaluable when later challenged on the decision. The auditee might not like your finding but facts cannot be argued with!
  9. Be firm but fair: while the outcome of the material finding may have significant ramifications and the auditee may pressure you to change the outcome, as noted above, you should stand by your decision, and remember that you have a duty to act professionally, accurately and in an unbiased manner[4].
  10.  Professional development: it is your responsibility to keep up to date with industry and any other changes, which may have an impact on the way materiality needs to be assessed.

Need help with your next water quality audit? With many years of experience, Dr Annette Davison of Risk Edge and D2K Information and Matt Parkinson of Wai Comply can help you prepare for your next audit, helping you to understand and address your material risk gaps, before your next regulator scrutiny. Want to streamline your water quality auditing and improve assessment of audit results? Learn more here.

Annette Davison, Director and Principal Risk Analyst, Risk Edge Pty Ltd; Co-founder and R&D Manager, D2K Information Pty Ltd, Australia

Matt Parkinson, Senior Compliance Advisor, Wai Comply, New Zealand

waterqualityaudit #riskmanagement #materiality #ISO31000

[1] Davison, A. (2020) The Application of ISO 31000 to Drinking Water Quality Risk Management: A Practical Approach. Published by Risk Edge Pty Ltd, Sydney, Australia. ISBN: 978-0-9875560-0-4.

[2] https://www.auasb.gov.au/admin/file/content102/c3/ASAE_3100_Compliance_Engagements.pdf

[3] U.S. Chemical Safety and Hazard Investigation Board (2016) Investigation Report Volume 3 Report No. 2010-10-I-Os 4/17/2016 Drilling Rig Explosion and Fire at the Macondo Well.

[4] https://exemplarglobal.org/documents/certification-requirements/latest/pcf01-code-of-conduct.pdf