by Calvin Lakhan, Ph.D., Faculty of Environmental Studies, York University
One of the foremost issues with auditing approaches is the erroneous use of the term statistical significance.
I can’t tell you the number of times I have heard stakeholders from all sectors ask the question, “Was it a statistically significant sample?” A term many of us loosely remember from undergraduate statistics is often seen as the benchmark for establishing the credibility and reliability of results.
Most often, people think the term us used to describe achieving a sufficient number of samples to enable meaningful/credible analysis. However, calculating statistical significance is a mathematical exercise that denotes collecting enough samples such that unexplained variance falls below a confidence interval threshold (i.e. stated alternatively, statistical significance implies that we have collected enough samples such that the sample has a 95% probability of approximating for the actual population as a whole).
While it may not seem particularly important to make this distinction, incorrectly using the term statistical significance implies a level of precision/accuracy that is generally not possible through waste auditing.
The number of samples required to achieve true statistical significance can number in the thousands, which is neither feasible, nor practical for a municipality/stakeholder. I remember once telling a stakeholder that the number of audits they could afford (which was in the dozens), resulted in a confidence level of less than 30% given the size and distribution of their city. The disappointment was palpable.
It is absolutely critical that any auditing methodology stress the limitations of what can and cannot be done with respect to sampling, and provide guidance regarding how to interpret/analyze the data. The goal of an audit should never be precision – rather, a sound methodological approach would be premised on “what’s the best I can do with the resources that I have?”
On a number of occasions, the university has been asked to develop “Calculators” that will automatically generate an answer to help guide audit sampling. While this is technically possible, the only guidance that an automatic calculator can provide is distributing samples based on relative population weighting, subject to a series of constraints inputted by the user of the tool. It is a “brute force” tool, that fails to consider the multitude of factors that contribute to both why and how waste audits should be conducted.
A more meaningful approach to developing an audit plan is to include “qualitative” factors. The first question a municipality/stakeholder has to ask themselves is: What is the goal of my audit?
Depending on a municipalities needs, you may want to forego mathematical distribution, and rely on judgement that specifically informs the questions that you have (or alternatively, using a combination of both)
The University has prepared a short document outlining some the key considerations when developing a waste audit strategy, particularly for municipalities/stakeholders with limited budgets.
Given the considerable costs and time associated with conducting audits, it is of paramount importance that a methodologically sound approach be developed. So much of what we do in the waste sector (from policy planning, to waste projections) relies on this data.
About the Author
Calvin LAKHAN, Ph.D, is currently co-investigator of the “Waste Wiki” project at York University (with Dr. Mark Winfield), a research project devoted to advancing understanding of waste management research and policy in Canada. He holds a Ph.D from the University of Waterloo/Wilfrid Laurier University joint Geography program, and degrees in economics (BA) and environmental economics (MEs) from York University. His research interests and expertise center around evaluating the efficacy of municipal recycling initiatives and identifying determinants of consumer recycling behavior. Calvin has worked as both a policy planner for the MOECC and as a consultant on projects for Stewardship Ontario, Multi Material Stewardship Manitoba, and Ontario Electronic Stewardship. Calvin currently sits on the editorial board for Advances in Recycling and Waste Management, and as a reviewer for Waste Management, Resources Conservation and Recycling and Journal of Environmental Management .
