Assessing interpretation variability among experts

Hi,

I would like to assess the difference in interpretation among the experts who used Collect Earth to assess tree cover and distribution in two districts in Ethiopia for 2010 and 2015. I want to know how much they agree in assessing percent land use-land cover, percent tree cover, percent of roads with trees, …. To do so, we had 4-5% of the plots duplicated so that they would be assessed by two different people.

I would like to hear from anybody having done this kind of analysis. How did you assess the difference in interpretation? Through the standard variation, error matrix, ...?

Most interestingly, we had twice the same group of experts (in 2017 they collected data for 2010, in 2018 they collected data for 2015). We spent more time the second time around agreeing on image interpretation and consensus building. So, I really want to do the analysis based on these duplicates and then assess whether spending more time on image interpretation made a difference or not (i.e., decrease difference in interpretation).

Many thanks,

Florence