- Several published NILM systems have been implemented and tested with a new, real-world dataset

- All algorithms perform poorly when applied to the new data set 

- The findings highlight the unsolved challenges when putting NIML systems to practical applications

Non-intrusive load monitoring (NILM) systems take aggregated consumption data of a household as input and disaggregate the data, providing consumption information and usage statistics on the level of individual appliances. NILM algorithms would be very valuable for energy consultancy and for marketing purposes; thus, many researchers proposed NILM systems over the last years and present test cases in which their systems perform well. Systems that work well in real-world environments, however, are not documented.

We collected time series data from five households over a period of several months, together with ground truth data on actual on-off times and energy use of a large number of appliances. We implemented several published NILM algorithms and evaluated them on this new real-world data set. 

All NILM algorithms performed much worse with the new data set, yielding recognition rates far below the reported values. Many researchers seem to heavily optimize their algorithms for their specific data. This would explain the large gap in performance between lab settings and field implementations. The results highlight the difficulties of developing NILM systems that are suitable for a mass rollout.