Where there are tests, and where the results have a meaningful impact, people are tempted to change how the item or person is being tested in order to improve their performance. Sometimes, this is dressed up as "optimizing" and at other times it's down to "cheating." Recent news stories have pointed the finger at Volkswagen for installing "defeat" technologies into their cars, that is, a technology designed to detect when it is being tested for emissions and exhaust gases and put itself into an economy and low emission mode. On the road, the car does not detect it is in this mode and behaves differently, using more fuel. In the Android and device world, we have seen device manufacturers design the operating system to recognize when a performance benchmark is being run and to temporarily relax any thermal limits and to keep the system-on-chips running in their maximum performance mode, ensuring that the benchmark result is the best possible – but not reflecting the real world performance of the device. And now the largest Android device manufacturer, Samsung, is under investigation for another of its technologies using defeat technology in order to flatter the power consumption tests: its televisions.
The story here is that, just as with Volkswagen, independent tests have found that in the real world, the performance and in particular, energy management of the Samsung televisions do not mirror official test conditions. The story originates back in February when Samsung televisions sets were tested by the ComplianTV laboratory. The report said: "The laboratories observed different TV behaviours during the measurements and this raised the possibility of the TV's detecting a test procedure and adapting their power consumption accordingly. Such phenomenon was not proven within the ComplianTV tests, but some tested TVs gave the impression that they detected a test situation." The report did not specifically mention Samsung televisions, but lab studies found that Samsung's "motion lighting" feature reduced screen brightness (and power consumption) during international electrotechnical commission (IEC) test conditions, but in real world viewing conditions, the same motion lighting technology did not dim the screens. The IEC test involves playback of fast sequences in recorded television shows, DVDs and live shows.
This has prompted the question to Samsung: have you "optimized" your televisions to flatter the energy consumption tests? The European Commission is now involved and has promised to investigate any allegations of cheating, as well as a pledge to outlaw defeat devices from televisions and other consumer products. From this, it appears that using defeat devices is not against the law (but against the spirit of the law), according to Rudolf Heinz, ComplianTV's product lab project manager.
The technology in question is Samsung's "motion lighting" technology, which is designed to improve viewing performance. The technology dims and brightens the screen depending on the media being shown and has been used in mobile devices for a number of years, as Samsung have used a similar technology with its AMOLED panel handsets since the original Samsung Galaxy S. We've also seen similar technology used in other manufacturers' devices such as the ASUS ZenPad 8.0. The question for the European Commission is to determine if "motion lighting" has additional code designed to put the set into a dimmer brightness setting during consumption testing. Samsung said this on the matter: "There is no comparison [between motion lighting and VW defeat devices]. This is not a setting that only activates during compliance testing. On the contrary, it is an 'out of the box' setting, which reduces power whenever video motion is detected. Not only that, the content used for testing energy consumption has been designed by the international electrotechnical commission to best model actual average picture level internationally."
Around the world, other authorities have recognized that some television sets appear to flatter energy consumption tests. The Swedish Energy Agency's Testlab has come across sets that recognize the IEC standard film used and lower the screen brightness, thus flattering the test results. In the developed world, a television set is reputed to consume "up to 10%" of the electric used in the household and the figure is growing too, despite advances in technology. And now, more testing is necessary to establish is television manufacturers are optimizing their television shows to reduce power consumption during testing. Samsung may be hoping the results don't have a whiff of diesel about them.