Session

Model Evaluation under Class Imbalance Shift

Most classification problems in the field of cybersecurity are significantly class-imbalanced to detriment of the class of interest. The standard set of proper evaluation metrics is well-known but the usual assumption is that the test dataset imbalance equals the real-world imbalance. In practice, this assumption is often broken for various reasons. The reported results are then often too optimistic and may lead to wrong conclusions about industrial impact and suitability of proposed techniques. We introduce methods focusing on evaluation under non-constant class imbalance. We show that not only the absolute values of commonly used metrics, but even the order of classifiers in relation to the evaluation metric used is affected by the change of the imbalance rate. Finally, we demonstrate that resampling in order to get a test dataset with class imbalance equal to the one observed in the wild is not necessary, and eventually can lead to significant errors in classifier’s performance estimate. 

Biography

Jan is a machine learning researcher at Cisco, where he leads a team with focus on detecting threats in network telemetry and email data. They deal with very interesting and difficult datasets with many undesirable properties and need to develop innovative solutions for that. He is also a member of Cisco Security ML patent committee. Prior to joining Cisco, Jan has been working at ESET and has many years of software engineering experience. Jan holds a Master’s Degree in computer vision from CTU FEE.

 

Latest news

The positive development of a privacy preserv...

Since the beginning of the year, we have watched COVID-19 cases fluctuate around the world, turning ...

How AI can learn to win the cybersecurity “game”

Cybersecurity is an incredibly serious business. So thinking of it as a game may seem a little odd. ...

Global audience makes first virtual CyberSec&...

The first virtual edition of CyberSec&AI Connected took place on October 8th. Building on 2019...

CyberSec&AI Connected is here!

After a year of planning and preparation, CyberSec&AI Connected has arrived. Today sees delegate...