![]() ![]() In fact, Cereal reduces the AEC up to 65% compared to the best performing active sampling method, which typically produces biased underestimates of NMI. Our results show that Cereal reduces the area under the absolute error curve (AEC) up to 57% compared to uniform sampling. Our experiments across multiple real-world datasets, clustering algorithms, and evaluation metrics show that Cereal accurately and reliably estimates the clustering quality much better than several baselines. In this work, we study an underexplored area of research: estimating the clustering quality with limited annotations. This supervised evaluation step introduces a costly bottleneck which limits the applicability of clustering for exploratory data analysis. On the other hand, supervised evaluation metrics such as normalized mutual information (NMI) and adjusted Rand index (ARI) require a labeled reference clustering. Unsupervised evaluation metrics, such as Silhouette Index, often do not correlate well with downstream performance (von Luxburg et al., 2012). However, evaluating these clusterings can be challenging. Typically, data science and machine learning practitioners can obtain a wide range of output clusterings by varying algorithm parameters such as number of clusters and minimum inter-cluster distance (Mishra et al., 2022). Partition a given dataset into meaningful groups such that similar data points belong to the same cluster. Unsupervised clustering algorithms (Jain et al., 1999) Overall, CEREAL can efficiently evaluate clustering with limited We also extend CEREAL from clusterwise annotations to pairwiseĪnnotations. Our framework is agnostic to the choice of clustering algorithm and evaluation We perform an extensive ablation study to show that Our results show thatĬEREAL reduces the area under the absolute error curve by up to 57 Pipeline on three datasets across vision and language. We run experiments to estimate NMI in an active sampling Finally, we pseudo-label the unlabeled data with the Semi-supervised learning and train the surrogate model with both the labeledĪnd unlabeled data. First, we propose novel NMI-basedĪcquisition functions that account for the distinctive properties of clusteringĪnd uncertainties from a learned surrogate model. To that end, we introduce CEREAL, aĬomprehensive framework for few-sample clustering evaluation that extendsĪctive sampling approaches in three key ways. However, we find that their estimation can be biased and Model, the most informative data points for annotation to estimate theĮvaluation metric. Model evaluation literature to actively sub-sample, with a learned surrogate We adapt existing approaches from the few-sample We focus on the underexplored problem of estimating clustering Normalized mutual information (NMI) requires labeled data that can be expensive Originally spotted by the above box of Lucky Charms Crispy Rice Clusters Cereal evidently comes from a quasi-secret new product alert group run by General Mills-a crunchy cabal I’m putting on my best ninja garb to infiltrate (I only own shades of pastel, so it’s not easy).īy all first perceptions, Lucky Charms Crispy Rice Clusters simply can’t be seen as anything but a direct dig at Kellogg’s bafflingly stingy decision to turn Rice Krispies Treats’ buttery clusters into Frosted Krispies.Evaluating clustering quality with reliable evaluation metrics like Oh, Lucky, you beautiful chameleon of predatory cereal assimilation: you’ve done it again!Įarly last year, the breakfast aisle’s favorite impish Irelander threw all caution to the sugar-swirled wind with Lucky Charms Frosted Flakes, a marbit mashup that may not have tasted amazing, but was nevertheless a flippant play that earned my respect for poking the Kellogg’s tiger-bear.Īnd now, in a move so unprecedented in both shade and punctuality, it seems Lucky Charms wants to remedy the biggest cereal crisis to plague an infinity of earths: the death of Rice Krispies Treats Cereal as we knew it. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |