TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

jiggawatts

I just tested their online demo with a challenging photo of a snowboarder in dark clothing in front of a dark forest. The low contrast makes it difficult to distinguish their black helmet against the shadowed trees immediately behind and around it. Dinov3 segmented this perfectly, as good as a human might, TIPSv2 cut the head off and marked it with the same PCA values as the forest. Similarly, TIPSv2 "split" the snow in the foreground into two different PCA values despite it being visually (and physically) contiguous and not significantly distinct.