Interpretation of "TF-PRVR: Training-Free Partially Relevant Video Retrieval"

This started as an email report for a group meeting and is archived here for reference.

This paper was published at NeurIPS 2026. Its title is TF-PRVR: Training-Free Partially Relevant Video Retrieval.

The paper is the first to introduce the Training-Free paradigm to PRVR, offering a new direction that requires no model training.

Let us begin with the motivation.

1. Motivation

The motivation is developed in considerable depth, supported by many figures and tables.

Loading image…Image unavailable

The first figure, shown above, presents the overall motivation. Subfigure (a) illustrates the conventional two-branch retrieval framework commonly used in PRVR.

The paper argues that this framework has several problems, mainly in the CLIP Branch.

As shown in subfigure (b), prior work in the CLIP Branch typically divides video frames uniformly into segments and average-pools frame-level features within each segment to obtain segment-level features. The paper identifies the following problems, corresponding to the first two motivations:

1. For long-duration events, uniform segmentation may split a single complete event across different video segments, breaking its temporal integrity and preventing segment-level features from fully preserving the event’s overall semantic information.
2. For short-duration events, uniform segmentation may place such an event in the same segment as other events, mixing different semantics into the segment-level feature and reducing its semantic purity and discriminative ability.

Because event distributions and temporal spans vary across datasets, a fixed uniform strategy struggles to adapt to the temporal characteristics of different datasets. This can make a model overly dependent on the event distribution of a particular dataset and impair its generalization ability.

As shown in subfigure (c), Dataset A → Dataset B means training on Dataset A’s training set and evaluating on Dataset B’s test set. Several state-of-the-art methods, such as MSC-PRVR and GMMFormer, exhibit the problems analyzed above: their cross-dataset generalization is poor and far below that of zero-shot inference with CLIP directly. In contrast, the proposed method, shown by the blue bars, even outperforms CLIP.

Loading image…Image unavailable

Now consider the second figure in the motivation section and Table 1, shown above.

Figure 2 shows the cosine similarity distributions of positive and negative pairs in different datasets. The plot may contain an authoring mistake, since the negative similarities appear higher than the positive ones, but this does not affect the argument. The degree of overlap between the positive and negative distributions varies across datasets. Greater overlap indicates a harder dataset in which positive and negative samples are more difficult to distinguish.

Table 1 compares feature distributions across datasets for several state-of-the-art methods and zero-shot CLIP, quantifying the discrepancy with maximum mean discrepancy (MMD). MMD reflects distribution discrepancy: larger values indicate larger differences.

Table 1 shows that features obtained directly from CLIP-L have very small MMD values between any two datasets.

In contrast, the two state-of-the-art methods have relatively large MMD values between any pair of datasets.

Overall, the figure and table show that feature distributions differ substantially across datasets, reinforcing the motivation above.

Loading image…Image unavailable

The paper then presents three tables. Table 2 reports the number of frames per segment under a fixed sampling strategy, where each video in every dataset is uniformly divided into 32 segments. Frame spans differ across datasets. ActivityNet mainly contains long videos, so each segment has 204.01 frames on average, whereas a segment in TVR has only 50.87 frames. Even within a dataset, frame spans vary considerably. In Charades STA, for example, the shortest segment contains only 16 frames while the longest contains 606.

Table 3 splits each dataset into two subsets containing shorter and longer videos, respectively, and then evaluates several state-of-the-art methods on the full datasets and both subsets. Table 4 reports the total duration of videos and the total duration of ground-truth segments for each dataset.

These two tables show that ActivityNet has relatively long action segments, accounting for 30.8% of the total. Under the existing one-size-fits-all fixed segmentation, long actions in short videos are fragmented, so current methods score higher on long videos. TVR, however, has extremely short target actions, accounting for only 11.9%. When long videos are forcibly split into equal parts, the segments inevitably contain a large amount of irrelevant background, causing severe “semantic dilution.” Consequently, the models perform better on short videos. These tables provide direct evidence that fixed segmentation is too rigid and again support the motivation.

Loading image…Image unavailable

In the final part of the motivation, the paper presents three more plots. The x-axes of subfigures (a) and (b) are both the fusion weight of the two branches; the y-axis of (a) is model performance. The optimal fusion weight generally differs across datasets.

The y-axis of subfigure (b) is the number of text queries that retrieve the correct video. The distribution is highly polarized, indicating that some queries rely strongly on static semantic information in frame-level features while others rely on dynamic semantic information in segment-level features. Few queries need both types of information equally.

The x-axis of subfigure (c) is the normalized temporal distance between the highest-similarity features selected by the two branches for a text query; the y-axis is the number of correctly retrieved texts. The clear long-tailed distribution shows that the two branches often select features at severely misaligned times and align them to different events. This is the third motivation. It exposes the temporal inconsistency caused by independent aggregation in existing frameworks and naturally motivates the unified graph structure proposed later to enforce consistency.

2. Method

Next is the method section. The overall framework is shown below:

Loading image…Image unavailable

Overall, as shown in subfigure (a), frozen CLIP encoders extract text and video features. The video features are then passed to the proposed hierarchical segmentation module in subfigure (b), which produces segment-level features at multiple levels. The text features and segmented features are then fed into the proposed graph-based multi-scale consensus retrieval module in subfigure (c) to obtain the final retrieval score.

The two modules are described in detail below.

First, the hierarchical segmentation module:

Loading image…Image unavailable

As shown above, the method first computes the velocity-change features between adjacent frame-level features according to Equation 1, where $f$ denotes video frame features and $\delta$ denotes velocity-change features; hatted variables are normalized features.

The method first calculates velocity-change features between every pair of adjacent features with Equation 1, and then computes their dissimilarity with Equation 2. Intuitively, a higher dissimilarity indicates a greater likelihood of a boundary between events.

The resulting dissimilarities are arranged in temporal order into a signal sequence $s$.

Using Equation 3, the paper applies a wavelet transform with the db4 wavelet basis and decomposes $s$ into signal sequences at multiple frequency levels. Here, $a_L$ is the low-frequency information and reflects the global trend, while $d_k$ denotes high-frequency information at different levels; smaller $k$ indicates higher frequency and stronger attention to local changes.

Given these multi-scale signals, Equation 4 applies an inverse wavelet transform to each level’s detail coefficients separately to reconstruct a temporal response curve at the corresponding scale. The method then detects local maxima on the curve, after taking the absolute value of the inverse-transform result and determining the threshold from the sequence mean and variance. These points naturally become “inherent segmentation boundaries” at different temporal granularities. This operation eliminates rigid fixed-length segmentation and lets the video’s own semantic content determine where boundaries should fall, completing adaptive hierarchical video segmentation.

Suppose the result at the $k$-th scale divides a video into $M_k$ segments. Average-pooling frame-level features within each segment yields segment-level features.

After obtaining segment-level features, the next component is the graph-based multi-scale consensus retrieval module, shown below:

Loading image…Image unavailable

Specifically, the paper treats the segmented features as graph nodes and constructs a unified multi-scale graph.

Edges fall into two categories. The first connects nodes within the same level (Equation 5); it considers both visual cosine similarity between segment features and temporal proximity through exponential decay. The second connects nodes across levels (Equation 6), mainly linking semantically similar segments from adjacent scales whose temporal spans overlap, measured by IoU. Combining and symmetrically normalizing these relations produces a graph adjacency matrix that captures global spatiotemporal relationships.

With the graph built, the text query enters. The method first computes initial similarities between the text feature and each segment feature. To address the temporal misalignment identified in the motivation, Equation 7 introduces a closed-form relevance propagation mechanism. The operation is elegant because it requires no training: through matrix multiplication alone, the initial similarity scores propagate along the graph between segment nodes. Temporally coherent and semantically related multi-scale segments therefore reinforce one another, explicitly aligning feature consistency across scales. Here, $\beta$ controls how much of a node’s feature is mixed with neighboring node features, and $K_{total}$ is the total number of graph nodes.

Finally, Equation 8 performs moment-aware scoring. The paper constructs a dense temporal grid over the entire video. For each time point on the grid, the system averages the propagated scores of all segments at every level that cover that time point ($\tau$), producing a “consensus score” for that moment. The final retrieval score is the maximum consensus score over the video. This suppresses isolated false positives that happen to score highly at a single scale but lack global consensus, allowing genuinely stable and multi-scale-consistent events to stand out.

3. Experiments

Now for the experiments:

Loading image…Image unavailable

This figure shows retrieval results for state-of-the-art methods on ActivityNet Captions. The proposed method achieves the best overall performance.

Loading image…Image unavailable

This figure presents the main performance table of the proposed method on three datasets. Its results are considerably better than zero-shot inference with CLIP-L.

Loading image…Image unavailable

This figure shows the ablation study. Subtable (a) ablates different backbones.

In subtable (b), HS denotes the hierarchical segmentation module described above, GP denotes the graph construction module, and MAS denotes the consensus scoring module.

Subtable (c) ablates different signals for constructing the dissimilarity sequence. The first row uses fixed uniform segmentation, the second directly uses dissimilarity between adjacent frame-level features, the third uses a self-similarity matrix, and the fourth uses the proposed velocity-direction-change features.

Overall, the paper’s default configuration performs best.

Loading image…Image unavailable

This figure visualizes the results. For text queries of different lengths, the multi-level segmentation can find relatively pure event segments.

4. Conclusion

In summary, the paper proposes the first Training-Free PRVR framework. It uses wavelet transforms to let each video’s semantic content drive adaptive hierarchical segmentation, addressing the semantic dilution caused by rigid equal splitting. It also builds a multi-scale graph structure and a score propagation mechanism to enforce temporal consistency in retrieval results. Despite requiring no training, the framework demonstrates strong cross-dataset generalization.