Deep metric learning (DML) aims to learn an embedding space where instances from the same class are encouraged to be closer than those from different classes.
A family of DML approaches are known as pair-based, whose objectives can be defined in terms of pair-wise similarities within a mini-batch, such as contrastive loss, triplet loss, lifted-structure loss, n-pairs loss, multi-similarity (MS) loss and etc.
The performance of pair-based methods heavily relies on their capability of mining informative negative pairs. Negative pairs are pairs of data points that belong to different classes.
Negative pairs can be easily increased by increasing the size of mini-batch, however, this solution has a limitation, that the mini-batch size is limited by the GPU memory and computational cost.
This paper proposes a solution for increasing the hard negative without much computational overhead.
Loss calculation in pair based DML
The loss has 2 parts :
1. Sum of similarity in negative pairs
Here Sij is the similarity between the anchor and a negative query
2. Sum of similarity in positive pairs
Here Sij is the similarity between the anchor and a positive query
We have to decrease the similarity of negative pairs and increase it for the positive pairs.
Hence the total loss becomes:
As loss will decrease, the similarity of negative pairs will decrease and that of positive pairs will increase.
From this equation, we can conclude that increasing the negative pairs will give more information to the model and the model will train more efficiently.
The embeddings of past mini-batches are usually considered out-of-date because the model parameters are changing throughout the training process. And hence, old embeddings should be discarded. But this is not always true.
With a certain number of training iterations, the embeddings of instances can drift very slowly,resulting in marginal differences between the features computed at different training iterations.
This phenomenon is known as Slow Drift.
Hence we can say that if slow Drift of embeddings is occurring then some previous embeddings of mini-batches are useful as not much is learned by the model.
Cross Batch Model (XBM)
XBM provides plentiful hard negative pairs by directly connecting each anchor in the current mini-batch with the embeddings from recent mini-batches.
As the feature drift is relatively large at the early epochs, the neural networks are warmed up allowing the model to reach a certain local optimal field where the embeddings become more stable. Then we initialize the memory Batch by computing the features of a set of randomly sampled training images with the warm-up model.
A queue Data structure is used to maintain the Memory Batch. At each iteration, the enqueue of the latest embeddings and dequeue of the earliest embeddings is done. Thus memory batch is updated with embeddings of the current mini-batch directly, without any additional computation.
The Memory batch constructed consists of a good amount of negative pairs and hence helps the network to learn faster.
train network f conventionally with K epochs initialize XBM as queue M for x, y in loader: # x: data, y: labels anchors = f.forward(x) # memory update enqueue(M, (anchors.detach(), y)) dequeue(M) # compare anchors with M sim = torch.matmul(anchors.transpose(), M.feats) loss = pair_based_loss(sim, y, M.labels) loss.backward() optimizer.step()
The model was tested with SOP, In-shop and Vehicle ID Data sets
import torch import tqdm from ret_benchmark.data.build import build_memory_data class XBM: def __init__(self, cfg, model): self.ratio = cfg.MEMORY.RATIO # ratio of size of memory batch vs size of training set # init memory self.feats = list() # stores embeddings (memory batch) self.labels = list() # stores labels self.indices = list() # stores indices corresponding to data point model.train() for images, labels, indices in build_memory_data(cfg): with torch.no_grad(): feat = model(images.cuda()) # getting embeddings from model corresponding to in put self.feats.append(feat) self.labels.append(labels.cuda()) self.indices.append(indices.cuda()) self.feats = torch.cat(self.feats, dim=0) self.labels = torch.cat(self.labels, dim=0) self.indices = torch.cat(self.indices, dim=0) # if memory_ratio != 1.0 -> random sample init queue_mask to mimic fixed queue size if self.ratio != 1.0: rand_init_idx = torch.randperm(int(self.indices.shape * self.ratio)).cuda() self.queue_mask = self.indices[rand_init_idx] #selecting random indexes from dataset corresponding to ratio # feats = embeddings corresponding to input features in model # indices = index of the input features def enqueue_dequeue(self, feats, indices): self.feats.data[indices] = feats # updating features corresponding to indices in memory batch if self.ratio != 1.0: # enqueue self.queue_mask = torch.cat((self.queue_mask, indices.cuda()), dim=0) #enqueueing new indices # dequeue self.queue_mask = self.queue_mask[-int(self.indices.shape * self.ratio):] # removing starting indices def get(self): # returns memory batch with labels if self.ratio != 1.0: return self.feats[self.queue_mask], self.labels[self.queue_mask] else: return self.feats, self.labels
paper : https://arxiv.org/abs/1912.06798
If you are passionate about deep learning, or simply want to say hi, please drop me a line at [email protected]. Any suggestion regarding the blog are welcome. https://www.linkedin.com/in/srijan-agrawal-695652164/