Vision and Language Pre-Trained Models
ALIGN
Introduced by Jia et al. in Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionIn the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss (formulated as normalized softmax) that pushes the embeddings of the matched image-text pair together and pushing those of non-matched image-text pair apart. The model learns to align visual and language representations of the image and text pairs using the contrastive loss. The representations can be used for vision-only or vision-language task transfer. Without any fine-tuning, ALIGN powers zero-shot visual classification and cross-modal search including image-to-text search, text-to image search and even search with joint image+text queries.
Source: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Language Modelling | 54 | 7.34% |
| Large Language Model | 33 | 4.48% |
| Semantic Segmentation | 19 | 2.58% |
| Retrieval | 17 | 2.31% |
| Domain Adaptation | 14 | 1.90% |
| Image Generation | 13 | 1.77% |
| Decision Making | 13 | 1.77% |
| Question Answering | 12 | 1.63% |
| Object Detection | 12 | 1.63% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
