Skip to content

Latest commit

 

History

History
120 lines (89 loc) · 7.94 KB

README.md

File metadata and controls

120 lines (89 loc) · 7.94 KB
Raw

Awesome Multimodal Named Entity Recognition 🎶📜

A collection of resources on multimodal named entity recognition.

Sorry, this repo will be not updated, since I already changed my research topic.

Content

1.Description

🐌 Markdown Format:

  • (Conference/Journal Year) Title, First Author et al. [Paper] [Code] [Project]
  • (Conference/Journal Year) [💬Topic] Title, First Author et al. [Paper] [Code] [Project]
    • (Optional) 🌱 or 📌
    • (Optional) 🚀 or 👑 or 📚
  • 🌱: Novel idea
  • 📌: The first...
  • 🚀: State-of-the-Art
  • 👑: Novel dataset/model
  • 📚：Downstream Tasks

2. Topic Order

  • 👑 Dataset
    • (AAAI 2017) Adaptive Co-attention Network for Named Entity Recognition in Tweets [paper]
    • (ACL 2018) Visual Attention Model for Name Tagging in Multimodal Social Media [paper]

3. Chronological Order

  • 2020

    • (ACL 2020) Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer [paper]
    • (COLING 2020) RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER [paper]

  • 2021

    • (AAAI 2021) Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance [paper]
    • (AAAI 2021) RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER [paper] [code]
    • (EMNLP 2021) Can images help recognize entities? A study of the role of images for Multimodal NER [paper] [code]

  • 2022

    • (COLING 2022) Flat Multi-modal Interaction Transformer for Named Entity Recognition [paper]

      • 📌 1st interpolating FLAT with MNER
      • 🚀 SOTA on Twitter15 with Bert_base_uncased but code is unavailable

    • (NAACL Findings 2022) Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction [paper] [code]

      • 📌 code using refined Twitter15 dataset

    • (WSDM 2022) MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition [paper] [code]

    • (SIGIR 2022) Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion [paper] [paper]

      • 📌 1st fully Transformer structure
      • 🚀 SOTA on Twitter17 using Bert_base_uncased but only implement on Twitter17

    • (NAACL 2022) ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition [paper] [code]

      • 📌 Roberta_large as backbone provides powerful improvements
      • 🌱 Using OCR ect without directly using images

    • (MM 2022) Query Prior Matters: A MRC Framework for Multimodal Named Entity Recognition [paper]

      • 🌱 1st MRC based framework for MNER

    • (SIGIR 2022) Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER [paper]

      • 📌 Trustworthy performance by reimplementation

    • (ICME 2022) CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention [paper]

      • 🚀 SOTA on Twitter15 and Twitter17 with Roberta_large
      • 📌 Require 8 V100 GPU

    • (DSAA 2022) PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition [paper]

      • 🚀 SOTA on Twitter15 and Twitter17 with Roberta_large
      • 📌 Require 8 V100 GPU
      • 🌱 Prompt-based

    • （arxiv 2022） Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media [paper] [code]

    • (arxiv 2021) Multi-Granularity Contrastive Knowledge Distillation for Multimodal Named Entity Recognition

      • submitted to ACL2021 but not accepted

    • (arxiv 2022) MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Grounding [paper]

  • 2023

    • (EMNLP 2023) Named Entity and Relation Extraction with Multi-Modal Retrieval [paper] [code in construction]
      • 🚀 retrieve augmentation in MNER image
    • (MM 2023) Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER [paper]
      • 🚀 generative alignment
      • 🌱 free from image during inference

4. Course

5. Thinking MNER

  • MNER is a hard task since it needs multimodal understanding in social media domain. However, existing methods simplify it to extacting helpful viusal clue to assist NER, with a simple showcase. In twitter datasets, the image-text pair always has no or vague relationship, which needs extra information or supervision for model to understand. Therefore, I believe that is why MNER-QG, MoRe, R-GCN and PromptMNER work. However, existing works are still nowhere near logical understanding, since they all introduce out-sample knowledge. Now I am trying to introduce knowledge graph in MNER to provide in-sample context.

  • Tricky task: When I developed my work (SOTA in two datasets but in submission), I found

    1 only tuning task head of BERT (freeze Bert and ViT) can achieve comparable results (0.2-0.5% drop). So I believe we can directly introduce prompt eniggering for text.

    2 large language model matters more than fancy innovation

    3 using a empty image to replace all valid images during test only drop 2%

    4 simple loss like contractive loss brings 2% improvement. So I think the model heavily focuses on text.

    5 Applying the same code in different environments even in different GPUs has results with large variance...

    6 It has two mainstreams depending on image+text or caption+text as input. DAMO-series works mainly focus on the latter one, which is proved to be more extandable and SOTA than others.

Contact Me