beyondBERT

11.5기의 beyondBERT의 토론 내용을 정리하는 repository입니다.

진행방식

• 논문 당 한명의 호스트가 모임을 진행합니다.
• 호스트는 배정된 논문의 idea 위주로 정리합니다.
• 사전학습으로 참가자는 주차 별 논문에 question을 한개 이상 준비하고, 논문 issue의 thread에 question을 등록합니다.
• thread에 달린 question에 thumb up을 하고, thumb up의 개수가 높은 question 위주로 모임시간에 토론하고 해결합니다.

주차 별 논문

week01

• ice breaking
• 진행 방식 결정

week02

• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• How multilingual is Multilingual BERT?

week03

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

week04

• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• Data Augmentation using Pre-trained Transformer Models

week05

• Reformer: The Efficient Transformer
• Longformer: The Long-Document Transformer

week06

• Mask-Predict: Parallel Decoding of Conditional Masked Language Models
• Unsupervised Data Augmentation for Consistency Training

week07

• You Impress Me: Dialogue Generation via Mutual Persona Perception
• Recipes for building an open-domain chatbot

week08

• ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues
• A Simple Language Model for Task-Oriented Dialogue

week09

• ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
• FastBERT: a Self-distilling BERT with Adaptive Inference Time

week10

• PoWER-BERT: Accelerating BERT inference for Classification Tasks
• TinyBERT: Distilling BERT for Natural Language Understanding

week11

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners