Chicken Chicken Chicken: Chicken Chicken

AAAS 2007

• Hits close to home.
• We Koreans love our fried chicken.
• Wish we had Chick-fil-A though.

Reflections on Trusting Trust

1984

• Ken Thompson’s Turing award acceptance speech.
• Presents a way to plant a Trojan in the complier.
• You build a faulty compiler A, and with it compile another compiler B. Distribute B.
• An interesting and fairly short read. Highly recommended!

Do sequence-to-sequence VAEs learn global features of sentences?

ACL 2020

• Short answer: No.
• Encoder memorizes tokens toward the start and the end. This is measured by distribution of reconstruction loss along sequence length.
• In other words, seq2seq VAEs tend to learn local features.
• Most VAE LMs in literature report low KL values in their latent structure. We don’t yet know if the reported values are acceptable or not.
• Replacing LSTM with bag of words helps VAE LMs learn global features and decrease first word memorization.
• Eliminating posterior collapse in VAEs is not sufficient. LMs must learn global features!

Open Korean Corpora: A Practical Report

NLP-OSS Workshop, ACL 2020

• A survey on Korean NLP datasets for a variety of tasks (32 corpora).
• Document will be updated on arxiv as more datasets become available.

Transformer-based Conditional Variational Autoencoder for Controllable Story Generation

2021

• Another rarely seen Transformer VAE.
• Follows the same latent injection scheme as OPTIMUS.
• SOTA in two controlled story generation datasets: WritingPrompts and WikiPlots.
• Compares perplexity, ROUGE-1, ROGUE-2, and ROGUE-L.

A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text

ACL 2019

• Yet again, we want to prevent posterior collapse in text VAEs.
• KL thresholding combined with encoder pretraining (training with AE objective and resetting the decoder) was the most effective in language modeling.
• However, encoder pretraining also exhibits posterior collapse if without KL annealing.
• Better ELBO does not necessarily mean better latent representation quality, as measured by mutual information or active units.

On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs

2021

• Normally, we pass the last hidden state of LSTM encoders to the decoder in text VAEs.
• This results in a dull latent space.
• If we mean-pool or max-pool from all encoder hidden states, we can mitigate posterior collapse.
• Somewhat reminiscent of Ray Mooney’s famous quote.