May 10, 2021
Open AI’s GPT-2 for lyrics generator
Advanced text generator model for lyrics
GPT stands for Generative pre-trained Transformer, which is an autoregressive language model that uses deep learning to produce human-like texts. The second generation of the GPT series created by OpenAI . It is considered to be the largest artificial neural network created to date. This family of models works like autocomplete in your phone that tries to predict the your input.
The ideal behind a generative language model is using supervised learning, which trains under a sequence of structured language and predicts the next words. The math behind the model is a probability distribution conditioned on the words given to the computer. Unigram model is similar to Natural Language Processing or Recurrent Neural Network, where it takes all the words and mix them, then it selects a random words from the previous trained words. Bigram language model(unsupervised multitasking)is conditioned on only the previous output , meaning it’s learning the probability distribution according to the previous words instead of randomly selecting.
The Beatles is well-known for their catchy yet classic songs. Their lyrics are a mixed of sentimental, vigilant, and romantic. The mixed style make it harder for AI to learn than uni-themed lyrics, but it’s also interesting to see what GPT-2 is capable of. Thus, I downloaded lyrics from Kaggle Lyrics Dataset and processed it into excel sheet, where each line of lyrics has its own line on excel( This format helped with structuring the output as GPT2 learns adaptly, because I have tried tokenized format and the output
The very first step is to understand and explore the dataset. What Beatles lyrics are alike? Which words are the most frequently used? Hence, I used Natural Language Processing and checked frequency of tokenized lyrics’ . Turns out “ love” is the most frequently used word in Beatles’ songs. My project collaborator did a word cloud on the data to give us a visually appealing way of understanding the data.
Before we set up the environment, I first needed to build a VM file path on Google Colab, so it’s easy for GPT2 to read the data.
Then we downloaded the GPT2 and its environment , then deloped the model.
Deploying the model by choosing the specific size of the model we want, which was 355M because that was the most efficient based on my experience with the smallest one; also , as the output was not ideal(didn’t make sense to read). I think the largest model would be pretty impressive according to the test results from Open AI GPT2 official website.
After downloaded the model and specify the fine-tuning conditions, we started to train the model using the Beatles lyric dataset.
After only 100 steps, there’re already one quite poetic and stunning sample lyrics came out:
Open AI soon will release the GPT-3 publicly, and Microsoft just announced an agreement with OpenAI to license OpenAI’s GPT-3. Although AI may outperformance human on certain tasks after training with large amount of data, the concern of AI will take over human is exaggerated, because AI can be biased based on the training data. It also lacks the logical thinking capabilities that humans possess. More importantly, it lacks the emotion and culture perspective of thinking that our ancestor passed on us for years.
References:
