Open AI’s GPT-2 for lyrics generator
Advanced text generator model for lyrics
GPT stands for Generative pre-trained Transformer, which is an autoregressive language model that uses deep learning to produce human-like texts. The second generation of the GPT series was created by OpenAI. It is considered to be the largest artificial neural network created to date. This family of models works like autocomplete in your phone that tries to predict your input.
The idea behind a generative language model is to use supervised learning, which trains under a sequence of structured language and predicts the next words. The math behind the model is a probability distribution conditioned on the words given to the computer. The unigram model is similar to Natural Language Processing or Recurrent Neural Network, where it takes all the words and mixes them, then selects random words from the previously trained words. The Bigram language model(unsupervised multitasking)is conditioned on only the previous output, meaning it’s learning the probability distribution according to the previous words instead of randomly selecting.
The Beatles are well known for their catchy yet classic songs. Their lyrics are a mix of sentimental, vigilant, and romantic. The mixed style makes it harder for AI to learn than uni-themed lyrics, but it’s also interesting to see what GPT-2 is capable of. Thus, I downloaded lyrics from Kaggle Lyrics Dataset and organized them into an excel sheet, where each line of lyrics has its own line Excel; This format helped with structuring the output as GPT2 learns the input format, so I tried tokenized input data format.
The very first step is to understand and explore the dataset. What Beatles lyrics are alike? Which words are the most frequently used? Hence, I used Natural Language Processing and checked the frequency of tokenized lyrics’. Turns out “ love” is the most frequently used word in Beatles’ songs. My project collaborator Tiffany did a word cloud visualization based on the data to give us a visually appealing way of understanding the data(see below).
Before we set up the environment, I first needed to build a VM file path on Google Colab, so it’s easy for GPT2 to read the data.
Then we downloaded the GPT2 and its environment, then deleted the model.
Deploying the model by choosing the specific size of the model we want, which was 355M because that was the most efficient based on my experience with the smallest one; also, the output was not ideal(didn’t make sense to read). I think the largest model would be pretty impressive according to the test results from Open AI GPT2 official website.
After downloading the model and specifying the fine-tuning conditions, we started to train the model using the Beatles lyric dataset.
After only 100 steps, there’s already one very poetic and stunning sample lyric came out:
Open AI soon will release the GPT-3/ChatGPT publicly, and Microsoft just announced an agreement with OpenAI to license GPT-3. I look forward to the new version of GPT.
References:
