NLP Techniques: Text Generation, Semantic Search & More
Classified in Electronics
Written on in
English with a size of 3.92 KB
Preprocessing the Dataset
a. Normalize the Text
Python
import re
from nltk.tokenize import word_tokenize
import nltk
nltk.download('punkt')
# Preprocessing Function
def preprocess(text):
text = re.sub(r'[^a-z\s]', '', text.lower())
return word_tokenize(text)
# Apply Preprocessing
df['Processed'] = df['Sentence'].apply(preprocess)
print(df)GPT-2 Text Generation
Python
from transformers import pipeline
# Load GPT-2 Model for Text Generation
generator = pipeline('text-generation', model='gpt2')
# Generate Text for a Given Prompt
prompt = "Once upon a time"
result = generator(prompt, max_length=50,
num_return_sequences=1)
print(result[0]['generated_text'])GPT-2 for AI Prompts
a. Prompt 1 - Future of AI
Python
prompt = "What is the future... Continue reading "NLP Techniques: Text Generation, Semantic Search & More" »