Effective Prompt Engineering guide to scale ChatGPT API for Sentiment analysis on multiple survey comments
Make Sentiment Inferences using ChatGPT API within 10 steps!
In my previous article, I shared the steps and code to take many survey comments, hit the OpenAI API, get high-level highlights (text summarization from the entire corpus of documents), and distill the overall sentiment, considering all the comments together. This is usually a first-pass analysis. Most times, the next step would be delving into individual documents (each text comment) within that whole corpus of comments to extract deeper insights.
In this article, I’ll extend the use case for OpenAI (ChatGPT) API to help an end user infer sentiments at a per-document level. In the process, I’ll also share the power of Iterative Prompt Refinement to get closer and closer to the results/format you desire from ChatGPT for your downstream analysis.
Goal:
Infer sentiments at a per-document granularity that can then be used for visualization and sharing with business stakeholders; zoom into any specific sentiment and share it with your leadership team with the associated comments, etc.
(Note: In NLP (Natural Language Processing), one text is one document, while a collection of documents makes a corpus.)
Code:
The main difference/complexity in the code captured in this post vs. the previous stems from the pre and post-data processing to prepare the documents for individual sentiment inference in a format that works for the OpenAI API to make inferences in bulk. Note that what has been covered here is just one of the many ways of handling the data for this task — feel free to tweak it!
# STEP 1: Load data from local file
from google.colab import files
uploaded = files.upload() # Choose the local file after executing this# STEP 2: Parse local file contents into Pandas DataFrame
import pandas as pd
import io
df = pd.read_csv(io.BytesIO(uploaded['survey.csv']))
print(df.shape)
df.head()Executing the above output a Pandas DataFrame, and my dataset only had one text column with the header comments. Note that I’ve intentionally masked out the raw comments with a gray box on top. You should be able to easily get your parsed DataFrame out with the above code on your text data!
One of the ways that I came across to get the OpenAI API to generate predictions at a document level was to loop over every document (text) and get the API to generate an individual inference/prediction. The most common for loops in Python I’ve come across are loops that perform transformations over a Python list . So, that’s how I’ve prepared my documents in the comments column — creating a comma-separated list of all documents I’m calling comments_list, before calling the API. The output of this list will be in this format: [document 1, document 2, document 3...].
# STEP 3: Convert values in text column to list
comments_list = df['comments'].to_list()
print(len(comments_list))
comments_listWe’re now ready to initialize the OpenAI API with the same steps covered in my previous post.
# STEP 4: Install the OpenAI Python library & pass your secret API key
!pip install openai # Install the library if this is your 1st time
import openai # import installed library
openai.api_key = "sk-...." # insert your own API key within " "# STEP 5: Create helper function for task
def comment_summarization(prompt, model="gpt-3.5-turbo"):
messages = [{"role": "user", "content": prompt}]
response = openai.ChatCompletion.create(
model=model,
messages=messages,
temperature=0, # this is the degree of randomness of the model's output
)
return response.choices[0].message["content"]Now that we have our data prepared and the API setup ready, the next step is to call the API to do its inference! For this task, I had to refine my prompts multiple times to get the response in the desired format! By iteratively refining my prompts, I got closer and closer to the desired output.
Prompt Iterative Refinements
Iteration 1:
# STEP 6: Looping through all reviews to get individual sentiments
# Iteration 1
# loop over all comments to follow the instructions in prompt and
# get an inference from LLM
for i in range(len(comments_list)):
prompt = f"""
Your task is to analyse and report the overall sentiment \
of only one out of the following sentiments: \
[positive, negative, neutral, mixed]. \
On the text delimited by triple backticks
Text to analyse: ```{comments_list[i]}```
"""
# call API to perform task in prompt
response = comment_summarization(prompt)
print(i, response, "\n")A note about the backslash (Source)
- In the course, we are using a backslash
\to make the text fit on the screen without inserting newline '\n' characters. - GPT-3 isn’t affected whether you insert newline characters or not. But when working with LLMs, you may consider whether newline characters in your prompt may affect the model’s performance.
The response from the prompt in Iteration 1 was (truncated to just a few comments for ease of digestion):
0 The overall sentiment of the given text is neutral.
1 The overall sentiment of the given text is negative.
2 The overall sentiment of the given text is positive.
3 The overall sentiment of the given text is negative.
4 The text provided, “N.A.”, does not contain any meaningful information to analyze and determine its sentiment. Therefore, it is not possible to report the overall sentiment as positive, negative, neutral, or mixed.
5 The overall sentiment of the given text is mixed. The text mentions both positive and negative aspects of the subject. On one hand, it states that…
Conclusions from Iteration 1:
- The indexes corresponding to a given text are 0, 1, 2...
- ChatGPT followed the boundary conditions I set in my prompt and did infer “positive,” “negative,” “neutral,” or “mixed”.
- It went above and beyond to report the sentiment inferences in a complete sentence format.
- It went above and beyond to explain rationale where required. For example, where it inferred a “mixed” sentiment (index 5) and where it could not infer a sentiment due to lack of context in the text it was inferring (index 4, which corresponded to an original survey response “N.A”), it justified the rationale of not being able to infer a sentiment out of the four sentiments it was given in the prompt.
Iteration 2:
Reflecting on the above output, I improved my earlier prompt to:
- Request ChatGPT to restrict its output to just one-word sentiments and ditch the sentence format for easy handling and downstream analysis
# Iteration 2: Improvising prompt from earlier to restrict to
# single word sentiment
# loop over all comments to follow the instructions in prompt and
# get an inference from LLM
for i in range(len(comments_list)):
prompt = f"""
Perform the following actions on the given text delimited by \
triple backticks:
- analyse and report the sentiment in strictly a single word \
out of [positive, negative, neutral, mixed].
Text to analyse: ```{comments_list[i]}```
"""
# call API to perform task in prompt
response = comment_summarization(prompt)
print(i, response, "\n")The response from the prompt in Iteration 2 was:
Conclusions from Iteration 2:
- For index#4, since in iteration 2, ChatGPT wasn’t given room to explain its inference and was just forced to respond with a single-word sentiment based on my prompt, it inferred that the text comment (originally just “N.A”) was “neutral” sentiment.
- That is not entirely wrong. Organizations often prefer not to leave comments like “N.A.” out of the text analysis in business settings. Instead, they bucket them as “neutral,” with neither positive nor negative comments to account for that input. So, this is rather subjective, and you can make your call based on business direction.
Iteration 3:
Reflecting on the above output, I improved my earlier prompt to:
- Take the stand where texts that didn’t convey any real meaning or where a survey respondent input text such as “n.a,” “not applicable,” etc., I wanted ChatGPT not to tag those comments to an actual sentiment. So, I further refined my earlier prompt to cover such edge cases and return a “not applicable” output.
- Adapted the output format to prepare for packaging into a Pandas DataFrame.
json.loads()handles conversion from the API’s JSON String output to a Python dictionary which can be easily packed into a DataFrame.
# Iteration 3: Improvising prompt from earlier to restrict to
# single word sentiment, edge case handling, and
# preparing to pack output into Pandas DataFrame
response_summary = [] # declare empty list to store all responses
# loop over all comments to follow the instructions in prompt and
# get an inference from LLM
for i in range(len(comments_list)):
prompt = f"""
Perform the following actions on the given text delimited by \
triple backticks:
1. analyse and report the sentiment in strictly a single word out \
of [positive, negative, neutral, mixed].
2. explain the rationale for the predicted sentiment.
3. where the comment is "n.a" or "not applicable" or no meaningful \
text is available for analysis, report the sentiment as "not applicable".
Return the output in the following format:
{{"index": {i},
"sentiments": ```sentiment here```,
"sentiment rationale":```sentiment justification here```}}
Text to analyse: ```{comments_list[i]}```
"""
# call API to perform task in prompt
response = comment_summarization(prompt)
# perform JSON String to Python dictionary conversion per document
response_summary.append(json.loads(response))Conclusions from Iteration 3:
The output in response_summary was in this format (a list of Python dictionaries, ready for DataFrame packaging)
[{‘index’: 0, ‘sentiments’: ‘positive’, ‘sentiment rationale’: “The text expresses a positive sentiment by mentioning that ….”}, {‘index’: 1, ‘sentiments’: ‘negative’, ‘sentiment rationale’: ‘The text mentions that the ... These negative aspects contribute to the overall negative sentiment.’},..]
Using Inferences
I finally created a Pandas DataFrame from the above dictionary, and that then opens doors to further analysis or generating visualizations as required, like the bar chart I created using the ChatGPT API’s sentiment inferences, to summarize that a higher proportion of survey respondents still responded favorably based on their inputs.
# STEP 7: Preparing API Sentiment inferences into Pandas DataFrame
import pandas as pd
sent_df = pd.DataFrame(response_summary)
print(sent_df.shape)
sent_df.head()
# STEP 8: Visualizations from Pandas DataFrame!
import matplotlib.pyplot as plt
# Calculate value counts for each sentiment
sent_counts = sent_df['sentiments'].value_counts(normalize=True)
# Create a bar chart
sent_counts.plot(kind='bar')
plt.title('Survey Sentiments')
plt.xlabel('Sentiments')
plt.ylabel('Values')
plt.show()
Conclusion
In this post, I covered a potential use case for ChatGPT API to infer Sentiments from survey comments, iteratively perfect prompts to get closer to desired results, and data formatting into and out of the API for further analysis.
More such experimentations are underway; stay tuned!
Inspired by knowledge gained from this awesome DeepLearning.AI tutorial.
