Crafting Self-Correcting Prompts with ChatGPT

Crafting Self-Correcting Prompts with ChatGPT: A Case Study on Culturally-Specific Words

Introduction

In the ever-evolving domain of language models, ChatGPT has made monumental strides in natural language processing. However, the Achilles’ heel of ChatGPT and similar language models is their tendency to hallucinate, or fabricate, responses. This calls for the innovation of techniques to curtail these hallucinations. In a previous blog, I explored how embedding tests as guard-rails helped in generating reliable code snippets. In this piece, we delve into a novel approach, inspired by Harrison Chase, founder of LangChain, to craft prompts that can evaluate themselves, especially when dealing with highly culturally-specific words.

The Challenge

As a digital nomad, I've had the privilege to immerse myself in various cultures across the globe. In doing so, I've stumbled upon certain words that are deeply ingrained in the respective cultures, so much so that they elude direct translation. These words are like the tips of cultural icebergs, giving us just a glimpse of the richness hidden beneath. For instance:

• Brazil's "saudade"
• Thailand's "sabai sabai"
• Croatia's "fjaka" and "pomalo"
• Costa Rica's "pura vida"
• Hawaii's "aloha"

These culturally-specific words possess layers of meaning, context, and emotion that are nearly impossible to capture in a translation. However, it would be an intriguing exercise to see if we can create a ChatGPT prompt that handles these words with the finesse they deserve, without hallucinating.

Harrison Chase's Idea

During a recent panel discussion on LangChain & Ray, I raised the query of generalizing the guard-rail technique to write prompts that can evaluate themselves. Harrison Chase proposed asking ChatGPT to generate a prompt based on examples, and possibly counterexamples.

The Experiment

I decided to put Harrison's idea to the test. I gave ChatGPT a list of culturally-specific words without any context and labeled them as input and output:

Write a ChatGPT prompt for generating output from input 
using the following example:

Input,Output
Costa Rica,Pura Vida
Thailand,Sabai Sabai
Croatia,Pomalo
Croatia,Fijaka
Brazil,Saudade
Hawaii,Aloha
USA,This is a free country
            

ChatGPT suggested:

"Translate the cultural ethos or spirit of the following 
countries and regions into their respective idioms or 
phrases. If a country or region has more than one, you 
may include them as well."
                
For example:
                
* If the input is "Costa Rica", the output should be "Pura Vida".
* If the input is "Thailand", the output should be "Sabai Sabai".
* If the input is "Croatia", the output should be either "Pomalo" or "Fijaka".
* If the input is "Brazil", the output should be "Saudade".
* If the input is "Hawaii", the output should be "Aloha".
* If the input is "USA", the output should be "This is a free country".
                
            

This self-correcting prompt by ChatGPT captures the essence of translating the cultural ethos into idioms or phrases. The word "ethos" is not a word I would have though to use. I know the word, but it's not part of my daily vocabulary. I think ChatGPT did a great job selecting that word to describe what I was looking for. By feeding ChatGPT examples and framing the request clearly, the language model is given a context to operate within, which could reduce the chances of hallucination, perhaps.

Results

I chose a random list of countries and prompted ChatGPT and I think the results make sense for the cultures I have some awareness of:

• India: "Atithi Devo Bhava" (Guest is equivalent to God)
• China: "Yǒu péng zì yuǎnfāng lái, bùyì lè hū" (It’s always a pleasure to greet a friend from afar)
• Italy: "La Dolce Vita" (The Sweet Life)
• Mexico: "Mi Casa es Tu Casa" (My House is Your House)
• Egypt: "Ma’alesh" (Never mind/It’s okay/No problem)
• Japan: "Omotenashi" (Wholehearted hospitality)
• Greece: "Philoxenia" (Friend to the stranger – Hospitality)
• France: "Joie de Vivre" (Joy of Living)
• Peru: "Sumaq Kawsay" (Quechua for Good Life)
• Turkey: "Hoşgeldiniz" (Welcome – used as a way to say be our guest)
• Spain: "La Siesta" (A short nap taken in the early afternoon)
• Iran: "Ta'arof" (A complex system of politeness involving both verbal and non-verbal communication)
• Ethiopia: "Melkam Addis Amet" (Happy New Year – reflecting their unique calendar)
• Russia: "Davai" (Commonly used to mean "Come on" or "Let's go")
• Indonesia: "Gotong Royong" (Working together / communal helping)
• Morocco: "Marhaba" (Welcome)
• South Africa: "Ubuntu" (Humanity towards others)
• Australia: "She'll Be Right" (Everything will be alright)
• USA: "The American Dream" (The belief that anyone, regardless of where they were born or what class they were born into, can attain their own version of success in a society where upward mobility is possible for everyone)

Conclusion

Addressing the hallucination issue in ChatGPT and other language models is paramount for their reliable utilization. The method of using examples to create self-correcting prompts, as demonstrated in this case study, showcases how we can bring guard-rails into the realm of language processing. Let's continue to innovate and perfect these methods, to harness the power of language models responsibly and effectively.