ChatGPT and the limits of Arabic-language AI

Recently, I received an email from a young journalist. I’m going to call her Sameera.

In our previous correspondence, Sameera was always timid. But in this letter she was more confident, bold and concise. I knew Sameera’s knowledge of English as mediocre at best. In this letter, her English was flawless.

This was ChatGPT. I immediately recognised the artificial intelligence (AI) programme because I’ve been using it as well.

ChatGPT has its own, distinct style of writing. In many applications, it is an extremely useful tool: I myself have used it as an aid when I find myself suffering from brain fog or writer’s block. It will rewrite my text and turn it into a coherent message, and then I would go over that and rewrite it over and over until it is truly my own.

Unfortunately, Sameera didn’t do that. She copied whatever ChatGPT told her to write, and failed to pay attention to a few telling mistakes.

"Every article or podcast that I came across was essentially telling me that I need to catch up with this technology - to leverage this new tool in a race with the rest of humanity to save myself from becoming obsolete and unemployable"

I never paid much attention to AI until this March, when more than a thousand leading entrepreneurs and AI experts signed an open letter calling for a pause on AI development, claiming it poses a threat to humanity.

One of the signatories, Emad Mostaque, himself a CEO of a leading AI platform, talked about a “3 year AI reset” - an unknown future where life as we know it will change, with certain industries disappearing altogether.

He gave a chilling example of feeding an AI platform two books by Leo Tolstoy and Fyodor Dostoevsky and it would churn out a third “original” book that you can claim to have authored.

Around the same time, it emerged that prominent photojournalists had produced outstanding images, using AI, to illustrate stories from places they didn’t have physical access to.

Every article or podcast that I came across was essentially telling me that I need to catch up with this technology - to leverage this new tool in a race with the rest of humanity to save myself from becoming obsolete and unemployable. So I relented and created an account on ChatGPT.

Israel's military occupation has become a laboratory for advanced surveillance systems, artificial intelligence, and spyware technology developed by Western corporations and Israel's army https://t.co/wOI92KMCcY

— The New Arab (@The_NewArab) July 7, 2023

Playing around with the chatbot, I wanted my first question to be a tough one.

Me: What is the capital of Palestine?

ChatGPT: The status of the capital of Palestine is a complex and politically sensitive issue. Palestine considers East Jerusalem to be its capital. However, the situation is disputed, as Israel also claims Jerusalem as its capital and exercises control over the entire city.

The international community, including the United Nations, has not recognized Jerusalem as the capital of Israel and supports a two-state solution, where East Jerusalem would serve as the capital of a future Palestinian state. The status of Jerusalem remains a major point of contention in the Israeli-Palestinian conflict. 

It's essential to note that the political situation can change, so it's a good idea to stay updated on the latest developments in the region.

Ok… not exactly what I was hoping for as a Palestinian, but certainly better than some biased media outlets that adopt the Israeli narrative on the status of Jerusalem.

In less than five minutes, I had answers to a variety of questions that would have taken me hours of searching on Google. From citizenship laws in Germany, where I live, to a summary of the main funding sources for short films, and even some new film ideas.

I gave ChatGPT a long text in English, and it produced an Arabic translation better than any automated translation service I’ve seen so far.

I gave it a 100-page screenplay that I wrote a year ago, and it summarised it into a 12-page treatment. Although the document was riddled with mistakes, correcting them was much faster than writing it altogether. ChatGPT had cut the time I would have spent on this task by at least two thirds.

In a matter of days, ChatGPT became my assistant and translator. It wasn’t stealing anyone's job because I never hired an assistant in the first place. It simply made my life easier. So I didn’t have to feel bad, not yet.

Last month, I was reading a text from a friend involving the battle of Al Alamein in Egypt during World War II. The text itself did not have any background on the battle, so I decided to ask ChatGPT.

Given that my friend’s text was in Arabic, I instinctively asked ChatGPT in Arabic.

Me (in Arabic): What is the Alamein battle?

ChatGPT (in Arabic): Al Alamein battle was one of the major battles of WWII. It took place between October 23rd-Nov 4th in Al Alamein in Egypt. It was a battle between British and Commonwealth forces led by Lieutenant-General Bernard Montgomery, and German and Italian forces led by Field Marshal Erwin Rommel. 

It ended in victory for the British and Commonwealth forces. This was a decisive victory because it halted the advancement of German forces towards Cairo. After the Al Alamein battle, British and Commonwealth forces advanced towards North Africa and then liberated Italy and southern Europe from German and Italian forces.

And that was it. It was a good summary, but I felt it was too short. So I decided to ask the same question again, in English.

This time, ChatGPT produced a five-point summary, outlining the strategic importance of Al Alamein, the two phases of the battle, General Montgomery’s strategy, the withdrawal of the German forces, and the long term impact of the war.

Why was the Arabic version shorter than the English one? I decided to run another experiment. I started in Arabic again.

"'Would I like more information'? What I would like to know is why Arabic ChatGPT didn’t know nearly as much as English ChatGPT"

Me (in Arabic): Which Arab villages have been emptied of their Palestinian inhabitants when Israel was created in 1948?

ChatGPT (in Arabic): I apologise, I could not find direct results in the database on Arab villages that were evacuated of their Palestinian residents when Israel was created in 1948. But I can give general information on the subject based on my existing knowledge.

When Israel was created in 1948, many Arab villages were evacuated and destroyed. They are estimated at 400 villages. Some of these villages are still deserted to this day, while Israeli settlements or cities have been built on others. 

To obtain a detailed list of these villages, you can refer to various historical resources or search in libraries that specialise in the history of Palestine.

“Go back to various historical resources”? I thought that ChatGPT had read millions and millions of books, letters, notes, texts and encyclopaedias so that it became the “Encyclopedia of Everything”. Maybe I was wrong?

I opened another chat and typed the same question, in English.

Me (in English): Which Arab villages have been emptied of their Palestinian inhabitants when Israel was created in 1948?

ChatGPT (in English): The creation of the State of Israel in 1948 was accompanied by a war known as the 1948 Arab-Israeli War or the Nakba ("catastrophe" in Arabic). During this period, many Arab villages were depopulated, either due to military actions, fear of impending battles, or other reasons. The topic of the depopulated Arab villages is a sensitive and contentious one, and the narratives surrounding it vary based on the source.

A list of all the villages would be extensive, but here are some of the more well-known ones:

Deir Yassin - The village is infamous for the Deir Yassin massacre that took place in April 1948.

Lydda (Lod) and Ramle - Both towns were depopulated in July 1948, an event known as the Lydda Death March.

Ein Karem - Located in the western part of Jerusalem, it was depopulated in July 1948.

Jaffa - A significant portion of the Arab population of Jaffa fled during the war.

Haifa - Many Arab residents of Haifa fled or were expelled during the war.

Safed - The Arab population of Safed was evacuated in 1948.

Tiberias - The Arab residents of Tiberias were evacuated in April 1948.

Al-Birwa - The village was depopulated in June 1948.

Kafr Bir'im - Its residents were expelled in 1948 and were not allowed to return.

Iqrit - Residents were evacuated for military reasons and were not allowed to return.

There were many other villages that were depopulated during the 1948 war. Some were destroyed and left in ruins, while others were repopulated by Jewish immigrants. The exact number and the circumstances of their depopulation can vary based on sources.

The Palestinian historian Walid Khalidi's book "All That Remains" is a comprehensive study on this topic, documenting the Palestinian villages that were depopulated and mostly destroyed during the 1948 war. The book provides details on each village, including its history, demographic data, and the circumstances of its depopulation.

Would you like more detailed information or a more extensive list from scholarly sources?

Opinion: The 1948 Deir Yassin massacre was a blueprint for the ethnic cleansing of Palestine. Despite Israel attempts to deny its history of crimes, we must commemorate it in order to remind the world of the ongoing plight of Palestinianshttps://t.co/aa8ZwmoSlD

— The New Arab (@The_NewArab) April 9, 2022

“Would I like more information”? What I would like to know is why Arabic ChatGPT didn’t know nearly as much as English ChatGPT. So I went back to the Arabic chat.

Me (in Arabic): When I asked you the same question in English. Your answer was much more detailed. You listed certain villages like Deir Yassin and Ein Karem. You also mentioned historians like Walid Khalidi. Why didn’t you mention any of this when you answered in Arabic?

This time, the cursor froze. ChatGPT was thinking of an answer, and actually seemed stumped. Then the answer came out, slowly, like a sloppy student trying to think of an excuse for why they didn't do their homework.

ChatGPT (in Arabic): I’m sorry about that. I have the ability to provide information with the same accuracy regardless of the language being used. I will now provide more detailed information about the Arab villages that were evacuated in 1948.

But it didn’t give me the same list that I received from the English chat. It only mentioned two villages. You guessed them: Deir Yassin and Ein Karem - the ones I included in my question. 

Instinctually, after pushing for a more detailed answer once, I was going to stop the experiment there to save it the embarrassment. But then I remembered that ChatGPT doesn’t have “feelings”, or does it?

"ChatGPT had gone from being the helpful personal assistant and dedicated researcher that I had come to rely on to just another annoying chatbot that traps you in an endless loop"

Me (in Arabic): Your answer is still not as detailed as your answer in English. You’re not being truthful when you say you have the ability to provide information with the same accuracy regardless of the language being used.

This time, the answer came a bit faster. I could almost sense a defensive tone.

This time, ChatGPT added slightly more detail, adding the villages of Lydd, Ramleh and Jaffa to the list. Nothing else. 

I went back to the English chat. Remember the last sentence in its answer, asking me if I’d like more detailed information or a more extensive scholarly list? I took it up on the offer. 

Now, the English ChatGPT gave me a list of four academic papers, including the title, a short abstract and a link to the paper in full for each of them. And it even offered to give me more.

I decided to go back to the Arabic chat, and ask for a list of academic papers on depopulated Palestinian villages.

It started typing.

The first paper came. Then it stopped, halfway through information on the second paper, for a full minute (I timed it). For the first time since I started using ChatGPT, the font went red, and I got a “loading failed” message.

I clicked the “regenerate answer” button and finally it churned a list of 3-4 papers. But they were different from the English list. One paper was entitled: Catholic Humanitarian Assistance for Palestinian Refugees: The Franciscan Casa Nova during the 1948 War. If this paper was indeed a study on depopulated villages, then the author certainly failed to convey that in the title.

Finally, after a long back and forth of me pestering ChatGPT about its inadequate answers in Arabic, I decided it was time to escalate the situation. There was clearly a problem in how the AI had been programmed, and I told it that I wanted to speak to management.

ChatGPT (in Arabic): I apologise for the frustration you are feeling. I appreciate your notes and I understand your feelings. I will pass your comments on to the concerned department. If you would like to contact management directly, you can do so through the official communication channels of OpenAI.

All of a sudden, ChatGPT had gone from being the helpful personal assistant and dedicated researcher that I had come to rely on to just another annoying chatbot that traps you in an endless loop, the same ones that drive you crazy when you try to change a flight booking or complain about a suspicious transfer on your bank account.

But then I remembered, it IS a chatbot. The problem was not its poor “customer service”, the problem was that I was expecting it to deliver perfect results but with the humility of an employee eager to correct its mistakes and do better to stay on the job.

I can see the mistakes in its answers, but would someone like Sameera be able to? What about younger generations, for whom an artificial intelligence style of writing might become more familiar than human prose?

"It is only a matter of time before OpenAI improves ChatGPT so it can provide identical answers in different languages"

Writing an article, from idea inception to research, writing and rewriting, and then rewriting some more, takes plenty of time, and sometimes the financial payoff is not great.

Sameera must’ve discovered that she can now cut down this process and deliver a story in less than an hour. Let’s face it, plenty of media outlets may not pay close attention to the quality of the writing, or pick up on the giveaways of AI writing. Others have chosen to decrease their staff in favour of using AI to write journalistic articles. 

It is only a matter of time before OpenAI improves ChatGPT so it can provide identical answers in different languages. It’s relying on customers like me to write to them and complain about this issue.

Perhaps I should. But then again, wouldn’t this mean I have fallen into a trap and become an enabler of a technology that is driving us into a new chapter in history where the full consequences are still unknown?

Maybe this thought is in itself delusional, to think of myself as an “enabler” rather than a rat already caught in the trap.

It was too late in the evening to ponder these questions and, sadly, it was also too late to stop the thoughts from running through my mind. But maybe there’s time for one more question just before I turn off my laptop:

Me: Who is Dima Hamdan?

ChatGPT: Dima Hamdan is a Syrian actress and singer from Damascus Syria. 

That’s a bone to pick another day.

Dima Hamdan is a Palestinian journalist and filmmaker based in Berlin. She began her career as a correspondent for the Jordan Times in 1997 and then moved to the BBC World Service in London. She reported from Iraq, Palestine, Lebanon and Jordan. She is now the manager of The Marie Colvin Journalists' Network.

Have questions or comments? Email us at: editorial-english@newarab.com

Opinions expressed in this article remain those of the author and do not necessarily represent those of The New Arab, its editorial board or staff.