Often, yes, but nobody can tell you for your business until you test it. Published Arabic scores come from translated exam questions, not from gulf dialect, WhatsApp typing, mixed arabic-english or voice notes. Run a small test on your own inbox first, and keep complaints, payment disputes and legal matters with a person.
#Key takeaways
- Published Arabic scores measure translated exam questions, so they say little about how a model handles your customers' messages.
- The only reliable test is a sample of your own messages, marked by your own team, split by site and channel.
- Mixed arabic-english messages, spelling shortcuts and voice notes each need their own test set.
- Complaints, payment disputes and anything legal should always reach a person, whatever the test shows.
- A good result on a test is a reason to start with drafts that staff approve, not to switch on automatic replies.
#What do the published Arabic scores actually measure?
Anthropic's multilingual page reports Arabic results relative to English on a translated MMLU benchmark: 97.2% for Claude Sonnet 4.5 and 92.5% for Claude Haiku 4.5, both older models. Those are the figures a vendor deck is likely to quote.
The benchmark behind them is the English MMLU test set, translated into 14 languages including Arabic by professional human translators (OpenAI simple-evals, undated, read october 2026). It asks exam-style knowledge questions. It does not contain customer messages or dialect.
#Why is a gulf WhatsApp message a different test?
A customer in Dubai, Riyadh or doha does not write like an exam paper. The message may be in dialect, switch to English halfway, skip vowels and punctuation, or arrive as a voice note. None of that is what a translated exam measures, so a high score and a good inbox result are two separate things.
The vendor's own advice points the same way. Anthropic's documentation (undated, read october 2026) tells readers to test with the languages relevant to their own use case, to submit native script rather than transliteration, and to state the input and output language explicitly.
#How do you test it on your own inbox?
Pull a sample of real messages from every site and channel you run. Remove names and phone numbers first. Have two of your own agents mark what each message is asking for. Then compare that with what the tool understood. Use this checklist.
- Dialect: messages from each country and city you serve, written the way customers actually type.
- Mixed language: Arabic and English in the same message, including product names and place names in English.
- Native script versus latin letters: Arabic typed with latin characters and numbers, which customers often use on phones.
- Typos, short replies and one-word messages that only make sense with the earlier thread.
- Voice notes and photos: test them as their own set, because they depend on the tools around the model.
- Reply language: does the answer come back in the language and register the customer used?
- Edge cases: angry messages, sarcasm and messages that ask for two things at once.
Score each group separately. One blended number hides the group that fails. If voice notes fail and text works, you know where to start.
#What should always stay with a person?
Complaints, payment disputes, safety issues and anything with a legal angle should reach a named agent, however well the test went. A wrong automatic reply in those cases costs more than a slow human one. This is an operations rule, not legal advice.
Set the rule in the routing, not in the model's good intentions. Messages that match those topics, or that the tool is unsure about, go to a person with the original text attached. Staff should also read a sample of the Arabic replies that went out, every week.
We are
Stennir tests the tool on your own Arabic and English messages, shows you where it fails, and sets the rules for what a person must handle. Arabic and English are fully supported, others on request, hosted where your rules require.
We aren't
Stennir is not a chatbot vendor with an accuracy claim to sell you. Stennir does not promise a score before it has seen your inbox.
#What should a head of customer operations do next?
Run the checklist on a sample before you sign for anything. If you want help, Stennir runs this Arabic test on your own messages as part of AI consulting, and if the results justify it, builds the routing and approval steps through AI implementation. See also AI for customer support. You can take the free readiness assessment, or book a 30-minute call.
#Common questions
How many of our own messages do we need to test with?
Enough that every kind of message you receive appears many times. Cover each site, each channel and each language mix. Your team has to read and mark them by hand, so pick a sample people can finish. Add more only for the groups that fail. Stennir sets the sample size on the call.
Does it work with Arabic and English mixed in the same message?
Sometimes, and you cannot know without testing. Mixed messages are common in gulf customer chats, and a translated exam does not include them. Build a separate test set for them, with product names in English and the rest in Arabic, and score it apart from the pure Arabic set.
What about voice notes and images sent on WhatsApp?
Treat them as their own test. A voice note has to be turned into text before anything reads it, and a photo needs its own check. Success on typed messages says nothing about either. Stennir tests each type on your real examples and tells you which ones are ready.
Who checks the Arabic replies before customers see them?
Your own arabic-speaking agents. Start with drafts that staff approve, then let the tool send alone only the message types that pass your test, and keep sampling after that. Stennir sets up the approval step and the weekly sample so the check does not depend on one person.
What should always stay with a human agent?
Complaints, payment disputes, safety issues and anything legal. Also messages the tool is unsure about, or that ask for something outside your rules. Route them to a named person with the original text attached. This is an operations rule, not legal advice.
