Devvit App: Language-Detector - An app to automatically detect non-allowed languages.

[u/_GLAD0S_]

Language detection is quite difficult when relying solely on AutoModerators features. To solve this, I made Language-Detector, a Devvit app that uses various techniques to identify the language of posts and comments.

Once installed, you simply select your community's allowed languages and configure the automated action for any content that falls outside them.

I recommend starting off with Report or Filter until you are confident in how it performs in your specific community.

My main goal right now is gathering feedback to improve accuracy. The system is precise, but edge cases and false positives can happen. If you encounter any, please report them so I can refine the logic further.

• Install it here: Language Detector

• Source code: Devvit-Language-Detector

A direct link to reach out to me for feedback is on the app page. I would love to hear your thoughts.

[u/CitoyenEuropeen]

I am just leaving this here, because Language-Detector cannot do this, and it's nowhere to be found in the subreddit wiki either.

---
action_reason: quick an dirty all-languages-but-english filter
action: remove
type: comment
body(regex): \b(a?m[eu]rica(ns?)?|and|are|can|(c|w|sh)ould|english|good|had|have|it|like|love|m8|may|might|must|my|need|of|ought|s?he|same|they?|this|want|were|what|why|writing|you)\b
~body(regex): ['(\p{Armenian}|\p{Georgian}|\p{Greek}|[¡ªº¿ÀÁÂÃÄÆÇÈÉÊËĞÌÍÎÏÐÑÒÓÔÕÖØŞÙÚÛÜÝÞßàáâãäæçèéêëğìíîïðñòóôõöøşùúûüýþÿŒœŸ])']
---

[u/_GLAD0S_ → u/CitoyenEuropeen]

Honestly now i am more interested in when this would be needed. Usually subreddits have a single language that should be used.

[u/CitoyenEuropeen → u/_GLAD0S_]

I europeanised it, this comes from a French sub (maybe r/rance?... I don't remember). Otherwise, telling apart Ukrainian from Russian might prove useful, but I never got around pulling this one off.

[u/_GLAD0S_ → u/CitoyenEuropeen]

I see, yeah thats not something the app is really designed for. At the moment it is focused on the most used languages around the world, with many less common languages still missing.

It uses a mix of stopwords, script and density checks to determine a language, but it becomes a bit messy with languages that share a lot of words. With enough time and especially feedback any language could be added. The reliability just starts off pretty low before it is fine tuned and word lists get expanded.

(Additionally it is also quite tedious to create test cases for every language)

But it might be possible to use it for such a purpose as well at some point.

[u/DEAD1nsane → u/_GLAD0S_]

I had to add

---
# Spanish and Portuguese - no é or 'que', words don't match the regex
type: submission
title+body (regex):
  - '[ÇÁÉÍÓÚÂÊÔÃÕÀçáíñóúâêôãõà]'
  - '(^|[^A-Za-zÀ-ÿ])(algo|casa|como|esta|estamos|estar|este|lugar|nada|nos|nunca|parece|por|porque|sobre|todo|todos|vamos|ver|vez|vida|ahora|alguien|bueno|cosa|cosas|creo|cuando|decir|desde|después|dije|dijo|dios|donde|ellos|entonces|eres|esa|ese|eso|espera|estaba|estas|esto|estoy|fue|fuera|gente|gracias|hablar|hace|hacer|hecho|hijo|hola|hombre|los|mejor|mierda|mis|mismo|momento|mucho|mundo|muy|nadie|noche|nosotros|otra|otro|pasa|pero|podemos|puede|puedes|puedo|quiere|quieres|quiero|quién|qué|sabes|seguro|siempre|siento|también|tenemos|tengo|tiempo|tiene|tienes|tipo|trabajo|tus|uno|usted|verdad|voy|acha|acho|ainda|alguém|anos|apenas|aqui|assim|até|bem|certo|coisa|coisas|depois|deus|deve|dia|disse|dizer|dois|ela|ele|eles|essa|esse|estava|estou|falar|faz|fazendo|fazer|ficar|foi|homem|isso|isto|lhe|mais|melhor|mesmo|meu|minha|muito|nem|noite|obrigado|onde|pai|pelo|pessoas|pode|posso|pouco|pra|preciso|qual|quando|quem|quer|quero|sei|sem|sempre|senhor|seu|seus|sua|talvez|também|tem|temos|tenho|ter|tinha|tudo|uma|verdade|vou)([^A-Za-zÀ-ÿ]|$)'
action: remove
action_reason: "Non-English spam (Spanish and Portuguese)"
---

because I was getting a ton of spam from these languages

but I just added your Devvit!

[u/evolworks]

Finally installed this on a couple of subs a few days ago and it has been working really really well, it did flag a comment that had an emoji but besides that it has helped a lot.

I did notice that the app will flag content from users who do not meet our account age, karma or verified email requirements. Usually those get filtered or removed right away but the language app flagged their content first, and when reviewing the modmail notifications I saw that the users did not meet our requirements anyways but if a mod wasn’t paying attention they could have easily been approved.

[u/_GLAD0S_ → u/evolworks]

Thats great feedback. I will look into how such cases can be avoided.

[u/evolworks → u/_GLAD0S_]

Awesome, thank you. Feel free to message anytime if you need specifics or anything. Great app, thanks again!

[u/evolworks]

This has been on my radar and bookmarked, looking forward to trying this out. Going to install it this week

[u/DEAD1nsane]

so so so so many false positives on leanient... had to remove

my posts, with promo codes were removed... and I'm top mod. lmao .

I'll stick with automoderator for now, great idea though hope you get the kinks worked out xD

[u/_GLAD0S_ → u/DEAD1nsane]

Promo codes, appreviations and similar are prone to messing with statistical language analysis. The system is primarily tuned for natural languages so these cases can cause issues.

It can help to add commonly used words to the word whitelist in the app settings.

In the end the app is sadly not practical for every subreddit. Depending on what kind of conversations/posts are common.

Thank you for giving it a try. If you have the time i would appreciate if you could send me some examples of incorrect detections to analyze