# Kemelang - robots.txt # ==================================================================== # NOTICE TO CRAWLERS AND AI BOTS: # Automated scraping of Kemelang data is strictly restricted. # If you wish to access our dictionary datasets, audio corpora, or APIs # for commercial or research purposes, please contact us at: # contact@kemelang.com # ==================================================================== # 1. Bloquer les scrapers et aspirateurs de sites connus User-agent: HTTrack User-agent: SiteSucker User-agent: WebCopier User-agent: wget User-agent: Scrapy Disallow: / # 2. Bloquer les crawlers IA d'entraînement User-agent: GPTBot User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Bytespider User-agent: Scrapy User-agent: HTTrack User-agent: SiteSucker User-agent: WebCopier User-agent: wget User-agent: Scrapy # Bots IA / Crawlers LLM User-agent: GPTBot User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Bytespider User-agent: CCBot User-agent: meta-externalagent User-agent: DotBot User-agent: Qwantify User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: BLEXBot User-agent: PetalBot User-agent: PetalCrawler User-agent: linkfluence User-agent: Mozilla User-agent: curl User-agent: cohere-ai User-agent: nuzzel User-agent: facebookexternalhit User-agent: DeepSeekBot User-agent: gpt User-agent: claude User-agent: perplexity User-agent: anthropic User-agent: python-httpx User-agent: python-requests User-agent: go-http-client User-agent: ruby User-agent: scrapy User-agent: curl User-agent: wget User-agent: HTTrack User-agent: SiteSucker User-agent: WebCopier User-agent: Fetch User-agent: WordPress User-agent: FlowIQLabsBot User-agent: Barkrowler User-agent: DataForSeoBot User-agent: Applebot-Extended User-agent: Amazonbot User-agent: ChatGPT User-agent: Amzn-SearchBot User-agent: dotbot-bot-ua User-agent: ia_archiver User-agent: Claude-User User-agent: Claude-Web User-agent: ChatGPT-Web User-agent: Discordbot User-agent: jscrawler User-agent: OAI-SearchBot User-agent: ExaSearchBot Disallow: / # 3. Règles pour les moteurs de recherche légitimes (Google, Bing, etc.) User-agent: * Allow: / Disallow: /accounts/ Disallow: /admin/ # Protection des fonctionnalités lourdes et dynamiques Disallow: /api/ Disallow: /autocomplete/ Disallow: /translate/ Disallow: /audio-compare/ # Anti-Duplicate Content (URLs de recherche et paramètres) Disallow: /*?*= Disallow: /*?*utm_ # Sitemaps officiels Sitemap: https://kemelang.com/sitemap.xml