https://uzmantarih.com Benim sitemin robots.txt dosyası şu şekilde,

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-login.php?$
Disallow: /?s=
Disallow: /search/
Disallow: /cgi-bin
Disallow: /wp-includes

User-agent: Baiduspider
Disallow: /


Sitemap: https://uzmantarih.com/sitemap_index.xml