Announcement

Collapse
No announcement yet.

[SEO] Xin giúp đỡ vấn đề không tìm nạp được file Robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • [SEO] Xin giúp đỡ vấn đề không tìm nạp được file Robots.txt

    Bác nào đã từng gặp trường hợp này rồi xin giúp em với ạ, em xin cảm ơn

    Em đã tạo a file robots.txt với nội dung như sau:
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /admin/
    Disallow: /download/
    Disallow: /system/
    Disallow: /vqmod
    và up lên thư mục gốc của web
    Nhưng vào phần submit file robots.txt trong webmaster tool thì nó bão lỗi như vầy
    loi-google-robots.PNG
    Đồng thời nó gửi mail về cho em là:
    ht...://*******.net/: Googlebot can't access your site
    Over the last 24 hours, Googlebot encountered 15 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

    You can see more details about these errors in Webmaster Tools.

    Recommended action

    If the site error rate is 100%:
    Using a web browser, attempt to access htt..://****.net/robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot.
    If your robots.txt is a static page, verify that your web service has proper permissions to access the file.
    If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.
    If the site error rate is less than 100%:
    Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors.
    The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website.
    If your site redirects to another hostname, another possible explanation is that a URL on your site is redirecting to a hostname whose serving of its robots.txt file is exhibiting one or more of these issues.
    After you've fixed the problem, use Fetch as Google to fetch ht..://***.net/robots.txt to verify that Googlebot can properly access your site.

    Learn more in our Help Center.

  • #2
    rewrite url sai chắc luôn
    Be different and always different
    http://archlinuxvn.org/
    http://theslinux.org
    http://lab.infosec.xyz

    Comment


    • #3
      Kiểm tra URL anh ơi

      Comment


      • #4
        Cảm ơn các bạn đã quan tâm tư vấn! Bữa đó VPS của mình bị DDOS, mình bật các tường lửa chống DDOS, trong số đó có một firewall ngăn chặn các IP từ nước ngoài truy cập vào nên robots của google không tìm nạp được. Bữa nay sắm mới 1 cái server luôn nên ngon rồi

        Comment

        LHQC

        Collapse
        Working...
        X