With the improvement of personal ability, I began to be responsible for some new project operations, although these projects optimized the keyword ranking progress is relatively good, but in the operation process I encountered a lot of problems, the discovery of these problems, solve let me realize the importance of seo work in the details, it can be said that the details determine success or failure. In particular, through the actual operation of the project, I have always realized the importance of robots files in website optimization. Therefore, seo children's shoes, especially some children's shoes in the field of seo, must be skilled in mastering the robots file instructions and matters.
The robots file, strictly speaking, is the robots.txt file, its true connotation should be understood? In fact, robots.txt is not a command, but a protocol. When the search engine crawls a web page, it will crawl the robots.txt file first, so the meaning of this file is to tell the search engine which pages can be crawled, which pages can not be crawled. When a search engine crawls a page within a website, it will first visit the robots file under the root directory of the website, if there is this file, the search engine will crawl according to the content defined in the file, if not, the search engine will crawl all allowed it can catch the page. In my understanding, a robots file is like telling search engines which pages to visit and which not to visit.
So what is the real writing of a robots.txt file? In the operation of the Rongli site, because this site used to be an old site, and is a dynamic page, is now being adjusted to a static page, so there will be a lot of original files are gone, so the search engine can not crawl, there will be a lot of crawling errors, as many as two thousand, This is very necessary to use robots files to define these pages that have not been found, so that search engines no longer crawl. These are related to the writing of the robots.txt file. There are two basic principles in a Robots file that are the simplest to write in a robots file. The first is User-Agent: a rover that applies the following rules; The second is Disallow: Web pages to be blocked, and Allow is defined to allow search engines to include the address. So the document of robots is as follows: User-agent: * (where * represents all search engine types, * isa wildcard)Disallow: /admin/ This definition is to prohibit crawling the directory below the admin directory Disallow: /require/ Disallow: /ABC/ Disallow: disallow: /ABC/ disallow: disallow: /require/ disallow: disallow: Disallow: /cgi-bin/*.htm Disallow access to all urls (including subdirectories) with the suffix '.htm' in the /cgi-bin/ directory. Disallow: /*? Disallow:/ab/adc.html Disallow:/ab/adc.html Disallow:/ab/adc.html disallow :/ab/adc.html Allow: /cgi-bin/ Allows crawling directories under the cgi-bin directory Allow: /tmp Allows crawling the entire tmp directory Allow:.htm$Only allows access to urls with extensions of '.htm'. Allow:.gif$allows to crawl web pages and gif format images and in my site operation, there are two basic principles that will be used, that is, the simplest robots file, they are User-agent: *Disallow: /english/(i.e., define the file below the english directory)
Note that the robots.txt file must be placed in the root directory of a site, which generally needs to be added in the background of the site; The robots.txt file name must be lowercase. These two points seem very simple, in the operation is a lot of seo children's shoes are very easy to make mistakes, so it is still a reminder. Now there are more and more seo projects on hand, and I really feel that seo work is not a simple thing. This also tells me: in the future, when operating seo projects, we should pay attention to details, so that the operation of the operation will be more and more confident.