UA string and Qualification

For SEO, you must be sure to grant permission against search engine bots or crawlers such as google, yahoo and being while shut out the bad bots. This feature is possible to fulfill your wishes by giving a pair of “UA string” and “Qualification” separated by an applicable rule which can be “:” (pass) or “#” (block).

Syntax and Synopsis

UA string

You can specify a part of user agent string (case sensitive). An asterisk “*” matches all user agents.

Qualification

Currently, you can obtain 7 types of qualification listed bellow:

Qualification Description
FEED True if the request is the feed url.
HOST True if the result of host name is available.
HOST=string True if the host name includes string.
REF=string True if the HTTP referer includes string.
Country code True if the request comes from the specified country.
IP address (CIDR) True if the IP address is within the specific range.
AS number True if the AS number is matched.
* Always true

The host name HOST corresponding IP address will be retrieved via DNS reverse lookup which is disabled by default.

Negative operation

A negative operation “!” can be placed just before a qualification. It inverts the meaning of qualification.

DNS reverse lookup

DNS reverse lookup will attempt to fetch a dns PTR record for an IP address. Its cost against the server’s resources is relatively high. For example, the following picture shows the result of 1st and 2nd time which is available at “Installation information” in “Plugin settings” section. It means that the fetched result would be cached in your server.

DNS reverse lookup

If “DNS reverse lookup” is disabled, then HOST and HOST=... will always return true. For example, Google:HOST would be converted to Google:*. In this case, only user agent string would be checked. If the converted result is one of *:*, *:!*, *#* and *#!*, it would be eliminated.

Examples

UA string and qualification

Sample Description
*:2620:101:4000::/42 Pass if an IP address is within the range.
Google:HOST Pass if UA includes “Google” and host name is available.
Google#!HOST Block if UA includes “Google” and host name is unavailable.
Yandex#* Block if UA includes “Yandex”.
*#HOST=amazonaws Block all UA if host name includes “amazonaws”.
Note: The rules in "UA string and Qualification" will be processed in order. For example, when bot:HOST appears before xyz-bot#HOST, then all the bots will be passed whose user agent string includes "bot" and its host name is available.

References

See also