Spider & human filtering for redirection & outbound links

Dec 8th, 2010 by No Comments

Spider Filtering & Link Profiles

When creating masked links, we can allow human visitors to continue to the remote location and send all spiders back. We also apply this technology to our cookie, redirection, and popup modules as well to prevent excess resource overhead.

For Redirection Profiles

We can also redirect human visitors and ignore non-human visitors.

How do we determine human versus non-human?

We use a variety of methods to determine what constitutes a spider and what constitutes a human. Our methods include a mixture of useragent detection and browser feature detections, making sure that visitors meet both criteria before we consider them to be a human.

Alternatively we can specify specific IP addresses to consider as ‘other/non-human’.

Block spiders by useragent or IP Address

Leave a Reply