Web Content Crawler

mindUp is using crawler systems for a variety of web applications. The reason, why a crawler visits your domain and pages can be:

• You are using an advertisement element in your web pages from an advertising network or online marketing service. Some advertisements are sensitive to the page context, so a crawler has to read all other content from the page, to generate information of the best fit for advertisements.
• You are publishing interesting content on a domain. The Datenbutler visits your domain and some web pages to understand, what your pages are about. It will never read the complete domain and all pages and will never store the complete content.

The mindUp crawler only stores those data, that is legally allowed. The crawler will never read and store the complete content of all pages. For content classification purposes the crawler stores keywords and some text parts relevant to the context.

Back to FAQ