NLRentIndexBot

A crawler. This page is for site operators who found it in an access log.

NLRentIndexBot collects publicly accessible Dutch rental-property listings so that people looking for somewhere to live can search them in one place.

What it does

• Reads robots.txt before the first request to a host, and obeys it. If we cannot read your robots.txt — a 5xx, a timeout, an HTML page where the policy should be — we treat that as stop, not as permission.
• Makes at most one request per second per domain, usually far fewer.
• Sends User-Agent: Mozilla/5.0 (compatible; NLRentIndexBot/1.0; +https://nlrentindex.nl/bot) and From: crawler@nlrentindex.nl on every request. It never pretends to be a browser or another company's crawler.
• Uses conditional requests (If-None-Match, If-Modified-Since), so a page that has not changed costs you a 304 and no body.
• Reads listing pages. It does not submit forms, does not create accounts, does not log in, and does not touch anything behind a paywall or a login.
• Honours Crawl-delay, X-Robots-Tag, <meta name="robots">, and machine-readable TDM reservations (TDM-Reservation header, /.well-known/tdmrep.json).

What it will not do

• Change or hide its identity to get past a block.
• Retry a 401, 402, 403, 407 or 511. Those are answers, and we treat them as answers — we do not re-request under a different name, from a different address, or through a proxy.
• Solve a CAPTCHA or a challenge page.
• Download your images. When we need to confirm a photo exists, we send one HEAD request and record the status and content type. We never copy the file, and the app links to your image on your server rather than rehosting it.

How to exclude it

Add this to your robots.txt:

User-agent: NLRentIndexBot
Disallow: /

That takes effect on our next read of your robots.txt, within 24 hours. To exclude part of the site instead:

User-agent: NLRentIndexBot
Disallow: /klanten/
Disallow: /zoeken

To slow us down rather than stop us:

User-agent: NLRentIndexBot
Crawl-delay: 10

We will use the slowest of your declared delay and our own default.

How to object

E-mail crawler@nlrentindex.nl. Tell us the hostname. We will:

• stop crawling it, permanently, on the next pass — the block is set by a person and no automated process can clear it;
• remove what we hold from anything we publish;
• reply and tell you it is done.

You do not need a legal basis, a letterhead or a reason. "Stop" is enough.

If you are a private individual whose personal data appears in a listing we show, the same address works, and your rights under the GDPR are set out in the privacy statement of the site that publishes the search — including who the controller is.

How to verify a request really came from us

Anyone can put our User-Agent string in a request. Two ways to check:

• Reverse DNS. Our requests come from a fixed set of addresses, and each one resolves back to a hostname under nlrentindex.nl. Look up the IP that hit you; if the reverse lookup does not end in nlrentindex.nl, it was not us.
• The list. Egress addresses are not yet published here: we are moving to a fixed server address and will list every one of it here, with matching reverse DNS, before this section claims otherwise. We do not rotate addresses and we do not use residential proxies.

If something claiming to be NLRentIndexBot is hitting you from anywhere else, we would like to know: crawler@nlrentindex.nl.

Legal basis, briefly

We aggregate publicly accessible listing information under the legitimate interest of helping people find housing, we identify ourselves on every request, and we stop at every technical access control and every objection. We keep a dated record of each request we make — the URL, the response status, and the robots.txt that was in force at that moment — so that any question about what we did on a given day has an answer rather than an assurance.