Minor updates and fixes applied

We’ve made some minor updates to our spider for the online and windows version of our sitemap generator.

Binary files

Binary files such as Word Document, Adobe PDF, etc. previously just showed the HTTP server response such as “Ok”. We now simply show the filename of such files.

Domain names

We have relaxed validation on domain names so it is not essential to use punycode. You can therefore decide which domain name you want to use for your sitemap.

Note for you to use one or more domains with your site you need to ensure you are using relative urls within your site. The spider will only follow urls within the same domain.

Canonical, redirect urls and no index urls

We have resolved the issue where urls and pages with a canonical url were still showing in the sitemap, but without a proper title. The urls no longer appear and the redirect / canonical url are spidered.