Minor updates and fixes applied
We’ve made some minor updates to our spider for the online and windows version of our sitemap generator.
Binary files such as Word Document, Adobe PDF, etc. previously just showed the HTTP server response such as “Ok”. We now simply show the filename of such files.
We have relaxed validation on domain names so it is not essential to use punycode. You can therefore decide which domain name you want to use for your sitemap.
Note for you to use one or more domains with your site you need to ensure you are using relative urls within your site. The spider will only follow urls within the same domain.
Canonical, redirect urls and no index urls
We have resolved the issue where urls and pages with a canonical url were still showing in the sitemap, but without a proper title. The urls no longer appear and the redirect / canonical url are spidered.