Other Sitemap Settings

These are more miscellaneous settings that relate to the sitemap generation process.

Extensions to ignore

When we spider a web page we look for links. Each link gets processed to check for HTML / text content which we then search for more links.

Contacting the server and downloading headers / content takes time, so excluding certain files from this process can dramatically improve performance for larger websites with lots of none text pages. For example gallery websites with lots of image links.

We automatically list a number of recommended page extensions that our spider skips over.

Concurrent page requests

You can increase the number of concurrent requests we will make to your server which can increase the speed at which we process your sitemap.

However it should be noted :

• if your website cannot handle the increased number of requests this can not only slow our spider down, but degrade the overall performance of your website for other users while we are spidering.
• in some cases, where website are poorly written or heavily resource constrained, this can even crash some servers.

Email address

This is the email address to which you want email notifications sent to for when your job completes. This means you don’t have to wait for it to complete.