How long does it take to generate my XML sitemap?

Our spider can sometimes take a while to process your website and people ask how long they should wait. The time to generate a sitemap can vary quite dramatically from a few seconds to over 10 minutes and this can be influenced by a number of factors.

Key Influencing Factors

Your website performance

The key limiting factor will usually be how fast your website / web server can respond to our spiders requests. If your web server is on the other side of the world and/ or slow to respond this will delay our spider. Your website may seem fast to you if you are geographically close to it, but our spider may not be.

Size of pages

Our spider has to download and process every page to find links. The smaller and more efficient your pages the fast we can spider. If your pages are large it will take longer to download files.

Number of pages

The more pages you have on your site the longer it will take. So if you have a slow server or lots of larger pages, the more pages you have the longer it will take.

For example if you have 10 pages that take 5 seconds it will take at least 50 seconds to complete. If you have 200 pages and it takes 2 seconds for each it will take at least 400 seconds (6.7 minutes).

Our service load

During busy times our spider may be indexing lots of websites and we have limited CPU, memory and bandwidth. This can cause the service to run slower.

Recommendations

Try to ensure your web pages are well designed and as lean as possible to aid the spidering of your website.

Ensure that you are not throttling the browsing of your website for our user agent
“XmlSitemapGenerator - http://xmlsitemapgenerator.org”

Most importantly when you generate your XML sitemap ensure you enter your email address. That way if there is a delay you will be notified when your sitemap is ready by email.