Sitemap generation performance issues

We would like to apologise for the recent variable performance over the last few days. We believe we are now starting to get it back under control….

The recent improvements created some additional problems when dealing with slow websites and hung connections. Some websites started locking up resources that were not being released quick enough  to serve other sitemap requests.

To address this we have added a number of controls to the existing list which are summarised below.

Control

Status

Description

Max of 1000 pages

Existing

This was  recently increased from 250 to 1000

Page size of 80K

Updated

This was 100K, but to improve performance we had to bring it down. The majority of websites should be unaffected by this.

Response time 30s

New

Your website must respond to a request and send all of the page content within 30 seconds. For the majority websites this should not be an issue.

3 concurrent downloads

New

This has been increased from 1 to 3 to improve sitemap generation speed.

We are continuing to monitor the situation and make further improvements over the coming days.