Faster 1000 page XML sitemaps and still free!

1000 pages 

We are please to announce a new milestone! We have increased the sitemap file limit to 1000 urls and best of all the service remains free to all.

To help enable this we have also been working on the system performance and have introduced a couple of new features to aid this transition.

Faster parallel page processing

We now scan ahead and download multiple pages in parallel so that the spider engine is not kept waiting for pages. this means we can process pages faster. This will put a slight increase load on your server, but this should not be a problem for most websites. If you wish to disable the feature simply click on advanced settings and un-tick the “Multi threading” check box.

100Kb max page size

One limitation we have introduced is on web page file size. The majority of users will not be affected by this, however any web pages that exceed 100K will only be partially downloaded to protect resources. If your pages are over 100Kb we would strongly urge you to fix this as this is very large indeed and even 50Kb would be excessive especially in the mobile world.

Fixes

There was an issue with processing iFrame urls which has now been fixed.

We’ve also done some work to help prevent our spider tripping over bad HTML as this is where most of our low ratings come from.