Release notes blog posts
📌 Sitemap generation performance issues
Saturday 12 April 2014
Sitemap generation performance issues We would like to apologise for the recent variable performance over the last few days. We believe we are now starting to get it back under control…. The recent improvements created some additional problems when dealing with slow websites and hung connections. Some websites started locking up resources that were not being released quick enough to serve other sitemap requests. To address this we have added a number of controls to the existing list which are summarised below.
📌 Faster 1000 page XML sitemaps and still free!
Monday 7 April 2014
Faster 1000 page XML sitemaps and still free! 1000 pages We are please to announce a new milestone! We have increased the sitemap file limit to 1000 urls and best of all the service remains free to all. To help enable this we have also been working on the system performance and have introduced a couple of new features to aid this transition. Faster parallel page processing We now scan ahead and download multiple pages in parallel so that the spider engine is not kept waiting for pages.
📌 Improved HTML spidering (Robots, Canonical, Rel)
Friday 28 March 2014
Improved HTML spidering (Robots, Canonical, Rel) We now now parse a number of HTML elements to better understand your website and which files should be in your sitemap. Canonical urls We now detect the link rel=“canonical” tag. Where we detect this tag and it points to another page we will not include the current page in the sitemap and will instead spider the url specified in href attribute of the tag.
📌 Text, HTML sitemaps, Robots.txt and more
Sunday 23 March 2014
Text, HTML sitemaps, Robots.txt and more This version includes some new updates that people have been asking for including a new number of sitemap formats. We recommend that you have a valid robots.txt file, an xml sitemap and an HTML sitemap in your website root folder to optimise your sitemap coverage. HTML sitemaps The great thing about an HTML sitemap is that when you publish it any search engine can deal with it whether they officially support sitemaps or not.
📌 Improved download and error reports
Sunday 2 March 2014
Improved download and error reports We’ve made some improvements to the sitemap download page to make it easier to download your sitemaps. Firstly we’ve made all the files available as a single zip file download. We have also added a simple table that gives you access to your XML Sitemap, RSS sitemap and a New error report.
📌 Improved support for character encoding and redirects
Saturday 15 February 2014
Improved support for character encoding and redirects Character encoding We’ve improved our spider so that it can cope with a wider range of character sets including Arabic and Chinese. Don’t forget that for this feature to work correctly it is important that we can understand your website encoding otherwise our spider won’t interpret it correctly and your sitemap will contain strange characters and symbols. http://www.w3schools.com/tags/ref_charactersets.asp Improved HTTP 301 redirect and 302 Moved handling Not only do we now follow HTTP 301 and HTTP 302 automatically.
📌 Spider Performance Improvements
Thursday 13 February 2014
Spider Performance Improvements Our spider was looking a little tired! At times some users were waiting a while for it to complete its job and in some particularly busy periods getting timeout errors. The good news is we have done some house keeping, clearing out millions of records, re-building, de-fragging, etc. and we are now ticking over a bit more smoothly. Don’t forget if you are having problems you can always contact us.
📌 Support for http-equiv="refresh" added and more ....
Sunday 9 February 2014
Support for http-equiv=“refresh” added and more …. On the 9th of Feb we made some minor updates….. New : Follow meta refresh tag e.g. http-equiv=“refresh” We found quite a number of websites using meta tags in their homepage to redirect to another page. Previoulsy we were not detecting this so our spider only found a hompage. We now take the http-equiv tag url and follow it. e.g. : New : Automatically follow both www.
📌 Support for RSS Sitemaps
Monday 29 April 2013
Support for RSS Sitemaps You can now view and down load your XML Sitemaps in RSS format.
📌 Index all file extensions by default
Friday 26 April 2013
Index all file extensions by default To make your life a little easier you can now index all file extensions by default.