Change Log

2.0.0

Oct 3rd, 2023

Saturday, 9 May 2020

  • New : Hosting arrangements
  • New : Updated Canonical URL rules
  • New : Robots meta and rel nofollow / no index rules
  • New : Diagnostic tool improvements
  • New : Increased the max page size to 300Kb
  • New : Detect Javascript sites and warn
  • New : Detect and warn about 403

Thursday, 19 March 2020

  • New : Privacy policy updates.
  • New : Consent and use of cookies.
  • New : Automatic download of your data.
  • New : Right to erase your data

Thursday, 11 October 2018

  • New : Serverless computing code and infrastrcuture update (see below).
  • New : Updated processing rules and limits to reduce wasted resource (see below).
  • New : Updated EULA and privacy statement.
  • New : Updated captcha / human detection.
  • Fix : Tweaked error logging to remove redundant content.
  • Fix : Error and log links on sitemap download page.
  • Fix: Image parsing unprotected “critical section” error.

Monday, 5 September 2016

  • New : Upgraded website to Zurb Foundation 6
  • New : Improved website performance
  • New : Minor cosmetic / content updates to our website
  • Fix : Sitemap Image url issue.
  • Fix : Contact us form issue
  • Fix : Donate form
  • New : More reliable email notifications

Monday, 30 May 2016

  • New : Website diagnostic tool -Fix: processing of http equivelent refresh meta tag.
  • Fix : noindex / nofollow rules to ensure pages that we’re not indexed we’re still followed where there wasn’t a nofollow value.

Monday, 16 May 2016

  • New - Sitemap Upgrade / update

Sunday, 2 March 2014

  • New : Sitemap files available as a single zip file download.
  • New : Improved download page
  • New : New error report.

Saturday, 15 February 2014

  • Fix : Improved Character encoding
  • Fix : Improved HTTP 301 redirect and 302 Moved handling
  • Fix : Improved pattern matching
  • Fix : Fixed https validation

Thursday, 13 February 2014

  • Fix : Spider Performance Improvements

Sunday, 9 February 2014

  • New : Follow meta refresh tag e.g. http-equiv=“refresh”
  • New : Automatically follow both www. and non www. urls.
  • Fix : Errors with duplicate urls for frames.

Monday, 29 April 2013

  • New : Support for RSS Sitemaps added

Friday, 26 April 2013

  • New : Index all file extensions by default

Monday, 21 January 2013

  • Fix : Indexing websites with port number.
  • Fix : Some issues with indexing framesets and image maps.

Thursday, 10 January 2013

  • New : File limit increased from 100 to 250 (as a trial).
  • New : Index iFrames and Frameset option.
  • New : Default settings changed so it will spider all links and query strings by default.
  • Fix : Some problems with checking querystring and file extensions.

Saturday, 5 January 2013

  • New : Email alert facility to let you know when your sitemap is ready.
  • Fix : Error locating sitemap when spidering from a non root location.

Tuesday, 1 January 2013

  • New : Spider will now follow image-maps