Website diagnostic tool
We sometimes get contacted by people with questions about why their sitemap wasn’t generated as expected , indeed in some cases returning very few or no pages.
In most cases the answers are to do with the structure or format of their website and how it responds to our spider.
To help users understand and address these problems we’ve started to automate some standard checks we should normally do manually to help resolve these issues.
The diagnostic tool runs through a series of tests designed to mimic the behaviour of our spider.
We test things like
Accessing your server
The server response codes
Parsing of the HTML content
Important tags such as
titles and headers
canonical urls and
Number of urls found
The tool will return a list of results which can be helpful when trying to understand how our sitemap generator interprets your websites.
Exampe diagnostics output
You can access the diagnostic tool when you download your sitemap and from the help section.
In this release we also fixed a couple of bugs.
- Fix: processing of http equivelent refresh meta tag.
- Fix : noindex / nofollow rules to ensure pages that we’re not indexed we’re still followed where there wasn’t a nofollow value.