Support for http-equiv=“refresh” added and more ….

On the 9th of Feb we made some minor updates…..

New : Follow meta refresh tag e.g. http-equiv=“refresh”

We found quite a number of websites using meta tags in their homepage to redirect to another page. Previoulsy we were not detecting this so our spider only found a hompage.

We now take the http-equiv tag url and follow it. e.g. : 

New : Automatically follow both www. and non www. urls.

Some users were not understanding what domain they were using and where they were specifying the full URL within pages mixxing the www and non www equivelent.
e.g. http://www.xmlsitemapgenerator.org and http://xmlsitemapgenerator.org

If you enter one we now follow both by default but we don’t duplicate your urls. We list them once using the domain you entered on the setup page.

Fix : Errors with duplicate urls for frames.

We noticed that when there were frames on a page with the same url it was causing a spider problems and throwing an error so we fixed this!