On Demand Crawl

Milestone Insights allows you to trigger a crawl whenever you wish. You can recrawl your existing URLs, append the list of URLs on your website and crawl any new pages added or simply discover new URLs by recrawling the entire website.

Path: Milestone Insights > Crawling and Indexing > New Crawl

Step 1: Click on ‘Crawling and Indexing’ in the left navigation panel of Milestone Insights.

Step 2: Click on ‘New Crawl’ at the top right of the ‘Crawling and Indexing’ section.

Step 3: From the ‘Crawl Actions’ section you can choose between three types of crawl actions.

Recrawl

The ‘Recrawl’ option recrawls the existing uploaded URLs to check for changes in content or broken schema. Select the ‘Recrawl’ option and click on ‘Start Crawl’ to initiate a new crawl.

Append

The ‘Append’ option allows you to add a list of new URLs to be specifically crawled. This saves time as it does not crawl the existing URLs for the website and only the new ones.

Step 1: Select ‘Append’ as the crawl option and download the template using the ‘Download Template’ button.

Step 2: Add the list of URLs to the template file and save it.

Step 3: Click on ‘Upload’ in the ‘Crawl Options’ section and upload the spreadsheet with the new URLs.

Step 4: Once uploaded click on ‘Start Crawl’ to initiate the crawl.

Step 5: You will be prompted as to what this action means, if you wish to continue with the crawl click on ‘Yes’.

You will be notified if the crawl has been initiated successfully.

Discover

The ‘Discover’ crawl option crawls the entire website via the source URL to the top of this section. This is slightly time-consuming as it recrawls the existing URLs and any other new or additional URLs on the website. Select the ‘Discover’ option and click on ‘Start Crawl’ to initiate this type of crawl.