Botify is a cloud-based Search Marketing Analytics software. It enables Digital Marketers and SEO Managers to rapidly increase their organic traffic and revenue through powerful & comprehensive website analysis.
Our product combines two technologies to provide unparalleled insights into your website, however large it is: the Botify Crawler helps you establish how Google (and other search engines) should see your website, while the Log Analyzer enables you to collect search engine exploration data from web server logs to identify how Google actually sees your website.
Botify provides results in a high speed, highly detailed, highly-technical SEO website analysis. Our product has the highest crawling capacity in terms of volume as well as speed. Hundreds of indicators are made available through a graphical and flexible interface customizable to each website, the data of which is complemented by organic and social traffic information (imported from Google Analytics). All this information is consolidated into the Botify report, showing a full picture of the website, which parts are currently explored by search engines, and where traffic is currently generated (or not). Plus, with the addition of the Botify Chrome Plugin, users can see all indicators in real time as they navigate their site. With these critical insights, the path for improvements and site structure updates becomes crystal clear.
Crawl Optimization
ratio of urls found by googlebot and botify
botify compares the number of urls found by its crawl of the site and googlebot’s crawl as reported by server logs.
botify splits their tool into two separate features:
analyzing issues hindering site crawling
analyzing googlebot’s crawl patterns
issues hindering site crawling are found through crawls made by their own bot. googlebot’s crawl patterns are analyzed through the supplied server logs. while you can opt to use one or the other, subscribing to both services will combine the crawl and server log data into a single report for additional insight into crawling patterns and issues.
that said, i would argue their server log analysis tool is by far their most valuable offering, especially for large web sites. it’s a rarely seen feature in seo tools today but one i have found to offer a real glimpse into how googlebot (and other bots, too) are prioritizing your content.
getting started with log analyzer
Getting started is pretty simple. i had to upload the server logs to their server and botify would take care of the rest. they recommended having at least a month’s worth of data imported to have an accurate view of server activity. once everything is configured and you have access to the dashboard, the real fun starts.
You can segment the data by three different data points:
dimension type, such as page type, host, or mimetype
time frame the data was captured as a pre-configured range or custom date ranges
bot type, such as googlebot images, google smartphone, or bing
moving around the dashboard, you will find various reports with details around sets of pages or specific urls within a set of pages.
unique urls crawled by tag
botify groups pages by their url structure, known as tags, and reports over time the activity recorded such as the amount of unique urls crawled.
as you drill down into the data sets, you can start see over time the distribution of all this data. it gives you a great idea as to what the short and long term trends of your site are in relation to bot crawling. once you have the data you need, botify makes it easy to extract with the option to export as a csv.
Crawl & Log Reports
some reports botify includes but not limited to are:
ratio of missing, unique, and duplicate html tags (i.e., titles, meta descriptions, and h1s)
distribution of http codes over time by bot
number of unique crawls vs number of visits by tag
average internal links per tag
the above are pretty standard reports one would expect from a crawling tool. however, while i can’t touch upon everything today, i will go through a few reports that really stood out to me while using botify.
screaming frog and other tools will tell you the distribution of pages at each level of depth but did you know how many of those pages by each level of depth are actually crawled by google? botify knows and while it’s at it, botify also presents the number of urls by depth that also had at least one visit (according to the user-agent string).
ratio of crawled and uncrawled pages
crawled and uncrawled pages by level of depth by googlebot.
if you’re not interested seeing it by level of depth, you can easily adjust the report and view the same data by category to get a different perspective on how much crawling each section of your site is receiving.
daily volume of crawling
curious to see the first time a page was crawled by google or when the last time it was crawled before a visit occurred? with the date picker, you can compare different time frames in order to understand when crawling is increasing or decreasing over time (it’s been hinted at that crawl volume may impact how often a page receives traffic but of course you will need to dig into your own data to find out.)
Botify Features
- A/B Testing
- Auditing
- Competitor Analysis
- Content Management
- Dashboard
- Google Analytics Integration
- Keyword Research Tools
- Keyword Tracking
- Link Management
- Localization
- Mobile Keywords
- Rank Tracking
Pricing
As mentioned, their crawler and log analyzer are two separate platforms and therefore both have different pricing structures. the pricing options for the crawler is a bit overwhelming with monthly and pay as you go tiers, but i believe botify does a good job catering to those not looking for much or not knowing how much they need. small agencies can expect to pay anywhere between $50-$300 a month while large agencies will most likely opt for payments at around $400-$600 a month.
