Mysterious Bingbot Sitemap Failures

A while ago, our Bing Webmaster Tools started reporting Failure status for Bingbot crawling our sitemap. While researching the issue, I didn't find much good information, so I wanted to share my own experience and solution.

Mousing over the failure status icon, we saw the very generic error details:

We encountered an issue when we tried to download this sitemap. Please check the sitemap for errors and resubmit the sitemap.

I checked out sitemaps to make sure they were accessible and error free, and then resubmitted the sitemaps for a week, but Bingbot crawls were still failing.

I contacted Bing Webmaster Support, who pointed me in the direction that a DNS failure prevented Bingbot from crawling the sitemap, despite it working for Google crawler, browsers, etc.

I ran some traceroutes, which did show a request timeout, but that actually still happens now with the problem solved.

We host our site on Google App Engine. In our case, the problem turned out to be that our domain ownership verification (for GAE, Google Search Console, etc) stopped working. Our site was still working correctly in GAE though, which is perplexing. After verifying our domain ownership for GAE, and resubmitting our sitemap to Bing Webmaster Tools (to make sure it attempted a crawl again), the sitemaps worked!

Here's the steps to check your custom domain ownership for your Google App Engine site:

1. Go to https://console.cloud.google.com to go to your Google Cloud Platform Admin Console.

2. In the hamburger menu in the upper left corner, select "App Engine".

3. Select "Settings" from the "App Engine" in the left panel.

4. Select "Custom domains" in the right panel, and if you don't see your domain listed anymore, click "Add a custom domain".

5. In section 1 only, enter the domain that should have been listed and click "Verify". This should take you to the Webmaster Central page to check/setup your domain ownership verification settings.

6. After we fixed our buggy DNS verification txt record, our domain settings all reappeared in Google Cloud Platform Admin Console, with no additional tweaks needed.

If your issue isn't this one, I highly recommend contacting Bing Webmaster Support; they were helpful and provided us more detail beyond the generic failure message in Bing Webmaster Tools.