# Fort Worth Star-Telegram content is made available for your personal, non-commercial # use subject to our Terms of Service here: # https://www.star-telegram.com/customer-service/terms-of-service/ # Use of any device, tool, or process designed to data mine or scrape the content # using automated means is prohibited without prior written permission from # The Fort Worth Star-Telegram. Prohibited uses include but are not limited to: # (1) text and data mining activities # (2) the development of any software, machine learning, artificial intelligence (AI), # and/or large language models (LLMs); # (3) creating or providing archived or cached data sets containing our content to others; and/or # (4) any commercial purposes. # Global allow for Mediapartners - this is used by Google to place ads in content, # not indexing purposes. User-agent: Mediapartners-Google Disallow: # Throttles to decrease load User-agent: Yandex Crawl-delay: 3 User-agent: Baiduspider Crawl-delay: 3 User-agent: bingbot Crawl-delay: 2 User-agent: Slurp Crawl-delay: 2 User-agent: FacebookBot Crawl-delay: 3 # Story blocks for selected bots User-agent: archive.org_bot Disallow: /*.html User-agent: ia_archiver Disallow: /*.html User-agent: ia_archiver-web.archive.org Disallow: /*.html # Complete blocks of uninteresting bots User-agent: A6-Indexer Disallow: / User-agent: AddSearchBot Disallow: / User-agent: aiHitBot Disallow: / User-agent: Andibot Disallow: / User-agent: ankiro.com Disallow: / User-agent: AspiegelBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: bigsur.ai Disallow: / User-agent: BLP_bbot Disallow: / User-agent: Brightbot 1.0 Disallow: / User-agent: Buck Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGLM-Spider Disallow: / User-agent: CheckMarkNetwork Disallow: / User-agent: coccocbot-web Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: Cotoyogi Disallow: / User-agent: crawler4j Disallow: / User-agent: Crawlspace Disallow: / User-agent: dataprovider Disallow: / User-agent: Datenbank Crawler Disallow: / User-agent: Diffbot Disallow: / User-agent: EasouSpider Disallow: / User-agent: Echobot Bot Disallow: / User-agent: EchoboxBot Disallow: / User-agent: ecoResearch Disallow: / User-agent: FirecrawlAgent Disallow: / User-agent: Grok Disallow: / User-agent: GrokBot Disallow: / User-agent: GrokBot/1.0 Disallow: / User-agent: Grok-DeepSearch/1.0 Disallow: / User-agent: Genieo Disallow: / User-agent: https://hada.news Disallow: / User-agent: iaskspider/2.0 Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: img2dataset Disallow: / User-agent: JenkersBot Disallow: / User-agent: Jetslide Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: KunatoCrawler Disallow: / User-agent: legistorm.com Disallow: / User-agent: LivelapBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Magus Bot Disallow: / User-agent: MauiBot Disallow: / User-agent: MAZBot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: MistralAI-User/1.0 Disallow: / User-agent: Mojeek Disallow: / User-agent: MojeekBot Disallow: / User-agent: MyCentralAIScraperBot Disallow: / User-agent: news-please Disallow: / User-agent: NewsNow Disallow: / User-agent: Netvibes Disallow: / User-agent: nlcrawler Disallow: / User-agent: NovaAct Disallow: / User-agent: Omgilibot Disallow: / User-agent: omgili Disallow: / User-agent: Oracle Secure Enterprise Search Disallow: / User-agent: panscient.com Disallow: / User-agent: PetalBot Disallow: / User-agent: PanguBot Disallow: / User-agent: Poseidon Research Crawler Disallow: / User-agent: Primalbot Disallow: / User-agent: Riddler Disallow: / User-agent: QuillBot Disallow: / User-agent: quillbot.com Disallow: / User-agent: R6_CommentReader Disallow: / User-agent: SBIntuitionsBot Disallow: / User-agent: Scrapy Disallow: / User-agent: SeekportBot Disallow: / User-agent: Seekr Disallow: / User-agent: SeekrBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: SeznamHomepageCrawler Disallow: / User-agent: Sidetrade indexer bot Disallow: / User-agent: Spinn3r Disallow: / User-agent: TaraGroup Intelligent Bot Disallow: / User-agent: Thinkbot Disallow: / User-agent: Timpibot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: Verity Disallow: / User-agent: Verity/1.1 Disallow: / User-agent: ViennaTinyBot Disallow: / User-agent: voxgov.com Disallow: / User-agent: webintegration.at Disallow: / User-agent: WebVac Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: WebZip Disallow: / User-agent: WRTNBot Disallow: / User-agent: xAI Disallow: / User-agent: xAI-Grok/1.0 Disallow: / User-agent: yacy Disallow: / User-agent: yacybot Disallow: / User-agent: YouBot Disallow: / # Prevent bots from wasting crawl budget on template/non-content pages User-Agent: * Disallow: /search Disallow: /web Disallow: /mrcontent Disallow: /webapi-public Disallow: /adperfect Disallow: /*amp.html?* Disallow: *.ece* Allow: *.ece*.pdf # Block indexing of specific sections/content at the market's request User-Agent: * Disallow: /advertise/advertorial Disallow: /advertise/advertorial/bridal Disallow: /advertise/circulars Disallow: /advertise/classifieds Disallow: /advertise/sponsored-content Disallow: /advertise/sponsored-content/sponsored-posts Disallow: /advertise/welcome Disallow: /advertise/welcome/welcome-page-lg Disallow: /advertise/welcome/welcome-page-md Disallow: /advertise/welcome/welcome-page-sm Disallow: /advertise/welcome/welcome-page-xs Disallow: /customer-service/24hoursale Disallow: /customer-service/404 Disallow: /customer-service/410 Disallow: /customer-service/contact Disallow: /customer-service/contests-promotions Disallow: /customer-service/digitalsavings Disallow: /customer-service/newsletter-signup Disallow: /customer-service/plus Disallow: /customer-service/plus/trial-offer Disallow: /customer-service/rss Disallow: /customer-service/social-media Disallow: /customer-service/subscription/special-offers Disallow: /customer-service/subscription/special-offers/cyber Disallow: /customer-service/subscription/special-offers/dbm Disallow: /customer-service/subscription/special-offers/holiday Disallow: /customer-service/subscription/special-offers/offer-a Disallow: /customer-service/subscription/special-offers/offer-b Disallow: /customer-service/subscription/special-offers/offer-c Disallow: /customer-service/subscription/special-offers/offer-j Disallow: /customer-service/subscription/special-offers/offer-k Disallow: /customer-service/subscription/special-offers/others Disallow: /customer-service/social-media Disallow: /entertainment/best-of-dfw Disallow: /entertainment/events Disallow: /entertainment/events/things-to-do Disallow: /entertainment/movies Disallow: /entertainment/music Disallow: /entertainment/nightlife Disallow: /entertainment/nightlife/sports-scene Disallow: /entertainment/performing-arts Disallow: /entertainment/restaurants/bbq Disallow: /entertainment/restaurants/brew Disallow: /entertainment/restaurants/burgers Disallow: /entertainment/tv Disallow: /fort-worth-dallas-weddings/contact-us Disallow: /living/books Disallow: /living/family Disallow: /living/family/moms Disallow: /living/family/moms/maris-moments Disallow: /living/food-drink/weekend-chef Disallow: /living/health-fitness Disallow: /living/home-garden/neil-sperry Disallow: /living/indulge Disallow: /living/indulge/indulge-food-drink Disallow: /living/indulge/indulge-getting-around Disallow: /living/indulge/indulge-homes Disallow: /living/indulge/indulge-in-focus Disallow: /living/indulge/indulge-scene Disallow: /living/indulge/indulge-style Disallow: /living/indulge/indulge-the-list Disallow: /living/keller-magazine Disallow: /living/liv-columns-blogs Disallow: /living/liv-columns-blogs/social-eyes Disallow: /living/liv-columns-blogs/social-eyes/social-eyes-photos Disallow: /living/liv-columns-blogs/social-eyes/social-eyes-submissions Disallow: /living/religion Disallow: /news/business/aviation/american-airlines Disallow: /news/business/aviation/sky-talk-blog Disallow: /news/business/barnett-shale Disallow: /news/business/biz-columns-blogs Disallow: /news/business/biz-columns-blogs/baker-ahles-kaskovich Disallow: /news/business/biz-columns-blogs/teresa-mcusic Disallow: /news/business/technology Disallow: /news/local/community Disallow: /news/local/community/arlington-citizen-journal Disallow: /news/local/community/fort-worth/fw-stock-show Disallow: /news/local/community/fort-worth/hog-blog Disallow: /news/local/community/keller-citizen Disallow: /news/local/community/mansfield-news-mirror Disallow: /news/local/community/mansfield-news-mirror/mnm-faith Disallow: /news/local/community/mansfield-news-mirror/mnm-living Disallow: /news/local/community/mansfield-news-mirror/mnm-news Disallow: /news/local/community/mansfield-news-mirror/mnm-opinion Disallow: /news/local/community/mansfield-news-mirror/mnm-sports Disallow: /news/local/community/weatherford-star-telegram Disallow: /news/local/community/weatherford-star-telegram/wt-living Disallow: /news/local/community/weatherford-star-telegram/wt-news Disallow: /news/local/community/weatherford-star-telegram/wt-opinion Disallow: /news/local/community/weatherford-star-telegram/wt-sports Disallow: /news/local/obituaries/current-obituaries Disallow: /news/nation-world/ebola Disallow: /news/politics-government/national-politics Disallow: /news/politics-government/state-politics/politex-blog Disallow: /news/politics-government/texas-senate-race Disallow: /news/special-reports Disallow: /news/special-reports/jfk Disallow: /news/special-reports/van-cliburn Disallow: /news/special-reports/van-cliburn/van-cliburn-blog Disallow: /news/traffic/gas-prices Disallow: /news/traffic/honkin-mad-blog Disallow: /news/traffic/your-commute Disallow: /opinion/letters-to-the-editor/cheers-jeers Disallow: /opinion/opn-columns-blogs/michael-ryan Disallow: /sports/boxing Disallow: /sports/college/baseball Disallow: /sports/college/big-12 Disallow: /sports/college/big-12/baylor-bears Disallow: /sports/college/big-12/oklahoma-sooners Disallow: /sports/college/big-12/texas-tech Disallow: /sports/college/big-12/university-of-texas Disallow: /sports/college/college-confidential-blog Disallow: /sports/college/football Disallow: /sports/college/sec Disallow: /sports/college/sec/texas-a-m Disallow: /sports/college/unt Disallow: /sports/college/uta Disallow: /sports/dfwvarsity/prep-baseball Disallow: /sports/dfwvarsity/prep-softball Disallow: /sports/dfwvarsity/prep-volleyball Disallow: /sports/nfl/houston-texans Disallow: /sports/other-sports Disallow: /sports/other-sports/fort-worth-cats Disallow: /sports/outdoors Disallow: /sports/spt-columns-blogs/clarence-hill Disallow: /sports/spt-columns-blogs/gil-lebreton Disallow: /sports/spt-columns-blogs/jimmy-burch Disallow: /sports/spt-columns-blogs/the-big-mac-blog Disallow: /news/state Disallow: /sports/spt-columns-blogs Disallow: /opinion/opn-columns-blogs Disallow: /homes/foreclosures Disallow: /news/corrections Disallow: /advertise/sponsored Disallow: /site-services/newsletters Disallow: /site-services/testing Disallow: /mobile/m-best-week Disallow: /mobile/m-business Disallow: /mobile/m-life-arts Disallow: /mobile/m-local Disallow: /mobile/m-opinion Disallow: /mobile/m-sports Disallow: /mobile/m-top-stories Disallow: /site-services/newsletters/business-insider Disallow: /site-services/newsletters/mom2mom-dfw-newsletter Disallow: /site-services/newsletters/sports-tip-sheet Disallow: /site-services/promos Disallow: /site-services/promos/ez-pay-contest Disallow: /welcome-page Allow: /entertainment/nightlife/party-pics Allow: /entertainment/tv/listings Allow: /news/local/community/northeast-tarrant Allow: /news/special-reports/databases Allow: /news/state/texas Allow: /sports/college/big-12/texas-christian-university Allow: /sports/nfl/dallas-cowboys Allow: /sports/nhl/dallas-stars Allow: /sports/spt-columns-blogs/mac-engel Allow: /sports/mlb/texas-rangers Allow: /sports/nba/dallas-mavericks Allow: /opinion/opn-columns-blogs/other-voices Allow: /customer-service/subscription/special-offers/digital # Sitemap files Sitemap: https://www.star-telegram.com/sitemap/story/update.xml Sitemap: https://www.star-telegram.com/sitemap/story/archive.xml Sitemap: https://www.star-telegram.com/sitemap/story/googlenews.xml Sitemap: https://www.star-telegram.com/sitemap/blogpost/update.xml Sitemap: https://www.star-telegram.com/sitemap/blogpost/archive.xml Sitemap: https://www.star-telegram.com/sitemap/blogpost/googlenews.xml Sitemap: https://www.star-telegram.com/sitemap/section/update.xml Sitemap: https://www.star-telegram.com/sitemap/section/archive.xml Sitemap: https://www.star-telegram.com/sitemap/topic/archive.xml Sitemap: https://www.star-telegram.com/sitemap/topic/update.xml Sitemap: https://www.star-telegram.com/sitemap/video/archive.xml Sitemap: https://www.star-telegram.com/sitemap/video/update.xml