web developement

search engine crawlers are very required to reach audience to your website, but if they hit continuous it will harm your website performance, If you want to prevent search engine crawler to crawl here is first and basic way to do that i.e. robot.txt

robot.txt is file which tells crawlers to what part of your website need to crawl and what part or directory not, but in written format, robot.txt follows a format to let the crawls bots to performs on your website, it also block  and allow different bots by there names

below steps let you how robot.txt will works.

1. If your website name is yourdomain.com then robot.txt should placed here

http://yourdomain.com/robots.txt

2. They are some top Bots :

Googlebot
Yahoo
bingbot
AhrefsBot 
Baiduspider 
Ezooms 
MJ12bot 
YandexBot

3. Robot.txt have two main variables 

  • User-agent - define name of bot
  • Disallow - define which directory is allow to access

example:

1) If you to allow only Googlebot

User-agent : Googlebot

Disallow: *

2) If you want allow all bots

User-agent : *

Disallow: /

3) If you disallow particular bot

User-agent : *

Disallow: *

User-agent : AhrefsBot

Disallow: /

4) If you want to prevent some directory to crawl

User-agent : *

Disallow: /scripts/
Disallow: /themes/

 


Please Comment your thoughts and feedback below and add something if you found good in anywhere to help others

Hit a like Button If you like the Post.

Many Thanks

Add new comment

Other Articles

Nodejs With Xampp

Hello Guys, We can easily connect the Nodejs with xampp by follow simple steps -

Drupal 8 mutlisite setup

In Drupal 8 Multisite setup is almost same as the Drupal 7.

Wordpress Error with php 7.1 - PHP Fatal error: Uncaught Error: [] operator not supported for strings

Hey, This is error caused by php7 and later,many application will face this error Reason behind is that the php 7 will not convert the blank variable with is define as string to array:

Send Email using smtp module with Gmail

Here is the link that helps to configure stmp module with Gmail configuration and enables drupal

Enable Drupal 8 Error Reporting

To Enable Drupal 8 Error Reporting, Put This code at top of sites/default/settings.php.

Play Pause HTML5 Video using Jquery

HTML5 video playing on any web page is very easy, in this tutorial we can show you how to add video on html and make it play and pause without using html controls.

Convert a certificate into the cer,crt,der,pem format

For different reason we need the different format of certificates for different operating system, many of sites offering you to convert the certificates in different format online but if you have b

Embed Google maps on website (Static address )

Here using this website  we can get the embed the static map on our website.

How to override different version of Jquery in Drupal 8 using theme

Drupal 8 use its own jquery version by default and if you want to use the other version of jQuery instead of default than this tutorials steps is for you, sometimes we need different version to oth

How to get your MailChimp LIST ID

Mailchimp is marketing automation platform and is provide email for marketing your product or business whatever you like, today its most trusted service providing all over the globe.

Drupal 8 Opcache Problem with xampp

;Put this code in php.ini (C:\xampp\php\php.ini) zend_extension=php_opcache.dll [opcache] ; Determines if Zend OPCache is enabled opcache.enable=1 ; Determines if Zend OPCach

Render menu on page template in custom way Drupal 8

If you want to render the menu items in custom way in page or node twig template of drupal 8 then this article is for you :  

Test payment Card Numbers For online payment

This article is for developers who develop eCommerce or payment gateway system for apps and website.

Flexslider Arrows Changed by arrow Images

Flexslider uses the arrows of its own flexslider family icons, if you want to change or replace with yours images you can do using following css.  

Prevent to Load a js file on particular page using hook_js_alter Drupal 7

Drupal provided the lots of hook to extents its functionality, using this we can prevent a js file to load on particular page using HOOK_JS_ALTER

Fix Bug Phpmyadmin [sql.lib.php] + Php7.2 + Ubuntu 16.04

I have faced this error recently , look like this    Warning in ./libraries/sql.lib.php#601

Drupal 7 content type template

Open the template.php file in your theme for editing. Look for a function called yourthemename_preprocess_page (replace the yourthemename with your theme’s name).

Reindex the Array keys

Sometimes we need to slice the array for certain values using program a code and at that time key of array is also unset. If you faced the same then you can use these options

Drupal 8 correct a word by default browser auto correct options on textarea field causing by CKEditor

If you are facing the problem to to correct the word in drupal 8 text because of copy and paste default popup on right, we have a simple trick to get over it 

Render a drupal node d8 twig template

Render a Drupal node in drupal 8 twig template and module PHP code is a bit different from Drupal 7 because Drupal 8 uses Symfony and twig format, follow the step and you will available to render a

Import and Export Drupal 8 Views, Or Import and Export any configuration Drupal 8

Import and Export configuration is the basic protocol to staging the website or production website deployment, In Drupal 7, we need the extra modules to import and export or we can use features mod

Print array or object in watchdog drupal 7

hey, For many purpose we have to print the array or object while making a custom modules or debuging any old module, we have to print the array or object to see the values are fine or not, drupal p

Scroll to top button with html and jquery

Are you searching to add a button scroll to top, then this article is for you.

Wordpress Revolution Slider error after upgrade to php 7

A recently got the error below after updating wordpress website to php 7, error causing the website blank and when the debug mode on we got this error, caused by revolution slider.

Recent Posts

[SOLVED] PHP executable not found. Install PHP 7 and add it to your PATH or set the php.executablePath setting

If you get the same error like below in VScode editor then you are the right place to 

Redirect Old Domain to New Domain via .htaccess

Usually we develop the site on developement or subdomain and when we finally mirgate the to new domain then some of thing will remain or page redirect happens in  sometimes , so In this case .htacc

Create a Flag link programmatically and render flag link in twig template drupal 8

This article is refer to generate the flag link programmatically and render in drupal 8 twig template, before in drupal 7 its was quiet easy to render or generate flag link in just a single line of

Jquery prevent enter to submit form

If you want to prevent the form submit using jquery, then this page helps you to doing this.

Code:

Play Pause HTML5 Video using Jquery

HTML5 video playing on any web page is very easy, in this tutorial we can show you how to add video on html and make it play and pause without using html controls.

Import and Export Drupal 8 Views, Or Import and Export any configuration Drupal 8

Import and Export configuration is the basic protocol to staging the website or production website deployment, In Drupal 7, we need the extra modules to import and export or we can use features mod

How to override different version of Jquery in Drupal 8 using theme

Drupal 8 use its own jquery version by default and if you want to use the other version of jQuery instead of default than this tutorials steps is for you, sometimes we need different version to oth

Subscribe to Mailchimp programmatically via api with PHP

In this tutorial we can see how to subscribe to the mailchimp subscribe list using php programmatically via api,

For this you need a mailchimp account, api key and list id 

How you can get Mailchimp API KEY

By Follow steps below, you can find the api key or create the api key at mailchimp and use it to your website to connected the site with mailchimp to collect the subscribe in fastest way.

How to get your MailChimp LIST ID

Mailchimp is marketing automation platform and is provide email for marketing your product or business whatever you like, today its most trusted service providing all over the globe.

Drupal 8 add custom menu classes to using menu.html.twig

In Drupal 8 Twig templates give extra power to drupal to add dynamic content in twig style, In this tutorial we can see how we can add the custom menu classes to menu ul li using menu.html.twig.