Where is Robots.txt file located

[ibrendan]

I would like to edit my robots.txt file manually. Where does the automatically generated file get stored. I've looked in my root directory and can't seem to find it.

 

[Mar]

I would like to edit my robots.txt file manually. Where does the automatically generated file get stored. I've looked in my root directory and can't seem to find it.

Yes it is the root directory, public_html folder.

 

[Direct Webstore]

And if you don't have one, you will have to make one. Just create a normal txt file and rename it. I use All in One SEO and it has a built in robots.txt.

I'm guessing you want to use my suggestion to block Google from indexing images hosted on Aliexpress, right?

Here is a copy of my robots.txt file


User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: *
Disallow: /https://ae01.alicdn.com/*.jpg$
Disallow: /https://ae01.alicdn.com/*.jpeg$
Disallow: /https://ae01.alicdn.com/*.png$
Disallow: /https://ae01.alicdn.com/*.gif$

Insert URL to sitemap here.


Just paste that into a txt file, rename it to "robots.txt" and upload it to your root directory, like Mar showed above.

You should add the URL to your sitemap too. It would look something like this ...
Sitemap: https://yoursite.com/sitemap.xml.gz

Check the Alidropship settings for the URL. I'm not sure, but I think it's in the SEO section. Can't remember, I use Woo now,

 

[ibrendan]

And if you don't have one, you will have to make one. Just create a normal txt file and rename it. I use All in One SEO and it has a built in robots.txt.

I'm guessing you want to use my suggestion to block Google from indexing images hosted on Aliexpress, right?

Here is a copy of my robots.txt file


User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: *
Disallow: /https://ae01.alicdn.com/*.jpg$
Disallow: /https://ae01.alicdn.com/*.jpeg$
Disallow: /https://ae01.alicdn.com/*.png$
Disallow: /https://ae01.alicdn.com/*.gif$

Insert URL to sitemap here.


Just paste that into a txt file, rename it to "robots.txt" and upload it to your root directory, like Mar showed above.

You should add the URL to your sitemap too. It would look something like this ...
Sitemap: https://yoursite.com/sitemap.xml.gz

Check the Alidropship settings for the URL. I'm not sure, but I think it's in the SEO section. Can't remember, I use Woo now,

Yes, that's what I have in mind. So here's the thing: I do have robots.txt file (it shows up when I type the URL and also on search console), but I can't find it in C-Panel. Weird. I'm thinking alidropship addon creates a virtual file. My ther sites that don't use alidropship all have a physical robots.txt file which I created using Yoast. What's wierd is that yoast just throws up an error when I try to make a new robots file or edit the existing one.

 

[Direct Webstore]

I'm thinking alidropship addon creates a virtual file.

It does. And from memory (Not using Original anymore) there's no way to edit it as far as I know. Maybe I'm wrong.

I use All in One SEO and can edit the virtual robots.txt it generates. You can also do that with Yoast (https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/ )

All in One SEO is Yoast's biggest competitor and out of the two I think its' much better for e-commerce sites. Yoast is mainly aimed towards blogs. (You have to buy an extra plugin to use it with Woo so It wouldn't be any good for the ADS Original plugin either). All in One doesn't need any extra plugins for e-commerce.

You may want to consider using All in One SEO as Alidropship's builtin SEO is pretty basic. I use Woo now and am not sure if it works with the Original plugin's products ... but it's easy enough to try. After installing it ... just edit a product and see if it's added it's extra fields there.

It also generates sitemaps, has a "bad bot" blocker, handles social META tags and other goodies.

If you do decide to use it ... don't forget to disable the built in SEO.

 

[Direct Webstore]

I just had a thought. The Original plugin's virtual robots.txt may be stored in the Options table in the database. If so, you could edit it there via cPanel>PHPMyAdmin. Anyone know if it IS stored there?

You could just have a look I guess ... but the Options table has up to 20 or more pages to search through.

 

[ibrendan]

I just had a thought. The Original plugin's virtual robots.txt may be stored in the Options table in the database. If so, you could edit it there via cPanel>PHPMyAdmin. Anyone know if it IS stored there?

You could just have a look I guess ... but the Options table has up to 20 or more pages to search through.

Hey thanks for the help - I'll give it a shot. I'll let you know if I get somewhere...