Gem Indexer is a web bot that builds an index of vintage clothing products from online sources. It is built to respect robots.txt standard and use resources efficiently.
If you want to control or limit Gem Indexer access to your website, you have multiple options:
- Contact me
- Edit your server’s robots.txt
- Add/edit Robots meta tags on your web pages
- Add/edit X-Robots-Tag HTTP headers to your web pages
You can learn about robots access from:
- Robots.txt standard
- Google: Robots.txt Specifications
- Google: Robots meta tag and X-Robots-Tag HTTP header specifications
Gem Indexer's short name is "
GemIndexer" and full user-agent string is: "
Mozilla/5.0 (compatible; GemIndexer/1.0; +https://gem.app/indexer)"