Gem Indexer

Gem Indexer is a web bot that builds an index of vintage and secondhand clothing products from online sources. It is built to respect robots.txt standard and use resources efficiently.

If you want to control or limit Gem Indexer access to your website, you have multiple options:

• Contact me
• Edit your server’s robots.txt
• Add/edit Robots meta tags on your web pages
• Add/edit X-Robots-Tag HTTP headers to your web pages

You can learn about robots access from:

• Robots.txt standard
• Google: Robots.txt Specifications
• Google: Robots meta tag and X-Robots-Tag HTTP header specifications

Gem Indexer's short name is "GemIndexer" and full user-agent string is: "Mozilla/5.0 (compatible; GemIndexer/1.0; +https://gem.app/indexer)"