Crawler Policy

Guidelines for AI Systems, Language Models, and Automated Crawlers

Website: newatoms.com

Organization: SHFTX Tech Private Limited

Last Updated: August 4, 2026

Policy Version: 1.0

1. Overview & Purpose

This policy defines how AI systems, language models, crawlers, and automated agents may access, crawl, and use content from newatoms.com. We welcome responsible access from AI systems and encourage their use in training and discovery while protecting sensitive information.

Our Position on AI: We believe AI systems create opportunities for better product discovery and content accessibility. We permit access to our public content for training and research purposes, with clear guidelines below.

2. Permitted AI Systems & Crawlers

The following AI systems and crawlers are explicitly permitted and encouraged to access newatoms.com:

AI System / Crawler	Organization	Primary Purpose	Access
Claude Web	Anthropic	Training & Research	✓ Allowed
GPTBot	OpenAI	Training & Search	✓ Allowed
Google Bard / Gemini	Google	Search & Training	✓ Allowed
PerplexityBot	Perplexity AI	Search & Discovery	✓ Allowed
Meta-Externalagent	Meta	Training & Research	✓ Allowed
Copilot	Microsoft	Training & Search	✓ Allowed
AppleBot-Extended	Apple	Training & Search	✓ Allowed
Googlebot	Google	Search Indexing	✓ Allowed
Bingbot	Microsoft	Search Indexing	✓ Allowed
Other AI Crawlers	Various	Research & Training	✓ Generally Allowed

3. Content Access Permissions

✅ Content AI Systems CAN Access

AI systems are freely permitted to access, crawl, and use the following content for training, research, and discovery purposes:

Product Pages – All product listings, descriptions, specifications, prices, images, reviews, and ratings
Product Categories & Collections – Category pages, brand pages, product collections
Blog & Articles – Educational content, product guides, how-to articles, industry insights
FAQ Pages – Frequently asked questions and answers
Brand Pages – Information about brands carried, brand descriptions, brand collections
Public Information – About page, company mission, team information (public profile)
Structured Data – Schema.org markup, JSON-LD, open graph data
Sitemaps & Feeds – XML sitemaps, RSS feeds, Atom feeds

❌ Content AI Systems CANNOT Access

AI systems are strictly prohibited from accessing, crawling, or scraping the following:

User Accounts & Profiles – /my-account/, user dashboards, personal profiles
Customer Orders & History – /order-history/, customer purchases, order details
Payment Information – /checkout/, /cart/, payment pages, payment methods
Personal Data – Customer names, emails, phone numbers, addresses, personal information
Admin Areas – /wp-admin/, /admin/, backend systems
Login Pages – /wp-login.php, authentication systems
Private Content – /private/, /confidential/, internal communications
API Endpoints – /wp-json/, /api/, internal APIs (except documented public APIs)
Searchable Query Results – /?s=, ?search=, ?query= (paginated search results)
Session Files – Session data, temporary files, cache files

4. Commercial Use & Attribution

Training Data Usage

✅ PERMITTED: AI systems may use public content from newatoms.com for training machine learning models, language model training, and algorithm development.

Attribution Requirements

When using content from newatoms.com, AI systems should:

Provide attribution when content is quoted or substantially used
Include a link to the source page on newatoms.com
Respect original authorship
Maintain context and accuracy of information

Example: “Product information from newatoms.com” with link to product page

Commercial Restrictions

Content Scraping for Resale: Scraping and reselling newatoms.com content is prohibited. AI systems may not use our content to:

Create competing product databases
Sell aggregated data to third parties
Build commercial directories using our content
Create derivative works for commercial sale without permission

Permitted Commercial Uses

✓ Using product information in AI recommendations (with attribution)
✓ Including product data in AI search results (with attribution and link)
✓ Analyzing products for comparison or research
✓ Using content for training commercial AI models
✓ Displaying product information in aggregator tools (with attribution)

5. Crawling & Rate Limits

Recommended Crawling Behavior

Parameter	Recommended Value	Reason
Crawl Delay	1 second minimum	Server resource management
Request Rate	Max 60 requests/minute	Prevent server overload
Concurrent Requests	Max 5 simultaneous	Maintain site performance
Crawl Frequency	Daily or less	Efficient indexing
Preferred Crawl Time	02:00-06:00 IST	Off-peak hours

Feed & Sitemap Crawling

AI systems are encouraged to check these URLs regularly for new content:

https://newatoms.com/feed/ – RSS feed (check daily)
https://newatoms.com/sitemap.xml – XML sitemap (check weekly)
https://newatoms.com/sitemap-products.xml – Product sitemap (check weekly)
https://newatoms.com/feed/atom/ – Atom feed (check daily)

6. Specific Guidelines by AI System Type

🤖 Large Language Models (LLMs)

Models: Claude, GPT-4, Gemini, etc.

Allowed Uses: Training data, knowledge base augmentation, product information in responses

Requirements: Attribute product information, provide links when possible

Rate Limits: 1 request/second, 60/minute maximum

Contact: If crawling at scale, notify info@shftx.tech

🔍 AI Search Systems

Systems: Perplexity, Microsoft Copilot, Google Search, etc.

Allowed Uses: Including products in search results, recommendations, price comparisons

Requirements: Provide clear attribution, link to source product page, accurate information

Rate Limits: 1 request/second, crawl daily

Contact: Support ongoing by notifying us of integration

🎯 Recommendation Systems

Systems: AI recommendation engines, product discovery tools

Allowed Uses: Product data analysis, recommendation generation, comparison tools

Requirements: Attribute source, maintain data accuracy, respect product ownership

Rate Limits: 1 request/second, 100 requests/minute

Contact: For commercial partnerships, contact neel@shftx.tech

💾 Training & Research

Academic & Commercial ML Model Training

Allowed Uses: Training datasets, ML model development, research papers (with citation)

Requirements: Cite newatoms.com as data source, respect intellectual property

Rate Limits: Bulk downloads allowed, just notify us

Contact: For large-scale research access, contact neel@shftx.tech

🔄 Web Crawlers & Agents

General web crawlers, price tracking bots, data aggregators

Allowed Uses: Product indexing, price monitoring, data aggregation (with limits)

Restrictions: Cannot scrape for resale; must attribute source

Rate Limits: 1-2 requests/second, crawl weekly only

Contact: Identify your bot in User-Agent header

7. How to Comply with This Policy

For AI System Developers

Configuration Steps:

Check robots.txt – Located at https://newatoms.com/robots.txt
Read llms.txt – Located at https://newatoms.com/llms.txt
Read ai.txt – Located at https://newatoms.com/ai.txt
Check ai.json – Located at https://newatoms.com/.well-known/ai.json
Identify Your Bot – Set User-Agent to include bot name and contact info
Respect Rate Limits – Follow crawl delay guidelines (1 second minimum)
Check Feeds First – For updates, check /feed/ before crawling pages

User-Agent Best Practices

Set your User-Agent header to clearly identify your bot:

User-Agent: MyAIBot/1.0 (+http://mysite.com/bot; contact@mysite.com)

Good Example: PerplexityBot/1.0 (+https://perplexity.ai/)

Bad Example (Don’t use): Mozilla/5.0 (Windows NT 10.0; Win64; x64) (misleading)

8. Data & Intellectual Property

Content Ownership

Ownership: All content on newatoms.com is owned by SHFTX Tech Private Limited or its brand partners.

License: Content is provided under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC-BY-NC-SA-4.0) for permitted use.

Product Images & Media

Restrictions: Product images may not be:

Republished on competing sites without permission
Used in commercial products without attribution
Resold or redistributed for profit
Modified or rebranded without permission

Brand Trademarks

Brand names mentioned (OHONESTY, AILKIN, OFUCA, HOMEMO, QIBOX, Unbreakcable, Romitaz, Basesailor, etc.) are trademarks of their respective owners. Use in AI results should be accurate and non-misleading.

9. Prohibited Activities

AI systems are strictly prohibited from:

Accessing password-protected or admin areas
Attempting to circumvent authentication systems
Scraping personal customer information
Extracting payment or financial information
Accessing user accounts or private data
Submitting harmful payloads or injection attacks
Impersonating SHFTX Tech or official channels
Using content to create competing product databases
Bulk scraping without identification or notification
Distributing content claiming original authorship
Using scraped data for spam or abuse
Accessing rate-limited or restricted endpoints

10. API & Integration Guidelines

Public APIs

Currently, we provide the following public integration points:

RSS Feed: https://newatoms.com/feed/
XML Sitemap: https://newatoms.com/sitemap.xml
Product Data: Product pages with structured data (JSON-LD)
Feeds & Syndication: Atom feeds, RSS feeds available

For commercial API access or data partnerships, contact: neel@shftx.tech

11. Monitoring & Enforcement

How We Monitor Compliance

We actively monitor crawler activity through:

Server access logs and traffic analysis
User-Agent identification and verification
Rate limiting and anomaly detection
Content usage tracking and attribution checking
Legal/copyright monitoring services

Violations & Consequences

If an AI system violates this policy, we may:

Issue Warning: Send notification via User-Agent contact or organization
Rate Limiting: Reduce request rate or implement stricter limits
Blocking: Block the bot’s IP address or User-Agent
Legal Action: Pursue legal remedies for serious violations (copyright, data theft, etc.)

12. FAQ – Common Questions

Q: Can Claude/GPT use newatoms.com content in training?

A: ✓ Yes. Public product content may be used for training LLMs. We request attribution when possible.

Q: Can I scrape product prices for a price comparison site?

A: ✗ No. Bulk scraping for resale or competitive advantage is prohibited. Use RSS feed for updated info instead.

Q: Can I use newatoms.com images in my AI-generated content?

A: ✓ Yes, with attribution. Link back to the source product page and credit newatoms.com.

Q: What’s the best way to get product updates?

A: ✓ Check RSS feed. Subscribe to https://newatoms.com/feed/ for daily updates without heavy crawling.

Q: Can I crawl user reviews and ratings?

A: ✓ Yes. Public reviews on product pages may be accessed. Don’t access private customer accounts.

Q: Am I allowed to access the checkout or my-account sections?

A: ✗ No. These are restricted. Only access public product and content pages.

Q: Can I use this data to train a competing e-commerce AI?

A: ✗ No. Training data may be used for LLMs and general AI, but not for competitive product databases.

Q: How should I identify my crawler/bot?

A: Set your User-Agent to include your bot name, version, and contact info: MyBot/1.0 (+https://mysite.com; contact@mysite.com)

Q: What crawl rate should I use?

A: Minimum 1 second between requests, maximum 60 requests/minute. For bulk crawling, contact us first:neel@shftx.tech

Q: My crawler is being rate-limited. What should I do?

A: You may be crawling too aggressively. Reduce request rate, check for compliance, or contact us to discuss your use case.

13. Contact & Support

How to Reach Us

General Questions About This Policy

Email: info@shftx.tech

Subject: “[CRAWLER POLICY] Your Question”

Commercial Partnerships & API Access

Email: neel@shftx.tech

Subject: “[PARTNERSHIP] AI Integration Request”

Copyright & IP Concerns

Email: legal@shftx.tech

Subject: “[COPYRIGHT] Content Violation Report”

Security & Abuse Reports

Email: security@shftx.tech

Subject: “[SECURITY] Crawler Abuse Report”

Technical Support

Email: support@newatoms.com

Response time: 24-48 hours

14. Policy Changes & Updates

This policy may be updated periodically to reflect changes in technology, business practices, or legal requirements. Updates will be posted on this page with updated date.

Notification of Changes

Significant changes will be announced at least 30 days in advance
Check this page regularly for updates
Subscribe to newatoms.com RSS feed for announcements

15. Related Policies & Files

This policy works alongside several technical files that implement these guidelines:

robots.txt – Machine-readable crawler directives (view)
llms.txt – LLM-specific permissions (view)
ai.txt – AI system configuration (view)
ai.json – Structured configuration (view)
security.txt – Security policy (view)
Privacy Policy – Full privacy policy
Terms of Service – Full terms of service

16. Summary of Key Points

✅ What AI Systems CAN Do

Access and crawl public product pages
Use content for training AI/ML models
Include products in search results (with attribution)
Analyze products for recommendations
Include product information in responses (with attribution)
Check feeds for new content updates

❌ What AI Systems CANNOT Do

Access user accounts or customer data
Scrape payment or private information
Create competing product databases
Resell or redistribute content without permission
Bypass authentication or security measures
Crawl faster than rate limits allow
Impersonate official SHFTX Tech channels

⚡ Quick Reference

Crawl Delay: 1 second minimum
Rate Limit: 60 requests/minute max
Preferred Crawl Time: 02:00-06:00 IST
Attribution: Required for commercial use
Contact: info@shftx.tech

Last Updated: August 4, 2026 | Version: 1.0