CRAWLING - SCRAPING, Osint Resources

AI-powered subdomain enumeration tool with local LLM analysis via Ollama – 100% private, zero API costs https://github.com/Vyntral/god-eye

OSINT tool that harvests metadata from public documents (PDF, DOC, XLS, PPT, etc.) found on target websites via Google searches. Extracts usernames, email addresses, server paths, and internal naming conventions—ideal for pre-engagement reconnaissance and password spray list generation. https://github.com/laramies/metagoofil

An all-in-one Shodan & ZoomEye supported tool to search, browse, preview and dump data leakage across 20+ services. Pulls real exposure straight from the sources instead of guessing. Drop it into your workflow and watch it surface leaks you won’t find anywhere else. https://github.com/GainSec/LeakScope

Web-based utility to quickly scan a target host, detect open and unusual ports, and check running services. Utilizes tools like Nmap and Unicornscan to provide detailed results for vulnerability and security testing, accessible from any device. https://portscanner.online/

Open-source HTTP toolkit for security research. Designed as an alternative to Burp Suite Pro, it’s useful for infosec, bug bounty, and even large-scale web scraping in OSINT workflows. https://github.com/dstotijn/hetty

Advanced search engine for Mega.nz, offering access to over 9 million indexed files, including educational materials, software, movies, and more. It employs intelligent crawling and categorization for efficient discovery. https://meawfy.com/

A powerful API for web scraping and graph data extraction, designed for OSINT researchers looking to collect structured data from websites. https://scrapegraphai.com/

Turn any webpage into structured markdown with Firecrawl—built for LLM pipelines, but great for extracting clean page content for investigations. https://www.firecrawl.dev/

Extract all HTML comments from any URL in seconds. Give it a list of URLs, and this tool will uncover all hidden HTML comments for deeper analysis. https://github.com/rohsec/Uncomment LinkedIn Post: https://www.linkedin.com/feed/update/urn%3Ali%3Ashare%3A7249312692144918529 #osint #cybersecurity #infosec

Use Wayback URL analysis for effective recon The Time Machine leverages Wayback Machine URLs to identify historical web endpoints, aiding bug hunters and OSINT professionals. With options for endpoint vulnerability checks, including XSS and SQLi, it provides a robust approach to discovering sensitive information across time. https://github.com/anmolksachan/TheTimeMachine #va #pentest #cybersecurity #infosec