CRAWLING - SCRAPING, Osint Resources, Web Scrapers

A powerful API for web scraping and graph data extraction, designed for OSINT researchers looking to collect structured data from websites. https://scrapegraphai.com/

Turn any webpage into structured markdown with Firecrawl—built for LLM pipelines, but great for extracting clean page content for investigations. https://www.firecrawl.dev/

Extract all HTML comments from any URL in seconds. Give it a list of URLs, and this tool will uncover all hidden HTML comments for deeper analysis. https://github.com/rohsec/Uncomment LinkedIn Post: https://www.linkedin.com/feed/update/urn%3Ali%3Ashare%3A7249312692144918529 #osint #cybersecurity #infosec

Use Wayback URL analysis for effective recon The Time Machine leverages Wayback Machine URLs to identify historical web endpoints, aiding bug hunters and OSINT professionals. With options for endpoint vulnerability checks, including XSS and SQLi, it provides a robust approach to discovering sensitive information across time. https://github.com/anmolksachan/TheTimeMachine #va #pentest #cybersecurity #infosec

Automate asset discovery. Mantis is a command-line framework designed to automate the workflow of asset discovery, reconnaissance, and scanning. It takes the top-level domains as input, then seamlessly progresses to discovering corresponding assets, including subdomains and certificates. The tool performs reconnaissance on active assets and concludes with a comprehensive scan for vulnerabilities, secrets, misconfigurations and…

ShodanSpider automates the process of crawling Shodan results, helping OSINT researchers and security professionals gather detailed insights on exposed devices. https://github.com/shubhamrooter/ShodanSpider #OSINT #Cybersecurity #Shodan

Search public AWS S3 & Azure Blob storage by keyword – Quickly uncover exposed cloud storage for OSINT investigations. https://osint.sh/buckets/ #OSINT #CloudSecurity #Cybersecurity #Reconnaissance

Buckets GrayHatWarfare maintains a searchable database of open Amazon S3 buckets, filtering out non-relevant files while indexing millions of exposed documents, updated daily. https://buckets.grayhatwarfare.com/ #OSINT #CloudSecurity #Cybersecurity #Reconnaissance

Automate credential extraction for web security and bug hunting: find subdomains, scrape archived URLs, and grep for sensitive data. https://github.com/gigachad80/grep-backURLs #OSINT #BugBounty #Cybersecurity #Reconnaissance

Open-source no-code platform for automated web data extraction: train scraping workflows in minutes and collect data at scale—no coding needed. https://github.com/getmaxun/maxun #OSINT #WebScraping #Automation