“Spying has become just another business-travel tool, thanks to cheap, comprehensive technology, and to a soaring demand for dependable real-time information about day-to-day conditions in the world.” — Joe Sharkey, a New York Times columnist.
A few decades ago, the most significant intelligence agencies in the world with massive budgets couldn't do what you have in hand today, just with an Internet connection.
They had to use extreme spying methods, such as wiretapping, intercepting mail, and lots of social engineering, just to gather enough intel.
We take it for granted, but today we have the kind of technology that intelligence agencies, two to three decades ago, would have only dreamt of.
Like hi-resolution satellite images of the entire world, the street-view showing how a place looks physically, access to “dark web” where criminals roam, people posting updates and pictures on social media, and a lot more.
This new type of open-source intelligence gave investigators an easier way to solve their cases, they started to refer to it as OSINT.
OSINT in Cyber-security
Wikipedia describes OSINT as “data collected from publicly available sources to be used in an intelligence context.”
OSINT is a practice that anyone with Internet access can do— if you are reading this, then you can also do it.
But that is the scary thing, even not-so-innocent black hat hackers, script kiddies, or hacktivists investigate and gather enough intel to perform attacks.
The amount of information that people can find available online is mind-boggling and if they know how to connect the dots, they can figure out lots of things.
A hacker usually starts an attack from a passive reconnaissance phase.
Their first step is to get to know the target systems (or users) by gathering enough information (without actively engaging with them).
They will want to see the type of public-facing assets; they’ll go to postings on social media to know about the insides of the organization, and some would even gather information from satellite and street-view images to know more about the premises.
OSINT tools can help you identify that open-source information of your organization that is publicly available on the Internet before hackers do.
3 Categories of Tools:
Finding a Company’s Assets Information
Today, most organizations have public internet-facing infrastructure information, from cloud-assets, hosting services, domain names, and more.
The OSINT tools can help you discover that internal/external asset information and give you a detailed report.
These tools passively explore the Internet to find information publicly available about the assets of the organization.
Keep in mind that finding information does not mean active reconnaissance— in other words, these tools do not directly interact with the systems.
Discovering Sensitive Information Outside the Perimeter
The company's employees might post sensitive information on social media.
For example, an IT manager could post something on his Twitter account like, “We finally finished our long day of Apache HTTP server updates, the new version 2.4 doesn't look so bad.
Time for those drinks!”. This post is gold for hackers.
OSINT tools can look for sensitive information outside the perimeters of your network, such as social media posts or domains that do not belong to your organization.
After gathering and discovering information on public sources, the OSINT tool can aggregate all data and provide actionable insights.
It is like the “Sherlock Holmes” of the recon tools.
These tools can't only gather information, but they can also put all the pieces of the puzzle together.
They can help you determine the who, what, when, where, and why.
Best OSINT Tools and Software of 2020:
The available OSINT is vast. There are tons of sources that hackers can use to gather information.
Before a hacker finds that sensitive information about your assets, employees, or facilities, an OSINT tool can lead you in the same direction (as a hacker would) so you can remove or deal with all that information from public sources.
Recon-ng is a web reconnaissance and OSINT framework written in Python.
It can automate the process of information-gathering by thoroughly and quickly exploring the open-source information on the web.
Once it finds valuable data, it will aggregate it and present it in an integrated and easy-to-read format.
The tool comes with an interactive command-line interface that is based on modules.
The independent modules are recon, reporting, import, exploitation, and discovery.
The interface looks a lot like the popular Metasploit Framework used for exploiting systems.
The interface also comes with helpful features like command completion, database interaction, contextual help, API key management, and standardized outputs.
Recon-ng only works on Linux and requires Python 2.0 for its installation.
Open source and 100% free.
Get Recon-ng for free from the GitHub repository.
Shodan is the first search engine for interconnected or IoT devices.
While Google indexes the web, Shodan indexes everything else on the Internet.
It can find webcams, servers, routers, surveillance, traffic lights, smart TVs, fridges, vehicles, anything that is connected to the Internet.
These IoT devices are often not searchable, but Shodan devised a way to find information on them and included their open ports and vulnerabilities.
This search engine is one of the only ones that can discover operational technology found in industrial control systems.
This makes Shodan, a critical tool for cybersecurity in the Industry.
Aside from discovering IoT devices, Shodan can also be used to monitor databases to see if they have data leaks on public sites, and it can even find hidden video game servers within corporate networks.
Shodan is offered in three different editions, Freelancer ($59 /month), Small Business ($299/month), Corporate ($899/month).
No downloads. Just sign up for their service and choose a plan.
Maltego is an OSINT and computer forensics tool.
It provides interactive data mining with rich visualizations that allow efficient analysis of links.
The software is used for online investigations of the relationships between data from different sources on the Internet.
It can discover relationships between people and companies and find publicly accessible information.
For example, Maltego can find links between emails, usernames, companies, websites, and more.
It takes this information and renders a graph showing all connections and data points.
Maltego is based on a library of transforms that allow discovery from several public sources.
By default, Maltego comes with a complete list of transforms, but you can extend it by running the API from other public information sources.
The software runs in Java, and it can be installed on Windows, macOS, and Linux.
Maltego comes in a variety of editions. Maltego CE is the free community edition.
Maltego Classic ($999 for the initial cost) and Maltego XL ($1999 for the initial price).
The software is also available for large-scale server installations, and the price starts from $40,000, including training.
Get the free community edition Maltego CE by registering on their website.
theHarvester is an OSINT passive reconnaissance simple tool written in Python.
It was designed for information gathering from different public sources like search engines, the SHODAN database of internet-connected devices, or PGP key servers.
The tool is terrific for finding information lying outside the perimeters of an organization, but it can also find public-facing assets inside the perimeters.
It can discover information about subdomains, URLs, IPs, email accounts, employee names, and more.
theHarvester is relatively easy to use.
Some of the most public sources like Google, Hunter, and Baidu, are included for passive reconnaissance.
You can install some modules with an API key, such as bingapi, gitHub, and more.
Open source and 100% free.
From theHarvester for free from the GitHub repository.
5. Recorded Future
Recorded Future is an integrated threat intelligence solution.
The software can gather and analyze large amounts of threat data in real-time.
It converts all data into valuable insights with the help of ML (Machine Learning), AI (Artificial Intelligence), and NLP (Natural Language Processing).
Among its many functionalities, Recorded Future can perform passive reconnaissance with the help of the Threat Intelligence Platform (TIP).
Leveraging ML, AI, and NLP, Recorded Future can collect and aggregate data from publicly available sources, such as domain registrations, social media profiles, third-party websites, and more.
It works automatically and sends real-time alerts when it finds data leaks, such as credentials, typosquat domains, code leaks, bank identification numbers, brand-talk in the dark web, and more.
Recorded Future comes in three different editions, Express, Core, and Advanced. To learn more pricing, contact them.
Metagoofil is a free passive recon metadata collector, written in Python.
It is used for extracting information from documents like pdf, doc, Xls, ppt, ODP, and ods that are found on the target’s webpage or any other public site.
The tool uses Google to find the documents, then downloads them to the local disk, and extracts all metadata.
Metagoofil analyzes the metadata of these documents and collects a good amount of information.
It can find sensitive information like usernames, real-names, software versions, emails, paths/servers.
Hackers can use Metagoofil to gather username information and perform easier brute-force attacks.
It can also show a hacker the file paths that can reveal OS, network names, shared resources, and more.
Free and open-source.
Metagoofil for free from the GitHub repository.
Searchcode is a unique type of search engine that looks for intelligence inside free source code.
Developers can use Searchcode to identify problems related to the accessibility of sensitive information in the code.
The search engine works similar to Google, but instead of indexing web servers, it looks for information between the lines of code of running apps or in apps in development.
The search results can help a hacker identify usernames, vulnerabilities, or flaws in the code itself.
Searchcode looks for code in repositories of code from GitHub, Bitbucket, Google Code, GitLab, CodePlex, and more.
You can also filter different types of language.
- Search using special characters
- Filter by programming language
- Filter by the repository
- Seek through source code
It is a free web-based search engine.
GHDB (Google Hacking Database) or often referred to as Google Dorks, is a database of search queries for Google that aims to find publicly available information.
The victims accidentally post sensitive information on the web, such as web consoles without security, open ports, login portals, sensitive directories, open webcams, files containing username information, and anything that is unintentionally exposed on the Internet.
The community in Google Dorks posts a series of advanced Google search strings every day.
An example of a Google dork query can be “intitle:index.of “users.db” that exposes files with users, or “intitle: “webcamXP 5″ -download” that shows all webcams with brand XP5 that are open for viewing.
GHDB is intended for pen-testers at the information-gathering phase or OSINT.
If a pen-tester knows how to craft sophisticated queries in Google or can find the right one in GHDB he/she can find security holes in the configuration or coding of about anything.
It is a database of queries; there is nothing to download.
SpiderFoot is an open-source reconnaissance tool.
It is often called the fingerprinting with the most extensive OSINT collection.
The tool can automatically send queries to more than 100 public sources and collect intelligence on IP addresses, domain names, web servers, email addresses, and more.
The software is written in Python.
To start with SpiderFoot, specify the target and choose among the hundreds of different fingerprinting modules.
Examples of the SpiderFoot modules can be “sfp_arin.py” that queries ARIN registry for contact information or “sfp_crt.py” which gathers hostnames from historical certificates in crt.sh.
Once you choose the modules, SpiderFoot will automatically collect information and build a report.
SpiderFoot is available for Windows and Linux.
Free and open-source
10. OSINT Framework
Last but not least is the OSINT Framework. If you haven’t found your perfect OSINT tool yet (or if it is not here on this list), the OSINT Framework will guide you in the right direction.
The OSINT framework is not specifically software, but a collection of tools that make your OSINT tasks much easier.
The OSINT Framework presents the information in a web-based interactive mind map that organizes the information visually.
The framework is popular among pen-testers and cyber-security researchers that are looking for tools on specific areas of information-gathering and reconnaissance.
With this framework, you can browse through different OSINT tools which are filtered by categories.
For example, some categories are the username, email address, geolocation/maps, dark web, search engines, transportation, public records, and a lot more.
It is a free web-based framework. There is nothing to download.
Final Words & Conclusion
OSINT does not only help you enforce the cyber-security standpoint of an organization but It can also help secure any area, from the company premises perimeter, improve anonymity on the web, remain safe from social engineering tricks, and even avoid terrorism.
If you know how to use Google Earth, its historical satellite imagery, and street views, you are already using open-source intelligence.
Knowing what information is available on public sources that can put you or your organization in risk is the key to block that critical passive recon phase and avoid an attack altogether.
Using OSINT tools and software shown here, take it a step forward, and automatically scan the publicly available information for you.
And even aggregate data and generate reports!!!