OSINT stands for open source intelligence and refers to the practice of collecting data from free sources that are available to the general public. As information becomes more available from a vast number of sources, skilled researchers can often find nearly any type of data they’re looking for, provided they know where to look. These sources include both public and private databases that hackers, journalists, spies, and ordinary people all use to do the work of collecting information every day.
Much of this information simply isn’t able to be retrieved through search engines and requires knowledge of where the right database is located to get the right answers. Fortunately, there are a multitude of tools that both beginners and seasoned investigators can take advantage of to use OSINT sources while conducting research.
Get the Free Pen Testing Active Directory Environments EBook
In this article, we'll cover the following:
- What should I do for OSINT to protect my organization?
- How to conduct an OSINT investigation
- OSINT Frameworks & Tools
- OSINT Techniques
- OSINT, SOCMINT, & HUMINT
- OSINT on Phone Numbers
- OSINT on Names
- OSINT on Businesses
- OSINT on Websites
- Resources for OSINT research
- Dangers of OSINT Research
- OSINT Regulations
- Examples of OSINT Investigations
- Who Uses OSINT?
What Should I Do for OSINT to Protect My Organization?
Every organization creates data, and much of this data may be public if it’s produced or recorded by a local or state government. Aside from business data, technical data from registration or development of websites or products can expose information about the internal workings of an organization that might otherwise be impossible to find.
The first thing to know about OSINT is that it cuts both ways, and the same data used to learn about a competitor can also be used to infer information about your company or organization. It can be incredibly rewarding to discover what information about you might be more public than you expect and understand how an outsider could know internal details about your organization that make it easier to attack.
How Do I Conduct an OSINT Investigation?
An OSINT investigation starts with answering an answerable question. Because there is no shortage of data out there, it’s easy to get lost in all the noise if what you’re looking for isn’t clear and answerable in the first place. An example of this might be questioning “is New York safer than Los Angeles?” versus “Have reports of police misconduct increased in Los Angeles county since 1999?” One question calls for an opinion and is difficult to answer, where the second can be easily answered by finding the right record holder.
After establishing the question to answer, the next step is to identify the most likely owner of the best quality data. Primary source information like government or corporate data are the best sources, followed by information created by trade-based organizations or non-governmental agencies (NGO’s) related to the industry or trade your question is related to. Finally, third-party aggregators of data can provide useful links back to a primary source of data, although this “tertiary” data cannot be used as evidence.
Generally, an investigation will develop an understanding of a subject by asking answerable questions and using the best quality answers to paint a picture. While third-party reports like a newspaper quoting another source are useful, it’s important to be critical of the sources you find during an OSINT investigation. Because there is so much information, much of it inaccurate or published by unreliable sources, OSINT researchers must be extremely critical of the sources of data used, preferring only to use verified primary source data to draw conclusions.
OSINT Frameworks & Tools
Because OSINT covers so many different types of data, there are many different types of investigations that can be conducted. This ranges from social media investigations using free tools to comb through vast amounts of Tweets, to geospatial investigations using satellite imagery to locate where photos were taken.
There are many different resources for investigators looking for more OSINT tools, including the osintframework.com website which lists a huge number of data sources for different types of OSINT investigations. There are also many fantastic free OSINT tools on Github, many of them curated into this “Awesome-OSINT” list.
What OSINT Techniques Are There?
OSINT tactics can be divided into active and passive techniques, with active tactics involving some sort of actual contact with the target, and passive tactics avoiding any contact with the target. Active techniques always involve some small risk of the target detecting you are investigating them, whereas passive tactics usually involve querying a database maintained by someone else, and usually do not involve any risk of being detected.
An active OSINT tactic could be as simple as scanning a website or web server owned by a target, or registering to download a competitor’s product catalog. While this small contact probably won’t blow your investigation, matching a download to your organization’s IP address could tip off a subject that your organization is investigating. Comparatively, a passive technique like using a search engine like Shodan to examine services a company is running without scanning them yourself would be nearly impossible for the target to detect.
How Does OSINT Relate to SOCMINT & HUMINT?
HUMINT, or human intelligence, is old-school intelligence collection done using human sources to collect information. With the explosion of data available to OSINT investigators, HUMINT has been supercharged by the ability to learn nearly everything about the person you need to interview before you even meet them. This gives the investigator a huge advantage when interviewing a source, and can make HUMINT a natural and extremely valuable add-on to any OSINT investigation.
OSINT is the secret power behind many HUMINT collection wins, providing the context to get a person to do something they’re not supposed to do. Often with the right OSINT information, it’s possible to get someone who is aware they hold sensitive information to either grant you access because they think they are supposed to, or because they think you already know the information they have access to.
By taking advantage of the information about a company’s employees on social media as well as internal policies leaked by documents easily found through passive search techniques, it’s simple to understand who has access to the information an investigator needs. This can cut down the time needed to get answers substantially by getting the exact right answer from the right source earlier.
How Do I Find OSINT on Phone Numbers?
Phone numbers are often linked to individuals through phone books, social media accounts, and data leaked by businesses who have worked with someone before. When phone numbers turn up in an OSINT investigation, third-party aggregators like www.opencnam.com, thatsthem.com, and truecaller.com can begin matching the number to businesses or people.
Once you have an initial match, you’ll want to pull details like addresses, names, and social media names to expand the scope of your discovery to other web accounts or documents related to your target. Matching each of these details to even more information is often how a phone number can lead to a business license or other more concrete details.
How Do I Find OSINT on Names?
Names often appear in documents related to business filings, so the best quality results for name searches related to business owners or powerful people will often be on websites like lilsis.org or OpenCorporates. For normal people, third party aggregators will often attempt to stitch together lots of information about people by name, especially if you know the general area they live in.
While third-party aggregators can’t be used as answers to your investigation, they often point to better sources of data. To start an investigation, sources like pipl.com, beenverified.com, and peekyou.com can get you started, but should only be used to point to other accounts or searchable details like past addresses or phone numbers.
How Do I Find OSINT on Businesses?
Businesses are some of the easiest entities to find information on, as they need to register a lot of paperwork with public entities to exist. This paperwork is often entirely public and searchable, giving investigators a trove of information full of details which help to expand an investigation in the early stages.
To get started, it’s best to search the secretary of state website of whichever state a particular organization does business in. This should provide a starting point of data, but can often be expanded on by searching additional states as well. Because there are tax and legal incentives to register in certain states in the US, it’s often worthwhile to check the secretary of state databases in Nevada, Delaware, California, and Wyoming as well.
After locating primary source data, third party aggregators like OpenCorporates give you the ability to search all secretary of state databases indexed by the service, often locating valuable primary source information somewhere you might have missed. Lilsis.org is also a good source of information about business people including stock options and tax information.
How Do I Find OSINT on Websites?
Websites represent a lot of information about a business, both technical and in the way the website has changed messaging and branding over time. For OSINT investigations, websites are often looked at for their technical information, like who registered it, what servers are in use, and what software is maintaining it. This can be done through services like Shodan, which allow you to profile an organization’s technical infrastructure without actually scanning it yourself.
Another source of information is the actual contents of the website, which can include files left unintentionally public, or information that may have been removed from the website in the past. Tools like the Internet Archive can show an investigator how the website has changed over time by comparing snapshots taken years earlier, often pointing to organizational changes or showing who has left or joined the company and when.
Google Dorking is also a powerful way of finding files and other details left open to the internet by accident. By structuring Google search queries to look for certain types of web pages or files, it’s easy to find any parts of a target’s website that might be leaking confidential information.
What Resources Are There For OSINT Research?
There is a huge community around OSINT investigations that loves to share tricks and techniques for investigations. In particular, Mike Bazell hosts an amazing website and podcast about OSINT that constantly points to new and innovative investigation techniques, and both Twitter and Github are full of new OSINT tools being developed and released by the OSINT community.
What Dangers Are There in Conducting OSINT Research?
While OSINT is based on public data, there are always risks when doing any sort of investigation that involves either making direct contact with the subject, or a third party who may sell your search to the third party directly. In the example of LinkedIn, you may find a link to a subject’s LinkedIn profile and forget to sign out of your own, prompting LinkedIn to offer the target the information that you were looking at their profile as part of their product.
This kind of contact can blow an investigation, as can accessing a web resource belonging to a target from your organization’s IP address. If you work for an organization that does not want to be identified while investigating a target, you must always use a VPN to hide the IP address of the place that you’re working from, or risk making your attention to any particular part of the target’s infrastructure obvious.
Is Data Collected in an OSINT Investigation Regulated Under Privacy Laws Like GDPR, CCPA?
Thanks to the many loopholes provided by CCPA, OSINT investigations are generally not something California residents need to worry about. GDPR, however, is more strict with these rules, meaning it’s important to take basic steps during an investigation to ensure you don’t expose personal information of a subject and that you’re storing it properly. This means using encryption to store your investigation notes and not leaving them where they can be accessed by a third party.
What Are Some Examples of OSINT Investigations?
Bellingcat has produced some of the most dramatic OSINT investigations in recent memory, in particular blowing the lid off denials by the Russian government of involvement in various operations by digging through databases to prove their involvement directly.
Memorable examples include tying Russian spies to their secret employment by digging through car registration databases (https://www.bellingcat.com/news/2018/10/04/305-car-registrations-may-point-massive-gru-security-breach/comment-page-5/), and linking social media photos to prove Russian soldiers were operating in areas the government claimed they weren’t. Many OSINT investigations cover topics like war crimes in areas too remote or dangerous to access remotely and was used extensively in the investigation into the downing of Malaysian Airlines Flight 17 (MH17). (https://www.bellingcat.com/news/uk-and-europe/2017/12/08/russian-colonel-general-delfin/)
Who Makes Use of OSINT Reports?
OSINT is a critical part of both public and private intelligence, arming businesses, governments, and individual investigators with a vast amount of high-quality information to base and make decisions on. Whether conducting an investigation for research, business intelligence, or threat analysis, OSINT can allow anyone to have access to some of the best available data in the world.
