Navigating the Hidden Wiki on GitHub

Comparison of Major Hidden Wiki GitHub Repositories

What is the Hidden Wiki on GitHub

The Hidden Wiki on GitHub refers to a collection of repositories that replicate and archive the content of the original Hidden Wiki, which operates as a directory of links to .onion sites on the dark web. The actual Hidden Wiki, first noted in 2007, functioned as a Tor hidden service allowing anonymous editing and serving as a resource for navigating the dark web¹. Over time, multiple iterations and mirrors of the Hidden Wiki emerged, particularly after the original site was compromised in 2014 and again in 2014 during law enforcement operations¹.

Developers mirror or archive Hidden Wiki content on GitHub for various reasons. Primarily, GitHub offers a public platform that is easily accessible and indexable, allowing users to contribute, update, and share resources in a collaborative manner. This contrasts sharply with .onion sites, which require the Tor network for access and are not indexed by traditional search engines. The repositories on GitHub enable easier access to curated lists of links and tools without the inherent anonymity challenges presented by dark web sites.

Examples of notable GitHub repositories include:

• notthehiddenwiki/NTHW: Created in January 2024, this repository bills itself as the largest collection of links related to cybersecurity, accumulating 1,914 stars and 282 forks from 60 contributors².
• thehiddenwiki2023: Established on April 30, 2023, this repository contains HTML and CSS code for verified onion links, with updates provided regularly⁶.
• TorBot: An open-source crawler designed to automate the extraction of data from .onion sites³.

These repositories exemplify the transition of Hidden Wiki content into a more accessible format while maintaining the spirit of the original link directory.

Major Hidden Wiki GitHub Repositories

Several active GitHub repositories continue the legacy of the Hidden Wiki, offering a range of resources from link directories to OSINT tools. Below are notable examples:

notthehiddenwiki/NTHW
Created on January 11, 2024, this repository serves as a comprehensive link directory for cybersecurity resources. It has garnered 1,914 stars and 282 forks, with contributions from 60 developers. The repository is actively maintained, allowing users to add new links, update existing ones, and improve documentation through fork and pull request mechanisms²⁷.

thehiddenwiki2023
Established on April 30, 2023, this repository focuses on mirroring verified onion links. It contains primarily HTML (92.3%) and CSS (7.7%) code, with the last update recorded on January 11, 2024. This repository aims to provide an accessible archive of links, reflecting the ongoing need for updated resources in the face of link rot due to the retirement of v2 addresses in 2021⁶.

TorBot
An open-source onion crawler, TorBot automates the extraction of data from .onion sites. It enables users to return page titles or host names, save link trees as JSON files, and check link validity. The maintenance status is dependent on community contributions, and updates occur based on user engagement³.

Darkdump
This OSINT tool fetches results from various dark web search engines, automatically filtering them against public abuse blacklists. It is designed to streamline the process of dark web reconnaissance, with updates contingent on user interaction and contributions⁴.

dark-web-scraper
This repository provides a web scraping tool that operates through Tor, featuring automatic IP rotation and the ability to extract key data such as crypto wallets and PGP keys from scraped content. Its maintenance status relies on community usage and contributions⁸.

OnionSearcher
A search engine scraper that checks for Tor connectivity and categorises links using keywords, OnionSearcher outputs results in CSV format. The repository's activity and updates depend on user participation and contributions⁹.

These repositories illustrate the diverse functions of Hidden Wiki successors on GitHub, serving as vital resources for navigating the complexities of the dark web.

Types of Resources Available

The GitHub repositories related to the Hidden Wiki host a variety of resources that can be categorised into several key types, each serving distinct purposes for users interested in the dark web.

Onion Link Directories

Link directories are curated lists of .onion addresses organised by category, facilitating navigation through the dark web. For example, the notthehiddenwiki/NTHW repository serves as a prominent link directory, providing users with a comprehensive collection of links relevant to cybersecurity². This type of resource is particularly useful for researchers and analysts seeking specific information within the dark web landscape.

Scraping Scripts

Numerous repositories contain web scraping scripts, primarily written in Python, designed to automate the extraction of data from .onion sites. Notable examples include:

• TorBot: An open-source onion crawler that returns page titles or host names, saves link trees as JSON files, and checks the validity of links³.
• dark-web-scraper: This tool scrapes .onion sites through Tor, employing automatic IP rotation and extracting critical data such as crypto wallets, PGP keys, and emails from the content⁸.
• OnionSearcher: A scraper that verifies Tor connectivity and outputs results in CSV format, categorising links based on keywords⁹.

These tools enhance the efficiency of data gathering from the dark web, making them vital for OSINT analysis.

OSINT Tools

Several repositories also focus on OSINT (Open Source Intelligence) tools that assist in gathering intelligence from the dark web. For instance, Darkdump fetches results from multiple dark web search engines and filters them against abuse blacklists, streamlining the reconnaissance process⁴. Such tools are essential for analysts aiming to compile and assess intelligence from diverse sources.

Educational Documentation

Some repositories include educational documentation, providing insights into the functioning and navigation of the dark web. This content is often formatted in markdown lists, offering structured information for users seeking to expand their understanding of dark web operations and security practices.

In summary, these resources on GitHub provide a comprehensive toolkit for navigating the dark web, each serving unique functions that cater to various needs within the cybersecurity community.

Using GitHub Hidden Wiki Resources for Research

The GitHub repositories related to the Hidden Wiki can be effectively utilised for various research purposes, including threat intelligence gathering, security research, and academic studies. These repositories, such as notthehiddenwiki/NTHW, serve as curated link directories that can be instrumental in identifying and researching dark web activities. For instance, analysts can explore links to .onion sites associated with cyber threats or illegal activities, aiding in the development of comprehensive threat models.

To clone a repository, users can execute the command git clone <repository-url> in their terminal, which creates a local copy of the repository on their machine. Navigating these repositories typically involves exploring the README files for instructions and using the provided directory structures to locate specific resources. For example, the notthehiddenwiki/NTHW repository categorises links by type, making it easier for users to find relevant .onion addresses for their research needs².

Integration with other OSINT tools enhances the utility of these resources. Tools such as TorBot and Darkdump can be employed alongside GitHub repositories to automate the process of gathering intelligence from the dark web. For example, TorBot can be used to crawl .onion sites for specific data, while Darkdump can filter search results from various dark web search engines, allowing researchers to compile and analyse data more efficiently³⁴.

A practical workflow for investigators might involve the following steps:

1. Clone the Repository: Use git clone to obtain the latest version of a relevant GitHub repository, such as notthehiddenwiki/NTHW.
2. Use OSINT Tools: Implement tools like TorBot to crawl and extract data from the links found in the repository.
3. Data Analysis: Filter and analyse the collected data using Darkdump to identify patterns or correlations.
4. Documentation: Document findings in a structured format, referencing the original sources from the GitHub repository for transparency.

This workflow illustrates how researchers can leverage the resources available on GitHub to enhance their investigations and gain deeper insights into dark web operations.

Technical Implementation and Scraping Tools

Python-based scraping tools are essential for extracting onion links from Hidden Wiki pages hosted on GitHub. Two of the most notable libraries used are Beautiful Soup and the requests library. Beautiful Soup facilitates parsing HTML and XML documents, allowing users to navigate and search the parse tree easily. The requests library simplifies making HTTP requests, enabling access to the content of .onion sites through the Tor network.

To extract onion links, a typical workflow involves sending a request to a Hidden Wiki page, retrieving the HTML content, and then using Beautiful Soup to parse the content for .onion URLs. Below is a simplified code structure overview:

import requests
from bs4 import BeautifulSoup

# Set up a session with Tor
session = requests.Session()
session.proxies = {'http': 'socks5h://localhost:9050', 'https': 'socks5h://localhost:9050'}

# Request the Hidden Wiki page
response = session.get('http://hiddenwikiaddress.onion')
soup = BeautifulSoup(response.content, 'html.parser')

# Extract all .onion links
onion_links = [a['href'] for a in soup.find_all('a') if a['href'].endswith('.onion')]

This script outlines how to set up a session through the Tor network, retrieve a page, and extract .onion links. It is crucial to ensure Tor is running before executing such scripts, as they depend on the Tor network for access.

In addition to the basic scraping tools, repositories also reference more sophisticated tools like OnionScan. OnionScan is designed to analyse .onion sites for vulnerabilities and provides insights into the security posture of these sites. Users can also explore other tools such as Darkdump, which automates the collection of data across multiple dark web search engines, filtering results for relevance and safety⁴.

For those interested in more comprehensive scraping solutions, the dark-web-scraper tool offers features like automatic IP rotation and the ability to scrape .onion sites while extracting sensitive information such as crypto wallets and PGP keys⁸. OnionSearcher is another utility that categorises results and checks for Tor connectivity, outputting the data in a structured format⁹.

When implementing scraping tools, it is vital to remain aware of the legal and ethical implications of accessing and using content from the dark web. Scraping should be performed responsibly and in compliance with applicable regulations.

Evolution of Hidden Wiki GitHub Archives

The evolution of Hidden Wiki archives on GitHub from 2015 to 2024 reflects significant shifts in accessibility, preservation, and collaboration practices within the dark web community. The earliest iterations of the Hidden Wiki emerged in 2007, serving as a directory of links to other .onion sites, but faced numerous challenges, including hacks and law enforcement interventions¹. By 2014, notable incidents such as the hacking of the Hidden Wiki site led to a critical need for more stable and collaborative archiving solutions¹.

The transition to GitHub as a platform for archiving these resources began to gain traction around 2015, driven by its accessibility and collaborative features. GitHub allows users to contribute easily, enhancing the repository's robustness through community engagement. For example, the notthehiddenwiki/NTHW repository was created on January 11, 2024, and has attracted contributions from 60 individuals, reflecting a collective effort to maintain an updated directory of links relevant to cybersecurity²⁷. This collaborative spirit is essential in the context of the dark web, where the stability of links can fluctuate due to various factors, including the frequent takedowns of .onion sites.

Changes within the Tor network, particularly the retirement of v2 addresses in 2021, further complicated archiving practices. This shift resulted in a significant loss of existing bookmarks, contributing to link rot and instability⁵. As a response to these challenges, many repositories have adopted strategies to regularly update and verify their links, ensuring that users can access reliable resources.

Several documented repository migrations and takedowns highlight the dynamic nature of this environment. For instance, the thehiddenwiki2023 repository, created on April 30, 2023, aims to mirror verified onion links and has seen ongoing updates to address the evolving landscape of dark web resources⁶. The need for continuous adaptation is evident, as users seek to preserve access to critical information amidst the uncertainties of the Tor network.

In summary, the evolution of Hidden Wiki archives on GitHub illustrates a response to the challenges of link stability and accessibility, fostering a collaborative approach to preserving valuable dark web resources. The ongoing contributions from the community play a vital role in maintaining these archives, ensuring that researchers and analysts have access to updated information.

Security and Privacy Considerations

Using links from GitHub repositories related to the Hidden Wiki presents distinct security and privacy risks compared to accessing live .onion sites directly. GitHub repositories are public and indexed, meaning any user can view their contents, which compromises anonymity. This exposure can lead to unwanted attention from law enforcement or malicious actors. Notably, while GitHub can serve as a useful resource for finding .onion links, users must remain cautious about the potential for outdated or malicious links¹.

Verification of onion addresses is essential before accessing them. Users should employ tools like OnionScan, which can assess the safety and functionality of .onion sites. This tool identifies vulnerabilities and provides information about the site's security posture, helping to mitigate risks associated with accessing potentially harmful content³. Additionally, using dark web scraping tools can assist in verifying the status of links by checking if they are live and extracting relevant data⁴⁸.

When engaging with dark web resources, it is advisable to use the Tor Browser alongside a reliable VPN. This combination enhances anonymity and security, protecting the user's identity and data from potential surveillance. Tor encrypts traffic and routes it through multiple servers, making it difficult to trace back to the user. Meanwhile, a VPN adds an extra layer of security by masking the user's IP address and encrypting their internet connection⁹.

It is crucial to remember that not all links found in GitHub repositories are safe or up-to-date. Users should exercise caution and conduct thorough verification before accessing any .onion site. This is particularly important given the dynamic nature of the dark web, where links can frequently change or become inactive, leading to potential exposure to harmful content¹.

Contributing and Maintaining Hidden Wiki Repositories

Community contributions to Hidden Wiki repositories on GitHub, such as notthehiddenwiki/NTHW, play a crucial role in maintaining the relevance and accuracy of these directories. Contributors can enhance these repositories by adding new links, updating existing entries, improving documentation, and reporting issues. To submit contributions, users are typically required to fork the repository, make their changes, and then submit a pull request. This process allows maintainers to review changes before they are integrated into the main repository, ensuring quality control and consistency⁷.

Verification of submitted links is essential to maintain the integrity of the directory. Contributors should ensure that links are functional and point to legitimate .onion sites. Tools like OnionScan can assist in assessing the security and status of these links, identifying vulnerabilities and confirming whether a site is operational³. It is advisable for contributors to test links before submission, as this helps reduce the incidence of broken or malicious links that could compromise the repository’s reliability.

Keeping directories current presents several challenges. The dynamic nature of the dark web, characterised by frequent changes in the availability of .onion sites, can lead to link rot. For instance, the retirement of v2 onion addresses in 2021 resulted in many previously valid links becoming obsolete⁵. To counteract this, repositories often implement regular review cycles where contributors are encouraged to revisit and verify existing links. This proactive approach can help maintain a more stable and useful resource for users.

Ethical guidelines for contributors are also vital. Contributors should refrain from adding links to illegal or harmful content, as this can have legal implications and damage the reputation of the repository. Transparency about the nature of the links and adherence to community standards is essential. Contributors are encouraged to engage respectfully with other community members, fostering a collaborative and supportive environment for maintaining these valuable resources.

In summary, contributing to and maintaining Hidden Wiki repositories involves a structured approach to ensure quality and relevance. By adhering to verification processes and ethical guidelines, contributors can help sustain the utility of these directories for researchers and analysts navigating the dark web.

Comparison: GitHub Archives vs. Live Onion Hidden Wiki

The comparison between GitHub archives of the Hidden Wiki and live .onion sites highlights key differences in accessibility, update frequency, link verification, safety, and searchability. Below is a summary of these aspects:

GitHub repositories, such as notthehiddenwiki/NTHW, serve as a collaborative platform for archiving links related to the Hidden Wiki. However, these versions are often outdated snapshots, as they may not reflect the current state of .onion sites. For example, the notthehiddenwiki/NTHW repository, created on January 11, 2024, allows for community contributions but may not always capture the latest changes in the dark web landscape²⁷.

Accessing live .onion sites can be more beneficial when looking for the most current information or when verifying the functionality of links. Live sites often provide up-to-date directories that reflect real-time changes. Users should consider using web scraping tools to facilitate this process, as these tools can help verify the status of links and extract relevant data⁴⁸.

It is essential to be cautious when using GitHub archives, as they can lead to outdated or potentially harmful links. Tools like OnionScan can assist users in verifying the safety and operational status of .onion sites before accessing them³. In scenarios where anonymity is crucial, accessing live sites directly through the Tor network is advisable, as it preserves user privacy while navigating the dark web.

In summary, the choice between using GitHub archives and accessing live .onion sites depends on the user's need for updated content, anonymity, and the ability to verify links.

Common Mistakes and Misconceptions

Assuming All GitHub Repositories Are Equally Maintained

Many users treat all Hidden Wiki GitHub repositories as equivalent resources without examining their maintenance history or contributor activity. The notthehiddenwiki/NTHW repository, created on January 11, 2024, has accumulated 1,914 stars and contributions from 60 individuals, indicating active community engagement². In contrast, the thehiddenwiki2023 repository saw its last push on January 11, 2024, after being created on April 30, 2023, suggesting potentially less frequent updates⁶. Before relying on a repository, check the commit history, number of active contributors, and frequency of updates to ensure you are working with a well-maintained resource rather than an abandoned archive.

Treating GitHub Archives as Real-Time Dark Web Directories

A prevalent misconception is that GitHub repositories mirror the current state of live .onion Hidden Wiki sites. These repositories typically contain static snapshots that may be weeks or months out of date, whereas the dark web itself changes rapidly due to site takedowns, migrations, and the retirement of v2 addresses in 2021, which broke a significant number of old bookmarks in a single change⁵. Using outdated links can lead to accessing defunct sites or, worse, compromised domains that have been taken over by malicious actors. For time-sensitive research, verify links against live .onion sites using tools like OnionScan rather than assuming GitHub archives reflect current availability.

Confusing Educational Archives with Operational Directories

Researchers often fail to distinguish between GitHub repositories designed as historical or educational resources and those intended as functional link directories. The original Hidden Wiki was a MediaWiki wiki operating as a Tor hidden service that could be anonymously edited after registration, with its main page serving as a directory of links to other .onion sites¹. GitHub repositories, by contrast, are public clearnet archives that sacrifice anonymity for accessibility and collaborative editing. Accessing a GitHub repository does not provide the same anonymity as browsing through Tor, and using these repositories for operational purposes without understanding this distinction can compromise privacy and expose research activities to surveillance.

Neglecting Link Verification Before Access

Users frequently access .onion links directly from GitHub repositories without verifying their safety or operational status first. This practice is particularly risky given the history of Hidden Wiki compromises, such as the March 2014 hack that redirected the kpvz7ki2v5agwt35.onion domain to Doxbin, and the November 2014 Operation Onymous takedown that resulted in law enforcement messages being served from the site¹. Before accessing any .onion address from a GitHub archive, use verification tools to check if the link is live, assess its security posture, and confirm it has not been compromised. Dark web scraping tools can extract metadata and verify link status without requiring direct access, reducing exposure to potentially malicious sites.

Overestimating the Completeness of Repository Collections

Many analysts assume that large GitHub repositories like notthehiddenwiki/NTHW represent comprehensive directories of all relevant dark web resources. While this repository describes itself as the largest repository of links related to cybersecurity and has accumulated 282 forks², no single repository can capture the entirety of the constantly evolving dark web landscape. There is no longer one single official Hidden Wiki, as several .onion websites host successors based on mirrors of the original, with many hosted for accessibility reasons due to frequent downtime and instability¹. Effective OSINT work requires consulting multiple repositories, cross-referencing sources, and maintaining awareness that any single archive represents only a partial view of available resources.

Ignoring the Legal and Ethical Implications of Repository Use

Researchers sometimes overlook the legal risks associated with accessing or contributing to Hidden Wiki repositories, particularly those containing links to illegal content. The Hidden Wiki came to prominence before October 2011 with its associations with illegal content¹, and while GitHub repositories may serve educational purposes, accessing certain links can still have legal consequences depending on jurisdiction. When contributing to repositories, avoid adding links to illegal or harmful content, as this can expose both the contributor and the repository maintainers to legal liability. Understanding the distinction between archiving information for research purposes and facilitating access to illegal services is essential for maintaining ethical standards in dark web research.

Your questions, answered

What is the Hidden Wiki on GitHub

The Hidden Wiki on GitHub refers to public repositories that archive links and directories originally found on dark web Hidden Wiki sites, making them accessible via the clearnet without requiring Tor. These repositories, such as notthehiddenwiki/NTHW created on January 11, 2024, serve as collaborative platforms where contributors can add, update, and document links related to cybersecurity and dark web resources². Unlike the original Hidden Wiki—a MediaWiki wiki operating as a Tor hidden service that could be anonymously edited after registration¹—GitHub versions sacrifice anonymity for accessibility and version control, allowing researchers to browse archived .onion directories through standard web browsers.

How to access Hidden Wiki GitHub repository

Accessing Hidden Wiki repositories on GitHub requires only a standard web browser and internet connection, as these are public clearnet resources. Navigate to GitHub.com and search for repositories like "notthehiddenwiki/NTHW" or "thehiddenwiki2023", then browse the repository files directly through the GitHub interface. The notthehiddenwiki/NTHW repository has accumulated 1,914 stars and 282 forks, indicating its visibility and accessibility to the research community². To download or clone a repository for offline analysis, use the "Code" button to access download options or copy the repository URL for use with Git command-line tools.

Is it safe to use Hidden Wiki links from GitHub

Using Hidden Wiki links from GitHub repositories carries significant risks, as these archives often contain outdated snapshots that may point to defunct, compromised, or malicious sites. The March 2014 hack that redirected the Hidden Wiki's kpvz7ki2v5agwt35.onion domain to Doxbin and the November 2014 Operation Onymous takedown that served law enforcement messages demonstrate the vulnerability of these directories¹. Before accessing any .onion address from a GitHub archive, verify the link's operational status and security posture using tools like OnionScan, and be aware that the retirement of v2 addresses in 2021 broke a significant number of old bookmarks in a single change⁵. Never assume that a link's presence in a repository guarantees its safety or legitimacy.

What is notthehiddenwiki NTHW repository

The notthehiddenwiki/NTHW repository is a GitHub project created on January 11, 2024, that describes itself as the largest repository of links related to cybersecurity, serving as a collaborative archive of dark web resources². With 1,914 stars, 282 forks, and contributions from 60 contributors, it represents one of the most actively maintained Hidden Wiki archives on GitHub². The repository accepts contributions including adding new links, updating existing links, improving documentation, and reporting issues, requiring contributors to fork the repository and submit pull requests⁷. This structure allows for community-driven maintenance whilst providing version control and transparency that the original dark web Hidden Wiki sites lacked.

How to scrape Hidden Wiki onion sites

Scraping Hidden Wiki .onion sites requires specialised tools that route traffic through the Tor network whilst handling the technical challenges of dark web content extraction. The dark-web-scraper tool scrapes .onion sites through Tor with automatic IP rotation, tries lightweight BeautifulSoup first and escalates to Selenium if JavaScript is needed, and extracts crypto wallets, PGP keys, emails, and onion links from scraped content⁸. TorBot, an open-source onion crawler, returns page titles or host names with addresses, can save link trees as JSON files, crawl custom domains, check if links are live, and build visual trees of link relationships³. These tools provide structured approaches to data collection whilst managing the anonymity and technical requirements inherent in dark web research.

What tools are available for dark web OSINT

Several open-source tools facilitate dark web OSINT work, each addressing different aspects of data collection and analysis. Darkdump fetches results from multiple dark web search engines and automatically filters all results against Ahmia's public abuse blacklist regardless of which engine is used⁴. OnionSearcher checks for Tor connectivity, rotates IPs, applies custom user agents for anonymity, categorizes links using keywords, and outputs results in CSV files sorted in ascending order of .onion links⁹. These tools complement scraping utilities like TorBot and dark-web-scraper by providing search capabilities, link verification, and automated filtering that help researchers navigate the dark web more safely and efficiently.

How often are GitHub Hidden Wiki repos updated

Update frequency varies significantly across Hidden Wiki repositories on GitHub, with no standardised schedule governing maintenance. The thehiddenwiki2023 repository saw its last push on January 11, 2024, after being created on April 30, 2023, suggesting a gap of several months between updates⁶. The notthehiddenwiki/NTHW repository, with 60 active contributors, likely sees more frequent updates due to its larger community engagement². Researchers should examine each repository's commit history and contributor activity rather than assuming regular maintenance, as the dynamic nature of the dark web—characterised by frequent changes in .onion site availability—means that even actively maintained repositories may contain outdated links.

Key Takeaways

• GitHub repositories provide clearnet access to archived Hidden Wiki directories but sacrifice the anonymity of Tor-based browsing, making them unsuitable for operational use where privacy is paramount.
• Always verify .onion links before access using tools like OnionScan or TorBot, as repositories frequently contain outdated addresses pointing to defunct or compromised sites following incidents like the 2014 hack and Operation Onymous takedown¹.
• No single repository offers comprehensive coverage—consult multiple sources such as notthehiddenwiki/NTHW and thehiddenwiki2023, cross-reference findings, and monitor commit histories to assess maintenance quality²⁶.
• Distinguish between educational archives and operational directories: GitHub repositories serve as historical references for research, not real-time link directories for active dark web navigation.
• Understand legal and ethical boundaries when accessing or contributing links, particularly given the Hidden Wiki's historical associations with illegal content prior to October 2011¹.

For practical guidance on verifying link functionality after identifying resources through GitHub repositories, consult Troubleshooting: Tor Browser Onion Links Not Working.