The Deep Web: Searching for Needles in a Limitless Haystack

Searching the web for specific information from a valid source has become akin to looking for a few needles in a huge haystack. The online haystack has grown mega-exponentially in just the last few years.

The good news is that the haystack contains a wealth of competitive information to help executives better achieve their goals – if those needles can be found.

As the databases of Google, Bing and other surfing and other web-searching tools sprout more roots to encompass wider and deeper layers of online information, successful searches take more time than ever and require more searching prowess.

Google Search, the largest of the surface “haystacks,” now accommodates a whopping three billion searches daily. But even more impressive and surprising to most is the enormous number of database indexes searchable by this online giant –billions more than the number of searches performed through it.

Search engines like Google and Bing are able to access only an estimated .03 percent of the content available to trained professionals on the Internet, comparable to pulling a teaspoon of water from the flooding Mississippi River as it rushes past. Beyond and beneath what is uncovered in the typical quick search by most people is a vast river of information known as the deep web.

Common Browsers Don’t Work

Also known as the hidden web or the invisible web, the deep web is a boundless collection of online resources that search engines don’t begin to touch. Common browsers such as Internet Explorer and Firefox can’t access the deep web. This deep web is distinguished from the “surface or searchable web” that most people use by the assurance of complete anonymity; users leave no cyber trail. A deep web searcher leaves no browser history and the user’s IP address cannot be identified.

Estimates put the size of the deep web at up to 500 times the content of the surface web – about 7.5 petabytes or perhaps 7.86 gigabytes – truly a mind-boggling collection of detailed information. Countless databases of articles, data, statistics and government documents are included in this mysterious cyber territory. How much of this information is useful and reliable lies in question, but individuals who are skilled in accessing the deep web have an enviable advantage.

Information is findable in the deep web with special browsers, some of which were identified recently when a contractor for the National Security Agency shared classified government information that he found by accessing the deep web.

Cyber sources report that downloads of private browsing tools have more than doubled in just the last six months, as more and more individuals and companies grow wary about their online actions unknowingly being monitored. The deep web’s influence in terms of third-party or competitive intelligence has only begun to be scratched.

Tool For Businesses

Researchers proficient in navigating the deep web say it can and is indeed being used for noble purposes as well.The deep web is particularly valuable for executives,researchers, law enforcement personnel, journalists and anyone intent on gaining as much customized, targeted intelligence as possible on any given topic.

In this age of identity theft, computer hacking and other vital threats to who we are, what we do and where and when we do it, it’s anyone’s guess where the deep web may take us.

One thing is sure: finding that needle in an ever-deepening haystack takes expertise, determination, time and intense ethical creativity.

Kerry Smith is principal of Informationworks, Inc., an information and research agency serving businesses and individuals throughout the Midwest from Edwardsville. A member of the Association of Independent Information Professionals, she can be reached atkerry@informationworks.org or 618.225.2253.