While both web directories and search engines gained popularity in the 1990s, search engines developed a life of their own becoming the preferred method of Internet search.
Wandex’s technology was the first to crawl the web indexing and searching the catalog of indexed pages on the web.Another significant development in search engines came in 1994 when Web Crawler’s search engine began indexing the full text of web sites instead of just web page titles.Only a month later on April 20, 1994, Brian announced the release of Web Crawler live on the web with a database of 4000 websites.On June 11, 1994, Brian posted to the Usenet group comp.infosystems.announce that the Web Crawler Index was available for searching.A little less than a year later, Web Crawler was fully operating on advertising revenue.
A young America Online, without access to the web, acquired Web Crawler on June 1, 1995.
Also as part of the deal, Info Space acquired rights to Web Crawler.
Ask Jeeves (now Ask.com) purchased the portal in 2004. Originally it was a highly regarded directory of sites that were cataloged by human editors.
Rather, in 1993 the graphical Mosaic web browser improved upon Gopher’s primarily text-based interface.
About the same time, Matthew Gray developed Wandex, the first search engine in the form that we know search engines today.
By November 14, 1994, Web Crawler served its one millionth query.