8

I am working on a project that requires that I understand different techniques used by search engines for the web.

I have a strong scientific and development background, so I am not afraid of highly technical information.

I am looking for all forms of technical information including information on web crawlers, other techniques for acquiring data, methods of data storage and how to query it, etc., etc.

I am completely new to the subject and I'm looking for useful resource and books on the subject. Any suggestions are appreciated.

RLH
  • 533
  • 4
  • 16
sebpiq
  • 365
  • 1
  • 3
  • 8
  • why not ask search engine itself ;-) , Type "How does google work" and you will know. – Pankaj Upadhyay Sep 03 '11 at 18:09
  • Steven Levy's new book on Google, In The Plex, is an in-depth and fascinating look at the company. It also covers the four main duties of a search engine, and how Google innovated in all go those dimensions. – Macneil Sep 03 '11 at 18:21
  • @Pankaj : that's true, there is a lot of resource online ... but it is quite sparse, and I am a bit lazy to gather everything to put the pieces together. If a good book existed, that would be great! And also, I guess there must be different techniques ... Google is of course "the guide" when it comes to search engines, but I want to learn about many techniques and options so I can choose the best ... – sebpiq Sep 03 '11 at 19:14
  • @Macneil: it seems very interestant indeed ! I might read it ... however it doesn't seem to talk about it from the technologic angle at all ... – sebpiq Sep 03 '11 at 19:14

3 Answers3

3

This area of study is known as Information Retrieval. This Wikipedia article contains a good summary and lots of useful links.

Andy Waite
  • 510
  • 2
  • 9
  • +1 having studied my masters in Information Retrieval, a "search engine" is just the tip of the iceberg – Darknight Sep 03 '11 at 21:46
  • I read the wikipedia article, and that looks like good theoretical information ! What I was looking for ! – sebpiq Sep 03 '11 at 22:24
  • Any good reference about "information retrieval" then ? @Darknight what books did you study from ? – sebpiq Sep 03 '11 at 23:13
3

First, Google actively participates in the "science" of technology and often shares their knowledge by releasing papers from their R&D department. You can find those papers from the reference link below. I haven't searched for specific papers on search retrieval algorithms but there should be more than enough information available on the subject from a very technical perspective, as well as papers on storing massive sets of data and effectively querying it.

Publications by Googlers

Other than Google's resources, I highly recommend that you look into Semantic Web research. Semantic web isn't a method of searching data and even though the concept at first seems a little vague, the clear implications of a semantic web "engine" would be to parse the information within the WWW and link relevant information with one another.

In short, semantic web is the science of what many forward-thinkers hope and are working for the internet to truly become where the information that is provided is well parsed, interpreted and correctly linked together. I haven't looked into it much myself, so some of my information may be a bit incorrect. However, their are plenty of resources available that discuss semantic web and many are hoping/waiting/working on a break through within the field with the hopes of making the "next big thing" for the internet.

A good starting point for learning about semantic web is, of course, Wikipedia.

These references may not be books but it is a lot of information. Reading and sifting through all of the technical information should keep you busy for a while.

RLH
  • 533
  • 4
  • 16
  • @Thanks a lot ! This link to google articles is great ! I didn't know that such papers existed, and free to consult !!! About the semantic web, I knew a bit about the principles already, I will take a look at it, but from what I remember it is something that has been a dream for 10 years, but is still far from being reality (I might be wrong), ... but thanks again !!! – sebpiq Sep 03 '11 at 21:08
  • Yes it is a bit of a dream, but while people dream, a lot of research is done to try and make it happen. I'll admit, it is a bit over my head but to my understanding, it is the 'hope' of the future, so it's kind of the cutting edge of the field. – RLH Sep 03 '11 at 21:17
  • Yes ... you are right ! I probably need to take a look at it, at least for the sake of culture ! – sebpiq Sep 03 '11 at 21:30
  • 2
    -1 although you answer is not "incorrect", the OP asked about the "theories" behind google/search engines. This area of study is know as Information Retrieval and Natural Language Processing (NLP). as answered Andy Waite – Darknight Sep 03 '11 at 21:49
0

Following advices from @Andy Waite I read the wikipedia page on information retrieval, and followed the references. There is a lot of online information, and I find this introduction to information retrieval, which is an online book from 2008, so probably up-to date, and which seems to be quite a good intro to the topic.

sebpiq
  • 365
  • 1
  • 3
  • 8