24

I want to write something that takes a sentence and identifies each word it contains and defines what part of speech each word is.

For example

Hello World, I am a sentence

would return this

verb noun, pronoun verb adjective noun

Ideally, I'd like to eventually take it one step further and take a sentence and programmatically have it understand what it is trying to interpret and maybe do something about it.

So my question is, has someone heard of something like this?

PersonalNexus
  • 2,989
  • 5
  • 27
  • 42
Vinny
  • 259
  • 1
  • 2
  • 5

4 Answers4

20

This is called Natural Language Processing and it's a huge, complex field. Something like you describe is a monumental achievement, and even the best solutions, like Watson, are nowhere near perfect.

Things like this make it challenging: "Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo"

a grammatically correct sentence in American English, used as an example of how homonyms and homophones can be used to create complicated linguistic constructs. It has been discussed in literature since 1972... It was also featured in Steven Pinker's 1994 book The Language Instinct as an example of a sentence that is "seemingly nonsensical" but grammatical...

The sentence's meaning becomes clearer when it's understood that it uses the city of Buffalo, New York and the somewhat-uncommon verb "to buffalo" (meaning "to bully or intimidate"), and when the punctuation and grammar is expanded so that the sentence reads as follows: "Buffalo buffalo that Buffalo buffalo buffalo, buffalo Buffalo buffalo." The meaning becomes even clearer when synonyms are used: "Buffalo bison that other Buffalo bison bully, themselves bully Buffalo bison."

gnat
  • 21,442
  • 29
  • 112
  • 288
Ryathal
  • 13,317
  • 1
  • 33
  • 48
  • 2
    This is what I'm looking for! Has anyone heard of anyone adapting this on a smaller scale? Open-source? Examples of this being used in smaller scales? – Vinny Feb 07 '12 at 21:25
  • @Vinny AFAIK nothing much is available in open source as solving these issues is highly profitable to companies, like MS Word grammar detection. There are some chatter-bot programs that are available I believe though. – Ryathal Feb 07 '12 at 21:31
  • 1
    @Vinny It's difficulty has nothing to do with scale. Natural language processing has an inherent complexity that is not reduced when you reduce the "scale". – Tulains Córdova May 28 '13 at 17:44
6

Though splitting a sentence and determining the grammatical correctness along with solving your first problem is easier than your second problem, many complexities like verb-nouns or gerunds like swimming, programming, etc and other such intricacies, it still is a challenge - See Morons' answer.

But your second problem - people have put in huge efforts to find a perfect solution, but a really perfect "interpretation" algorithm is not realizable practically for any natural language like English - there are variations that will screw up your algorithm. This field - a hybrid between AI, Computer Science and Linguistics is known as NLP. Consider this: Even Google Translate is not perfect when "interpreting" sentences.

But nevertheless, this is a very interesting field to dabble with.

yati sagade
  • 2,089
  • 2
  • 19
  • 27
  • @StriplingWarrior I just wanted to contrast sufficiently between the two problems posed by the OP. Noted. Edit on its way – yati sagade Feb 07 '12 at 18:35
4

I think you should start reading this Wikipedia article:

http://en.wikipedia.org/wiki/Part-of-speech_tagging

(it is a research field, don't expect any easy solution for it.)

Morons
  • 14,674
  • 4
  • 37
  • 73
Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • 2
    IT should be added that while NLP is huge, hard and probably intractable on today's computers, POS tagging is the easiest part of it, and with either sufficient corpus size or a lot of dedication and manual rule-writing it can be solved almost perfectly, certainly above 99% correctness. That may well be enough for your needs. – Kilian Foth Feb 08 '12 at 08:27
  • thank you , this is exactly what I was also searching for. – Amc_rtty Jun 09 '18 at 13:42
  • actually based on the description of OP, this should be the accepted answer as you correctly observe @KilianFoth – Amc_rtty Jun 09 '18 at 17:17
-1

A cheap way of doing this would be to set up a database of the dictionary (I'm almost positive that someone has done this).

Need two fields in the table: word and usage

Turn the phrase into an array of strings, (each word being a string) and independently:

select 'usage' from Dictionary WHERE 'word' = $word;

It's a heavy solution, but one that I've used in the past.

gnat
  • 21,442
  • 29
  • 112
  • 288
mheinke
  • 9
  • 5
    This assumes that each word has only one possible POS, and I assure you that’s not the case at all. – microtherion May 28 '13 at 17:56
  • Agreed - there is no way this could feasibly work (at least in English) with all the words that can act as nouns, verbs, etc., depending on the context. – Derek May 28 '13 at 19:45