Punctual Punctuation

I've been looking at the output of a text processor/pos tagger and noticed that there is a whole class of error I haven't been looking for but should and that is punctuation. It makes a big difference when predicting the start and end of a sentence on the tagging of POS. Especially with words that may be nouns or verbs depending on their context.

Mainly the biggest problem i see is in the handling of the period which for Americans at least appears as the decimal separator for numbers, in abbreviations and at the end of sentences. So it becomes important to discover these three cases accurately. I noticed in some cases that decimal was being concatenated with the prior word (clearly a bug with differentiating abbreviations from words!)

I believe I will treat this kind of error in the same class as lemma error in that the POS prediction is discounted and error is attributed to form. I've been working with this kind of measurement for a long time and I'm surprised I hadn't noticed this class of error before. However it is good to see it now and incorporate it into my methodology.

Perfection in NLP

Is there perfection in NLP? Let's take, for example, part of speech analysis. For many texts such as the one you are reading, the POS is easily discernible. Occasionally there will be some odd usage that is hard to classify but for the most part it is clear what the part of speech is. There are basically 60 or so parts of speech in English that are worth tracking. University of Pennsylvania's Treebank has around 110 with some being odd combinations of part of speech.

However you run into trouble with text messages and forum postings:

"C U L8r"
"That show was def"
"Another Halo Game? Interesting."

You have further trouble when you look at these musical lyrics (from Weird Al):

"What y'all wanna do?
Wanna be hackers? Code crackers? Slackers
Wastin' time with all the chatroom yakkers?
9 to 5, chillin' at Hewlett Packard?
Workin' at a desk with a dumb little placard?
Yeah, payin' the bills with my mad programming skills
Defraggin' my hard drive for thrills
I got me a hundred gigabytes of RAM
I never feed trolls and I don't read spam"

Those aren't so bad but now try to deconstruct "Welcome to the Terrordome":

"I got so much trouble on my mind
I refuse to lose
Here's your ticket
Hear the drummer get wicked
The crew to you to push the back to black
Attack so I sat and japped
Then slapped the mac
Now Im ready to mike it
(you know I like it) huh
Hear my favoritism roll oh
Never be a brother like to go solo
Lazer, anastasia, maze ya
Ways to blaze your brain and train ya
The way I'm livin, forgiven
What Im givin up"

OK so is it cheap racial demagoguery or social conscience writ large? The fact is that it makes sense to some people who can decode it. However from a machine learning point of view its a mess! And yet here we have written communication. It represents the problem of attaining perfection. I will say that I had as much trouble understanding what Chuck D was saying in the above lyrics as I did reading a macro economics text book. However the more I knew the more both made sense. The start, of course, is knowing where to begin and with just a simple part of speech analysis I think it is asking a lot of machines, heck people even, to get anywhere with text they don't understand. I'm reminded of the Chinese Room Argument.

We all know that these are the challenges. There is no arguing that language captured in text runs a huge spectrum of quality and consistency. The fact is that people who probably would score poorly in POS identification for the most part (guilty as charged here) we still manage to understand what we are reading and have the ability to associate information, spot entities of particular types and generally get by. Sometimes we miss things that are communicated via sarcasm - like that girl I was staring at who said "why don't you just take a picture." I didn't realize she wasn't serious... Sheesh.

Great so lets assume that we accept these problems and decide to identify POS anyway. We would like a measure of how "good" the system is and typically this means taking some text that is "regular" without any weird word plays, lyrics, noise or any other nonsense. I have three text files I use often for this purpose. One is a very dry description of plant maintenance. The other is a post game report on a Red Sox win (oh shut up they were losers all my life, let me enjoy the recent string of wins.) Finally I have a review of the Halo 3 game. I have this one because it is full of weird made up grammar. Gamers and programmers seem to prefer to invent POS for things instead of using "standard English." You know what I mean. "Spreadsheet these numbers for me." If you don't see the made up usage in the previous sentence you need to check out this blog.

I definitely suggest using multiple documents in different voices. Include a lot of things you are interested in - especially if you do entity extraction. If you are interested in people then a sports report is great because multiple people get mention in each post-game report. Find things that interest you and your work and begin there. Keep it short. Creating a reference file is a tedious process and long documents are hard to do because the tedium is a killer. At least for me. Maybe there is a way to make it fun. Maybe I should use Penthouse Forum articles instead.

The next problem is that the words internal to words that are co-located generally aren't their normal part of speech. "Welcome to the Learn to Fly Website!" "Fly" is part of a proper noun so is it a verb? The hell it is. That co-location is a noun. On a token by token basis MAYBE that word is a verb but together with it's neighboring tokens it is part of a noun.

So when do you test for part of speech, before determining co-locations or after? You probably want to do BOTH. However anything prior to determining co-locations is for debugging purposes only. You need to know what is going into your process for determining co-locations. Corrections here are important. For analytical purposes however you will score your POS after the co-locations are figured out.

Here is the rub. Your co-locations might not be correct. Some will be good, some will not. So how do you separate this problem from POS analysis? In my opinion you take the output and put one token or co-location per line along with the part of speech and as you score you check for the word count. If they match then you compare from the output to your idealized reference set. If they don't then you keep reading until the counts match. Each mismatch is a bad co-location. Keep score of good versus bad co locations separately and if its bad don't bother checking the part of speech. Just don't count it. In my opinion this produces the most even and fairest analysis.

Keep track of the types of POS you are using and the number right and wrong for each one. It isn't hard to calculate the f-measure for each part of speech but you need a statistically significant number of examples in order to get a reasonable number. There are a number of problems with F-Measure so its not really clear that it is helpful as a measure here. A pure % right calculation is also of questionable utility. However they will do for a start. I have friends working on a new measure that should help in this analysis.

As for the problematic cases such as lyrics and text captured from forums I suggest giving a hand at trying to determine the collocations and part of speech by hand first. Then try your process for POS prediction and see how it does. You will find a number of issues. How do you score where someone has inadvertently or deliberately used bad diction? In this article the author talks about the "Obama Affect" when I am pretty certain he meant the "Obama Effect." If we discount bad spelling, diction and grammar in text then how can we measure how accurate we are in modes of communication totally defined by this such as SMS messages? I think we can't do this kind of discounting. We have to pick, if not a POS, at least a role that the token is performing. Even a smiley face has a role, be it decorative or meaningful at a meta level.

The basics of Text Mining

The basic needs of any text mining effort are:

• Text Processing (formatting, cleaning, unencoding, encoding, etc.)
  
• Determination of co-locations (words that have more meaning together than apart like 'United States of America' is a single concept and should be grouped together.)
  
• Determination of parts of speech/role.

You can tell how advanced a system is by what they do with the text before analysis. Digital Reasoning, Attensity and others do all three before analysis. It is a key factor in "exhaustive extraction" and in the creation of advanced structures like associative networks. Without understanding the semantic structure how can one determine the actual meaning of the elements?

I've been getting more and more frustrated with search engines. You have to leap through so many hoops to do the types of searches I've gotten used to with semantics-aware engines. "Remains" is both a predicate and an entity - depending on how it is used. In order to find exactly what I am looking for I have to put in the term and look at what comes back. When I see it is bringing back a lot of incorrect cases I then help it disambiguate the responses by adding in negative examples. So if I am searching on "cold" and got a lot of responses back on low temperature when really I was more interested in the respiratory blockage I would add "-temperature" which is not bad unless some article I might want actually has temperature in the sense of fever ('running a temperature') in which case I would have just filtered it out.

When you are evaluating textual mining solutions it is important to make sure they can provide this kind of functionality. Google is a 90's technology. As simple as it is it does give access to a lot of information but it's simplicity also makes it a lot of work when we are examining large scales of data with subtle requirements in our search. In fact the more important it is to find "The Document" over general documents related to the search topic the more vital a semantic layer becomes.

Consider this - if a vendor is trying to sell you on a solution to determine "customer voice", or as I've heard it referred to elsewhere, sentiment analysis. Ask them how it knows when "sucks" refers to a negative connotation in vacuum cleaner reviews. They may laugh and say that is a minor example but the fact is, if they aren't dealing with it they also aren't dealing with a lot of other factors. Keywords are no longer a useful technology. Semantic understanding is required for subtle detection. Anyone who tells you differently hasn't a leg to stand on.

Arrrg. Working with other peoples data

The biggest pain with text analytics and data mining is working with other peoples data. Invariably it is all garbage. This file is ASCII, that file is UTF8, this other file has some weird code page, that file is 7bit. It can cause you to pull your hair out. This happens with data from every corporation or even the Federal Government. I was just working with the TraxIntel DB and was having trouble with the analysis missing what I thought was a lot of important information. Of course it was something as simple as just files not formatted the way I was expecting. This should have been expected since TI pulls in data from a lot of different sources. The lessons is, pre-process all of your input and make sure it is formatted exactly the way you want.

I should have known better because when I was working with the public collection of Enron emails they had all sorts of encoding in them that are unique to emails but that aren't text you'd want to analyze. The simple solution was to find plain text versions of the messages. The purist answer of course is to create a filter for various types of potential file formats and fix the input before it hits the semantic analyzers. What a concept.

Anyway, the point of this post is that sometimes when you are debugging this stuff it isn't the high level models and complex software that is broken. "Garbage in means garbage out." A lesson as old as the hills.

Artificial = Synthetic

From my perspective, the terms "artificial" and "synthetic" convey the same engineering meaning relative to machine learning. The intelligence of a computer system is derived from a human interpretation, model, of how one believes the learning process evolves. Since this process is bound by this constraint, any derived knowledge can be no better than the underlying algorithm that supports the process. To this end, the application has synthesized the function of the human brain's cognitive capabilities as modeled by the the algorithm. The term "artificial" or “synthetic” simply implies that the learning was done by machine.

Furthermore, I argue that Eliza or any any system that implements a similar fundamental concept are examples of how computers can be programmed to mimic the behavior of a human. I think it is very unfair to claim that Eliza is an AI system. Eliza was simply a program to dupe naive people into believing that the computer was performing an "initial psychiatric interview" with them. It did not offer any form of learning, but rather searched for key words that could be rephrased to answer/create new questions that the user would deem to be plausible. I view Eliza as a teaser to the capabilities that machine learning could offer in the future. Interestingly, I always considered Eliza to be a condemnation of psychotherapy. Based upon comments raised by my blog colleague in our offline discussions, I now recognize the adverse impact that it has had on the perception of AI.

As an engineer, I believe that a fair example of an AI system is a linear/non-linear control mechanism. Such systems are used in various applications, and using real-time data, adapt system performance to ensure that functionality and stability are maintained. A simple example of such a system is a one-step ahead controller. Examples of these systems can be found aboard ships, in machine-based automation, etc.

From many discussions that I have had with my academic peers in sociology and psychology regarding machine learning, I recognize the ongoing philosophical debate that surrounds it (i.e. artificial vs. synthetic). The bottom line is that until the cognitive function of the human brain is mapped, machine intelligence will remain an interpretative representation of this function. Philosophically, I believe that Artificial Intelligence is Synthetic Intelligence.

Artificial vs. Synthetic

Recently around the office there was some debate on the use of artificial intelligence vs. synthetic intelligence. Artifical intelligence implies that the intelligence is not genuine. Syhthetic intelligence suggests the intelligence is not just an imitation but is genuinely a form of intelligence specifically created.

There is an all to short article on the subject at wikipedia. I wish it was quite a bit longer. Its a fresh and healthy debate. While it is philosophical - the whole nature of what we are building is based upon philosophy.

I think for now I am on the synthetic intelligence end of things. The systems I'm seeing are able to perform intellectual tasks that I would do. They don't do them in the same way as I would but they produce similar if not better results and they certainly do it quicker and faster than I do.

Business TN hack job on Digital Reasoning

So I got forwarded this article from Business TN about the company I work for. At first the terrible journalism standards in it made me angry. I used to run a blog that followed the terrible and sensational journalism used in the Natalie Holloway Case. Then I had to sit back and laugh.

My favorite line is an un-cited quote about the CEO's mother (who is a board member) "calling all the shots." Ok lets deconstruct this. First, the poor woman has not been well for many months and has been in treatment. While she is recovered now she pretty much hasn't called any shots for the last year. Second, lets say for a second she was quite influential. What exactly is the problem? Is it that she is a woman? Is it that she is a mother? Really? Come on - how old school is that? It really goes a long way to prove Tim's original assertion that Venture Capital in Middle Tennessee is not really capable of evaluating a good thing when they see it and is driving good business ideas out of this state.

Frankly, if Tim moves the shop I would follow. DRS has great technology that has been proven in the field of battle. It needs to be in the hands of civilians now. Where as we will eventually develop something out of what we are doing with the government, that is a slow process. I have to shrug my shoulders. If that uncredited person actually did say the problem was Tim's mother then there really is no helping the local VC crowd as they are hopelessly short sighted and frankly misogynistic. It is well they weren't named because any female entrepreneurs in state who read their comments would instantly cross them off the list of people they would trust.

The fact that they go un-cited however makes me question the entire article itself however. I wonder how much of that is really just the invention and belief of the author. Now you would be right to say where do I get the right to critisize him when I blog. Well for one, I know bad journalism when I see it. Second I have left open the comments on this blog so if you want to call BS on something I wrote I would gladly engage in that. There are a lot of things in this blog that are opinion and not fact and sometimes I am not as careful to distinguish between them as clearly in my writing as I should.

I'd be glad to hear from anyone in the NLP field that also finds that the VC in their area don't get what they do and don't invest in innovative startups. I am sure it is not just Middle Tennessee. When you do a map of the location of the top 30% of the data mining/text analytics companies they are not surprisingly centered on a short list of hubs.