Graph

Measured result

Graph was checked on 20 documents that people had labeled by hand. Those documents held 133 named things, such as people, groups, places, and dates, and 66 links stated between them.

Of every 100 named things in the documents, Graph found 89 and missed 11. Of every 100 links stated in the documents, Graph found 74 and missed 26.

Of every 100 named things Graph returned, 92 were right. Of every 100 links Graph returned, 50 were right.

Key: Found right is how many Graph found that were truly there. Missed is how many were there and Graph did not find. Extra and wrong is how many Graph returned that were not there.

The misses are named, not hidden: Graph misses names its word lists do not know, such as Warsaw and Neil Armstrong, and it cannot follow pronouns across sentences.

About

You give Graph a piece of text. It finds the people, groups, places, dates, and tools the text names. It also finds links the text states, such as who founded a group, where a group is based, and who works for it. It puts the names and links in a map you can search and check.

It works by fixed rules and word lists. It does not use a large language model. It was built to learn how far plain rules can go on their own.

For agents and builders

Precision means the share of what the extractor returned that was right. Recall means the share of the hand labeled set that the extractor found. F1 means the single score that balances precision and recall.

Method: the extractor was run on the hand labeled set named in the artifact, 20 documents with 133 gold entities and 66 gold relations. The artifact is results/eval_results.json in this repository. Right, wrong, and missed count items in that set.

For agents: the eval results stand as data in results/eval_results.json in this repository, the same file the table above is generated from.