TLDR: Motivated by recent research, we investigate an approach of combining LLMs and Graph Neural Networks for the task of taxonomy construction.
A taxonomy is a directed acyclic graph (DAG) where directed edges from a node u to a node v, represents u is a v. Here is a simple example.
A taxonomy is a specific type of ontology; a structure that organises knowledge by modelling entities by how they relate to one another.
Ontologies are useful both in research, to understand a new domain, and for artificial intelligence to provide organised knowledge to automatically answer and verify answers (automated reasoning). This moves away from representing knowledge as uninterpretable weights of a model learned from data, towards a structured, human verifiable format.
This is particularly pertinent given the need to build AI which is both more explainable and verifiable.
Most taxonomy construction is done using domain experts in the field. This is outpaced by the increasing growth of research and new knowledge. Additionally, differences in authorship can lead to inconsistencies between taxonomies rather than being data driven. This motivates automating the task of taxonomy construction in a data driven manner.
We investigate the problem of constructing taxonomies using a graph and language model based approach. We build on the work of the state of the art graph-based taxonomy construction model, Graph2Taxo.
We start with an undirected graph that noisily connects potential hyponym-hypernym pairs between terms.
Nodes are represented as a BERT embedding of their corresponding term, and edges represent where there is an extracted lexico-syntactic pattern, as found in text.
In addition to the edges themselves, we represent additional information between terms as edge features. Specifically:
- Substring Features: string features found from the original textual data such as counts of where one of the terms contains, endswith or prefix and suffix matches.
- NLI Features: A prediction of logical entailment, neutrality or contradiction of the statement
[u] is a [v]for term u and v, using a trained natural language inference model. - TaxoLLama Features: The perplexity of
vbeing generated as a hypernym ofuand vice versa.
We apply a graph neural network to this noisy graph, to update the embedding representations. Due to the asymmetric nature of hyponym-hypernym relations, we experiment with applying a directed GNN. Additionally, due to the presence of incorrect edges in the noisy graph, we experiment applying a Graph Attention Network (GAT) to weight the importance of connected terms in message passing.
After enriching the embeddings through message passing, we apply a multilayer perceptron between any two connected terms to determine the existence of an "is a" relationship.
It is important to note that the MLP is only applied to terms already connected in the original graph. Although this reduces the search space from every possible pairing of terms, it means the taxonomy is a subgraph of the initial noisy graph. Consequently, any true "is a" relationship not captured in the original graph can never be recovered. A limitation of our method and Graph2Taxo is therefore that the recall of the taxonomy is upper-bounded by the recall of the noisy graph.
Motivated by the noisy graph acting as an upper bound, we attempt to expand it by introducing new edges. For each term, we apply NLI to the K closest terms in the embedding space. We add an edge where the contradiction score between terms is below some threshold.
We perform ablation experiments, to investigate how the introduction of various parts of the pipeline imapcts performance. Our main findings are the following:
Graph Structure Helps: Introduction of the GNN improves performance over applying the MLP directly with substring edge features to predict hypernymy. We posit that this confirms the intuition that graph structure is a useful inductive bias for taxonomy pruning.
BERT helps, but only modestly: Replacing random initialisations with BERT improves F1 score but only modestly. We believe this indicates that although a semantic prior is useful, the majority of the transferrable information is in the graph structure and edge features.
Neither Direction or Attention Help Improves the Average Model We observed minor degradations in performance when applying directed message passing and attention. We believe the most plausible explanation is that the initial graph is too noisy to benefit from them.
LLM Edges features help directed models: The addition of NLI edge features to the directed model yielded the largest increase in F- score of any of our changes in our ablation, from 0.360 to 0.385.
NLI and TaxoLLama are not additively beneficial in the augmented graph: Using both edge features leads to a reduction in performance, compared to using either alone. This implies that the two LLM derrvied signals largely capture mutual information rather than complementary information.
Overall, we found LLMs useful for edge level features rather than as a source of graph augmentation. We believe the main bottleneck for graph based taxononomy construction remains noisy graph quality.
For further technical detail on our motivation and methodology, our paper can be located at ./finalReport-G078.pdf.