Evaluate predictions

Following scripts can be found in the python folder.

Evaluate a single prediction file

Use this script to evaluate a single prediction file.

Script name: eval.py

Requires:

pip install scipy tqdm

Usage:

python eval.py {path to file containing predictions} {path to testset file}

Example output:

MRR: 0.389
Hits@1: 0.298
Hits@3: 0.371
Hits@10: 0.537

Evaluate an experiment

Use this script if you want to evaluate multiple datasets containing multiple prediction files at once (Multiple datasets -> Multiple prediction files).

Script name: eval_experiment.py

Requires:

pip install scipy tqdm

Usage:

python eval_experiment.py --datasets {list of datasets} --predictions {list of prediction file names}

File structure:

Each dataset should have its own folder. Evaluations are run

for each {dataset} in {list of datasets}:
   for each {prediction file name} in {list of prediction file name}:
      Path to prediction file: f”./{dataset}/predictions/{prediction file name}”
      Path to testset file: f”./{dataset}/data/test.txt”

Example:

python eval_experiment.py --datasets OBL WN18RR --predictions predfile1.txt predfile2.txt

---- OBL
    |
    ---- predictions
        |
        ---- predfile1.txt
        |
        ---- predfile2.txt
    |
    ---- data
        |
        ---- test.txt
---- WN18RR
    |
    ---- predictions
        |
        ---- predfile1.txt
        |
        ---- predfile2.txt
    |
    ---- data
        |
        ---- test.txt

Output:

OBL
predfile1.txt MRR: 0.389 Hits@1: 0.298 Hits@3: 0.371 Hits@10: 0.537
predfile2.txt MRR: 0.389 Hits@1: 0.298 Hits@3: 0.371 Hits@10: 0.537

WN18RR
predfile1.txt MRR: 0.389 Hits@1: 0.298 Hits@3: 0.371 Hits@10: 0.537
predfile2.txt MRR: 0.389 Hits@1: 0.298 Hits@3: 0.371 Hits@10: 0.537