README

ELLIOT

!PyPI - Python Version  !GitHub repo size !GitHub

[Docs] | [Paper]

[Docs]: https://elliot.readthedocs.io/en/latest/

[Paper]: https://doi.org/10.1145/3404835.3463245

 A NEW VERSION OF ELLIOT(v0.3.1) IS OUT

Elliot is a comprehensive recommendation framework that analyzes the recommendation problem from the researcher's perspective.
It conducts a whole experiment, from dataset loading to results gathering.
The core idea is to feed the system with a simple and straightforward configuration file that drives the framework 
through the experimental setting choices.
Elliot untangles the complexity of combining splitting strategies, hyperparameter model optimization, model training, 
and the generation of reports of the experimental results.

!system schema

The framework loads, filters, and splits the data considering a vast set of strategies (splitting methods and filtering 
approaches, from temporal training-test splitting to nested K-folds Cross-Validation).
Elliot optimizes hyperparameters for several recommendation algorithms, selects the best models, compares them with the 
baselines providing intra-model statistics, computes metrics spanning from accuracy to beyond-accuracy, bias, and fairness, 
and conducts statistical analysis (Wilcoxon and Paired t-test).

Elliot aims to keep the entire experiment reproducible and put the user in control of the framework.

We did our best to put all the bibliographic information of the methods, techniques, and metrics available in Elliot to give the right credits to their authors. Please, remember to cite them when you use Elliot in your research.

Actually, the same holds also for Elliot :-) If you find Elliot useful for your research or development, remember to cite the following paper:

bibtex
@inproceedings{DBLP:conf/sigir/AnelliBFMMPDN21,
  author    = {Vito Walter Anelli and
               Alejandro Bellog{\'{\i}}n and
               Antonio Ferrara and
               Daniele Malitesta and
               Felice Antonio Merra and
               Claudio Pomo and
               Francesco Maria Donini and
               Tommaso Di Noia},
  editor    = {Fernando Diaz and
               Chirag Shah and
               Torsten Suel and
               Pablo Castells and
               Rosie Jones and
               Tetsuya Sakai},
  title     = {Elliot: {A} Comprehensive and Rigorous Framework for Reproducible
               Recommender Systems Evaluation},
  booktitle = {{SIGIR} '21: The 44th International {ACM} {SIGIR} Conference on Research
               and Development in Information Retrieval, Virtual Event, Canada, July
               11-15, 2021},
  pages     = {2405--2414},
  publisher = {{ACM}},
  year      = {2021},
  url       = {https://doi.org/10.1145/3404835.3463245},
  doi       = {10.1145/3404835.3463245},
  timestamp = {Thu, 15 Jul 2021 15:30:48 +0200},
  biburl    = {https://dblp.org/rec/conf/sigir/AnelliBFMMPDN21.bib},
  bibsource = {dblp computer science bibliography, https://dblp

Top issues

Latest release

ELLIOT v0.3.1 Release Notes

 [v0.3.1] - 2021-07-05
 Changed
- fix AMF: insert of perturbation evaluation, insert multi-step adversarial perturbations
- fix AMR:  insert of perturbation evaluation, insert multi-step adversarial perturbations, improved the data pipeline
- start experiments with command line parameters (python start_experiments.py --conf name_config_file)
- released configuration files for DEMO paper (Under review at RecSys 2021)
- readme file in /data/ directory with links to datasets