Here at Mr. DLib we are happy to share our data (as long as the privacy of our partners and users is ensured), and allow external researchers and students to conduct their research with Mr. DLib. Mr. DLib is an ideal environment for research about recommender systems and digital libraries as well as research in the field of machine learning, citation analysis, natural language processing and several related disciplines.

Research Fields

Our primary research focus lies on:

  • Applications
    • Recommender Systems
    • Recommender-System Evaluation
    • User Modelling
    • Personalization
  • Techniques
    • Text Mining
    • Machine Learning
    • Natural Language Processing
    • Machine Translation
    • (Social) Network and Graph Analysis
    • Citation Analysis

In addition, we are interested in:

  • Digital Libraries
  • (Academic) Search Engine Optimization
  • PDF Processing
  • Information Extraction
  • Author name and citation disambiguation
  • Document Analysis
  • Web Crawling
  • Plagiarism detection
  • Trusted Timestamping

And as we are running a real-world recommender system, the following areas are of interest to us, too:

  • Big Data / Data Science
  • High Performance Computing
  • Distributed File Storage
  • Database (optimization)

Datasets

Our goal is to publish a dataset once a year that contains information about the recommendations we delivered to our partners. The information includes when and on which platform a recommendation was shown, which recommendation algorithm was used, if and when a recommendation was clicked, on which position a recommendation was displayed, how long it took to create the recommendation, and many many details more. This data will allow you to conduct analyses about the effectiveness of different recommendation algorithms, and running a recommender system in general. You may also use the dataset to calculate the relatedness of our documents to each other and use this information in your own recommender system. Finally, you could also use such a dataset to conduct offline evaluation of your own recommendation algorithms.

We expect to publish the first dataset in summer/spring 2017. If you are interested in analyzing the dataset, or helping us to create it, please contact us (actually, we already have all this data – we just need to anonymize, process, and publish it).

In addition, we publish datasets with the data that we used for our publications. You find them on Haravard’s Dataverse.

Living Evaluation Lab

We plan on opening Mr. DLib so researchers can integrate their own recommendation algorithms in our system, and evaluate their algorithm’s effectiveness. However, such a “living” evaluation lab is a huge project, and we cannot yet say when we will be able to offer it. If you are interested in either using Mr. DLib to evaluate your recommendation algorithms, or if you are interested in helping us to develop a living lab, please contact us.

Publications (by Year)

2017

Beel, Joeran, Bela Gipp, and Akiko Aizawa. “Mr. DLib: Recommendations-as-a-Service (RaaS) for Academia.” In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), 2017.

Beel, Joeran. “Real-World Recommender Systems for Academia: The Gain and Pain in Developing, Operating, and Researching them.” In 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR) at the 39th European Conference on Information Retrieval (ECIR), 2017. [short version, official], [long version, arxiv]

Gipp, Bela, Corinna Breitinger, Norman Meuschke, and Joeran Beel. “CryptSubmit: Introducing Securely Timestamped Manuscript Submission and Peer Review Feedback using the Blockchain.” In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), 2017.

Beierle, Felix, Akiko Aizawa, and Joeran Beel. “Exploring Choice Overload in Related-Article Recommendations in Digital Libraries.” In 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR) at the 39th European Conference on Information Retrieval (ECIR), 2017.

Siebert, Sophie, Siddarth Dinesh, and Stefan Feyer. “Extending a Research Paper Recommendation System with Bibliometric Measures.” In 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR) at the 39th European Conference on Information Retrieval (ECIR), 2017.

Langer, Stefan, and Joeran Beel. “Apache Lucene as Content-Based-Filtering Recommender System: 3 Lessons Learned.” In 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR) at the 39th European Conference on Information Retrieval (ECIR), 2017.

Beel, Joeran, Corinna Breitinger, and Stefan Langer. “Evaluating the CC-IDF citation-weighting scheme: How effectively can ‘Inverse Document Frequency’ (IDF) be applied to references?” In Proceedings of the 12th iConference. 2017.

Beel, Joeran, Stefan Langer, and Bela Gipp. “TF-IDuF: A Novel Term-Weighting Scheme for User Modeling based on Users’ Personal Document Collections.” In Proceedings of the 12th iConference. 2017.

Gipp, Bela, Norman Meuschke, Joeran Beel, and Corinna Breitinger. “Using the Blockchain of Cryptocurrencies for Timestamping Digital Cultural Heritage”. Bulletin of IEEE Technical Committee on Digital Libraries (TCDL). 2017.

Beel, Joeran, Siddharth Dinesh, Philipp Mayr, Zeljko Carevic, and Jain Raghvendra. “Stereotype and Most-Popular Recommendations in the Digital Library Sowiport” In Proceedings of the 15th International Symposium of Information Science. 2017.

Feyer, Stefan, Sophie Siebert, Bela Gipp, Akiko Aizawa, and Joeran Beel. “Integration of the Scientific Recommender System Mr. DLib into the Reference Manager JabRef”. In Proceedings of the 39th European Conference on Information Retrieval (ECIR). 2017.

2016

Beel, Joeran, Corinna Breitinger, Stefan Langer, Andreas Lommatzsch, and Bela Gipp. Towards Reproducibility in Recommender-Systems Research. User Modeling and User-Adapted Interaction (UMUAI) 26, no. 1 (2016): 69–101. DOI: http://dx.doi.org/10.1007/s11257-016-9174-x

Beel, Joeran, Bela Gipp, Stefan Langer, and Corinna Breitinger. “Research Paper Recommender Systems: A Literature Survey.” International Journal on Digital Libraries (2015):1–34. doi:10.1007/s00799-015-0156-0. 2016

Weiler, Andreas and Beel, Joeran and Gipp, Bela and Grossniklaus, Michael. 2016. Stability Evaluation of Event Detection Techniques for Twitter. Proceedings of the 15th International Symposium on Intelligent Data Analysis (IDA’16), Lecture Notes in Computer Science (LNCS), 368–380, doi://10.1007/978-3-319-46349-0.

2015

Beel, Joeran, and Stefan Langer. “A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems.” In Proceedings of the 19th International Conference on Theory and Practice of Digital Libraries (TPDL), edited by Sarantos Kapidakis, Cezary Mazurek, and Marcin Werla, 9316:153–168. Lecture Notes in Computer Science, 2015.

Beel, Joeran, Stefan Langer, Georgia M. Kapitsaki, Corinna Breitinger, and Bela Gipp. “Exploring the Potential of User Modeling based on Mind Maps.” In Proceedings of the 23rd Conference on User Modelling, Adaptation and Personalization (UMAP). Lecture Notes of Computer Science. Springer, 2015.

Beel, Joeran. “Towards Effective Research-Paper Recommender Systems and User Modeling based on Mind Maps.” PhD thesis. Otto-von-Guericke University Magdeburg, March 2015.

2014

Langer, Stefan, Joeran Beel. “The Comparability of Recommender System Evaluations and Characteristics of Docear’s Users.” In Proceedings of the Workshop on Recommender Systems Evaluation: Dimensions and Design (REDD), at the ACM Recommender Systems Conference (RecSys). 2014.

Beel, Joeran, Stefan Langer, Bela Gipp, Andreas Nürnberger. “The Architecture and Datasets of Docear’s Research Paper Recommender System.” D-Lib Magazine – The Magazine of Digital Library Research, vol. 20, 11/12, 2014.

Beel, Joeran, Stefan Langer, Marcel Genzmehr, Bela Gipp. “Utilizing Mind-Maps for Information Retrieval and User Modelling.” In Proceedings of the 22nd Conference on User Modelling, Adaption, and Personalization (UMAP). Springer, 2014.

2013

Beel, Joeran, Stefan Langer, Marcel Genzmehr, Bela Gipp, Corinna Breitinger, and Andreas Nürnberger. “Research Paper Recommender System Evaluation: A Quantitative Literature Survey.” In Proceedings of the Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys) at the ACM Recommender System Conference (RecSys). ACM International Conference Proceedings Series (ICPS), 2013.

Beel, Joeran, Stefan Langer, Marcel Genzmehr, Bela Gipp, and Andreas Nürnberger. “A Comparative Analysis of Offline and Online Evaluations and Discussion of Research Paper Recommender System Evaluation.” In Proceedings of the Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys) at the ACM Recommender System Conference (RecSys). ACM International Conference Proceedings Series (ICPS), 2013.

Beel, Joeran, Stefan Langer, Marcel Genzmehr, and Andreas Nürnberger. “Introducing Docear’s Research Paper Recommender System.” In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13), 459–460. ACM, 2013.

Beel, Joeran, Stefan Langer, Marcel Genzmehr, and Andreas Nürnberger. “Persistence in Recommender Systems: Giving the Same Recommendations to the Same Users Multiple Times.” In Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), edited by Trond Aalberg, Milena Dobreva, Christos Papatheodorou, Giannis Tsakonas, and Charles Farrugia, 8092:390–394. Lecture Notes of Computer Science (LNCS). Valletta, Malta: Springer, 2013.

Beel, Joeran, Stefan Langer, Andreas Nürnberger, and Marcel Genzmehr. “The Impact of Demographics (Age and Gender) and Other User Characteristics on Evaluating Recommender Systems.” In Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), edited by Trond Aalberg, Milena Dobreva, Christos Papatheodorou, Giannis Tsakonas, and Charles Farrugia, 400–404. Valletta, Malta: Springer, 2013.

Beel, Joeran, Stefan Langer, and Marcel Genzmehr. “Docear4Word: Reference Management for Microsoft Word based on BibTeX and the Citation Style Language (CSL).” In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13), 445–446. ACM, 2013.

Beel, Joeran, Stefan Langer, Marcel Genzmehr, and Christoph Müller. “Docears PDF Inspector: Title Extraction from PDF files.” In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13), 443–444. ACM, 2013.

Beel, Joeran, Stefan Langer, and Marcel Genzmehr. “Sponsored Vs. Organic (Research Paper) Recommendations and the Impact of Labeling.” In Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), edited by Trond Aalberg, Milena Dobreva, Christos Papatheodorou, Giannis Tsakonas, and Charles Farrugia, 395–399. Valletta, Malta, 2013.

Lipinski, Mario, Kevin Yao, Joeran Beel, and Bela Gipp. “Evaluation of Header Metadata Extraction Approaches and Tools for Scientific PDF Documents.” In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries (JCDL’13), 385–386, 2013.

2011

Joeran Beel, Bela Gipp, Stefan Langer, and Marcel Genzmehr. “Docear: An Academic Literature Suite for Searching, Organizing and Creating Academic Literature.” In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, JCDL ’11, pages 465–466, New York, NY, USA, 2011. ACM.

Joeran Beel and Stefan Langer. “An Exploratory Analysis of Mind Maps.” In Proceedings of the 11th ACM Symposium on Document Engineering (DocEng’11), Mountain View, California, USA, pages 81-84 2011. ACM.

Bela Gipp, Norman Meuschke, and Joeran Beel. “Comparative Evaluation of Text- and Citation-based Plagiarism Detection Approaches using GuttenPlag.” In Proceedings of 11th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’11), pages 255–258, Ottawa, Canada, June 2011. ACM.

Joeran Beel, Bela Gipp, Stefan Langer, Marcel Genzmehr, Erik Wilde, Andreas Nürnberger, and Jim Pitman. “Introducing Mr. DLib, a Machine-readable Digital Library.” In Proceedings of the 11th ACM/IEEE Joint Conference on Digital Libraries (JCDL‘11), 2011. ACM.

2010

Joeran Beel, Bela Gipp, and Erik Wilde. “Academic Search Engine Optimization (ASEO): Optimizing Scholarly Literature for Google Scholar and Co.” Journal of Scholarly Publishing, 41 (2): 176–190, January 2010. University of Toronto Press.

Bela Gipp and Joeran Beel. “Citation Based Plagiarism Detection – A New Approach to Identify Plagiarized Work Language Independently.” In Proceedings of the 21st ACM Conference on Hyptertext and Hypermedia (HT’10), pages 273–274, New York, NY, USA, June 2010. ACM.

Joeran Beel, Bela Gipp, Ammar Shaker, and Nick Friedrich. “SciPlore Xtract: Extracting Titles from Scientific PDF Documents by Analyzing Style Information (Font Size).” In M. Lalmas, J. Jose, A. Rauber, F. Sebastiani, and I. Frommholz, editors, Research and Advanced Technology for Digital Libraries, Proceedings of the 14th European Conference on Digital Libraries (ECDL’10), volume 6273 of Lecture Notes of Computer Science (LNCS), pages 413–416, Glasgow (UK), September 2010. Springer.

Bela Gipp, Adriana Taylor, and Joeran Beel. “Link Proximity Analysis – Clustering Websites by Examining Link Proximity.” In M. Lalmas, J. Jose, A. Rauber, F. Sebastiani, and I. Frommholz, editors, Proceedings of the 14th European Conference on Digital Libraries (ECDL’10): Research and Advanced Technology for Digital Libraries, volume 6273 of Lecture Notes of Computer Science (LNCS), pages 449–452, September 2010. Springer.

Joeran Beel and Bela Gipp.“On the Robustness of Google Scholar Against Spam.” In Proceedings of the 21st ACM Conference on Hyptertext and Hypermedia (HT’10), pages 297–298, Toronto (CA), June 2010. ACM.

Joeran Beel and Bela Gipp. “Enhancing Information Search by Utilizing Mind Maps.” In Proceedings of the 21st ACM Conference on Hyptertext and Hypermedia (HT’10), pages 303–304, Toronto (CA), June 2010. ACM.

Joeran Beel and Bela Gipp. “Academic search engine spam and Google Scholar’s resilience against it.” Journal of Electronic Publishing, 13 (3), December 2010.

2009

Bela Gipp and Joeran Beel. “Citation Proximity Analysis (CPA) – A new approach for identifying related work based on Co-Citation Analysis.” In Birger Larsen and Jacqueline Leta, editors, Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI’09), volume 2, pages 571–575, Rio de Janeiro (Brazil), July 2009. International Society for Scientometrics and Informetrics.

Joeran Beel and Bela Gipp. “Google Scholar’s Ranking Algorithm: The Impact of Citation Counts (An Empirical Study). In Andre Flory and Martine Collard, editors, Proceedings of the 3rd IEEE International Conference on Research Challenges in Information Science (RCIS’09), pages 439–446, Fez (Morocco), April 2009. IEEE.

Joeran Beel and Bela Gipp. “Google Scholar’s Ranking Algorithm: An Introductory Overview.” In Birger Larsen and Jacqueline Leta, editors, Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI’09), volume 1, pages 230–241, Rio de Janeiro (Brazil), July 2009. International Society for Scientometrics and Informetrics. ISSN 2175-1935.

Joeran Beel, Bela Gipp, and Jan-Olaf Stiller. “Information Retrieval on Mind Maps – What could it be good for?”In Proceedings of the 5th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom’09), pages 1–4, Washington (USA), November 2009. IEEE.

Joeran Beel and Bela Gipp. “Google Scholar’s Ranking Algorithm: The Impact of Articles’ Age (An Empirical Study).” In Shahram Latifi, editor, Proceedings of the 6th International Conference on Information Technology: New Generations (ITNG’09), pages 160–164, Las Vegas (USA), April 2009. IEEE.

Joeran Beel, Bela Gipp, and Christoph Mueller. “’SciPlore MindMapping’ – A Tool for Creating Mind Maps Combined with PDF and Reference Management.” D-Lib Magazine, 15 (11), November 2009. Brief Online Article.

2008

Joeran Beel and Bela Gipp. “The Potential of Collaborative Document Evaluation for Science.” In George Buchanan, Masood Masoodian, and Sally Jo Cunningham, editors, 11th International Conference on Asia-Pacific Digital Libraries (ICADL’08) Proceedings, volume 5362 of Lecture Notes in Computer Science (LNCS), pages 375–378, Heidelberg (Germany), December 2008. Springer.

Joeran Beel and Bela Gipp. “Collaborative Document Evaluation: An Alternative Approach to Classic Peer Review.” In Proceedings of the 5th International Conference on Digital Libraries (ICDL’08), volume 31, pages 410–413, Vienna (Austria), August 2008. ISSN 1307-6884.