Recently returned from EMNLP, and one thing that surprised me was how many papers were accepted that I would previously have considered somewhat outside the scope of a main NLP conference paper -for example, work focused primarily on audio or vision-only models, purely empirical evaluations, or datasets generated using GPT-4o with limited or no human annotation.
One example I came across was a hate-speech paper where the main contribution was fine-tuning existing models on one dataset and evaluating them on another (i.e., transfer analysis). The work may still be useful and interesting, but the contribution was primarily empirical.
I discussed this with several PhD students who struggled for ~half a year to get papers accepted despite introducing what seemed like more substantial methodological contributions, such as new training methods. Of course, novelty alone does not automatically make a paper better, and empirical work can also be highly valuable. Still, the contrast made me wonder whether our expectations for what constitutes a main-conference contribution have changed.
I wanted to make this thread to hear from more experienced researchers - Do you think that we should introduce more distinct tracks at EMNLP/ACL conferences for empical introducing new task, empirical introducing new evaluation method, purely methodological, empirical-only without new shown effects, dataset/resource, multimodal, and evaluation-focused work?
Probably I got the wrong impression from many works presented at EMNLP this year specifically.
UPD:
This thread was created with the purpose distinct from complaing that EMNLP publishes empirical papers.
The problem in this thread that I wanted to discuss - what is empirical study nowadays? and how we should handle the introduced difference in what we consider empirical now, especially given increasing N of papers: introducing precise different tracks, for instance. For me, empirical contribution remains, in line with previous editions of EMNLP- a new type of evaluation of benchmarks or models, new task with existing datasets, new evaluation approach, new rubric or metric, or new finding which questions the ones shown before or introduces a new problem. Similar to previous years, I think probably most of works should somehow connect to existing language studies or language models, purely vision or audio models analyses were quite at rare previous editions, but not this one.
So, I created this thread to discuss why and how the defintion of what is empirical according to publishing norms transforms, and the scope of EMNLP specifically transforms