PromptEng'24 - Workshop on Prompt Engineering for Pre-Trained Language Models

The recent achievements and availability of Large Language Models has paved the road to a new range of applications and use-cases. Pre-trained language models are now being involved at-scale in many fields where they were until now absent from. More specifically, the progress made by causal generative models has opened the door to using them through textual instructions aka. prompts. Unfortunately, the performances of these prompts are highly dependent on the exact phrasing used and therefore practitioners need to adopt fail-retry strategies.

In a nutshell, PromptEng provides the research community with a forum to discuss, exchange and design advanced prompting techniques for LLM applications.

This first international workshop on prompt engineering aims at gathering practitioners (both from Academia and Industry) to exchange good practices, optimizations, results and novel paradigms about the designing of efficient prompts to make use of LLMs.

Undoubtedly, the recent Large Language Models (LLMs) are becoming more and more omnipotent in many tasks. Different sub-fields from the Semantic Web such as Knowledge Graph construction, knowledge verbalization, Web pages summarization have considerably benefited from such a prompting mechanism. The ability to query and interact with them using prompts is crucial to generate high-quality output in the desired format. While existing contributions have been made towards prompt engineering, several difficulties and challenges remain to gain a better understanding of how those LLMs respond to different prompts. Typically, the way instructions are conveyed in prompts can lead to either distinct or similar output from the models.

Moreover, some instructions are better respected while others are simply ignored for some tasks. So far, LLM-practitioners have been mainly working on their own, developing and testing bespoke techniques to achieve their goals, re-starting the prompt-design tasks for each new model they have been using. Such an approach often leads to tackle problems which have already been explored by other researchers.

This workshop aims to investigate and analyze these behaviors, through experimental analysis and probing of LLMs, in order to gain insights into the models' sensitivity to different prompts. By uncovering significant findings, the community can greatly benefit in utilizing LLMs more effectively while also preventing the generation of harmful content. Ultimately, this workshop endeavors to compile and index successful and unavailing prompts with respect to both tasks and models.

Topics of interest include, but are not limited to themes related to the techniques of prompt engineering:

• Prompts & Chain-of-Thought Prompts Design
• Theoretical and Experimental Analysis of Prompting
• Prompts Transferability
• Specific prompt techniques for Web crawling
• Ontology generation combining LLM and Web data
• Semantic and Syntactic comparison of prompt performances
• Structured Prediction with Prompts
• Prompt Retrieval and Generation
• Visualization with Prompt Techniques

We envision five types of submissions covering the entire workshop topics spectrum:

1. Research Papers (max 10 pages), presenting novel scientific research addressing topics of the workshop.
2. Position & Demo papers (max 5 pages), encouraging papers describing significant work in progress, late breaking results or ideas of the domain, as well as functional systems relevant to the community.
3. Industry & Use Case Presentations (max 5 pages), in which industry experts can present and discuss practical solutions, use case prototypes, best practices, etc. at any stage of implementation.
4. Expression of Interest (max 2 pages), presenting a research topic, a work in progress, practical applications or needs, etc.
5. Technical prompting technique (max 2 pages), describing practically a prompt together with a minimal working example and an associated use-case motivating it.

In order to ease the reviewing process, authors may add the track they are submitting to directly in their titles, for instance: "Article Title [Industry]".

Submissions must be in double-column format, and must adhere to the ACM template and format (also available in Overleaf). The recommended setting for LaTeX is: \documentclass[sigconf, anonymous, review]{acmart}. The PDF files must have all non-standard fonts embedded. Workshop submissions must be self-contained and in English. Note: The review process is single-blind, no need for authors to submit anonymous articles.

All papers should be submitted to https://easychair.org/conferences/?conf=thewebconf2024_workshops.

• Submission: February 5^th, 2024 February 9^th, 2024
• Notification: March 4^th, 2024
• Camera-ready: March 11^th, 2024
• Presentation: May 14^th, 2024

Attending the workshop

PromptEng will take place on 14^th of May 2024 afternoon from 1:30pm to 5:00pm. All hours are local Singapore time.
(In the meantime, please, don't forget to register and attend... ☺)

Keynote #1

Title: Retrieval Augmented Generation for AI Search
Dr. Yong Liu, Senior Researcher, Huawei Noah's Ark Lab, Singapore
https://www.yongliu.org/

Abstract: Retrieval augmented generation (RAG) has become a promising solution to help large language models incorporate external knowledge from Web search engines or local knowledge databases to answer real-time and factual questions and solve the hallucination problems. Moreover, RAG techniques have also been exploited to build AI search services, e.g., Perplexity AI and New Bing, based on large language models. In this talk, we will first introduce some background about existing AI search services and the challenges of applying RAG to build AI search systems. Next, we will review the state-of-the-art RAG techniques, and share some hands-on experiences in building the AI search systems. Finally, we will have some discussions about some future potential research directions about applying RAG to build AI search systems.

Bio: Dr. Yong Liu is currently a Senior Researcher at Huawei Noah's Ark Lab, Singapore. Prior to joining Huawei, he was a Senior Research Scientist at Nanyang Technological University (NTU), a Data Scientist at NTUC Enterprise, and a Research Scientist at Institute for Infocomm Research (I2R), A*STAR, Singapore. He received his Ph.D. degree in Computer Engineering from NTU in 2016 and B.S. degree in Electronic Science and Technology from University of Science and Technology of China (USTC) in 2008. His current research interests include Large Language Models, Search and Recommendation Systems. He has been invited as an Area Chair/(Senior) PC member of major conferences such as ICLR, NeurIPS, KDD, WWW, SIGIR, ACL, IJCAI, AAAI, and reviewer for IEEE/ACM transactions.

Keynote #2

Title: On Understanding the Reasoning Capabilities of Language Models
Prof. Wei Lu, Associate Professor and Associate Head of Pillar, SUTD, Singapore
https://istd.sutd.edu.sg/people/faculty/lu-wei

Abstract: In this talk, I will share some of our recent endeavors that center around the following research questions, with the goal of gaining a better understanding of the reasoning capabilities of language models.
- How do we (better) unlock the reasoning capabilities in language models: The chain-of-thought (CoT) prompting method demonstrates language models’ capabilities to carry out step-by-step reasoning. We argue that language models may be able to perform structured multi-dimensional reasoning, which is demonstrated by our Tab-CoT prompting mechanism. We also discuss how this may serve as a step towards better understanding the emergent behaviours of language models, which is one of the most fundamental research problems in language models and is also related to the next research question.
- How reasoning capabilities emerge in language models: to answer this question, we aim to first investigate the pre-training stage of language models. One of our current projects is to build effective yet relatively small language models. I will elaborate on the significance of such a direction, what problems (especially those related to reasoning) we aim to address, how such models may benefit the community, and what their practical implications could be.

Bio: Wei Lu is currently an Associate Professor and Associate Head (Research) of the Information Systems Technology and Design Pillar at the Singapore University of Technology and Design (SUTD). He is also the Director of the StatNLP Research Group, which focuses on fundamental research in Natural Language Processing (NLP) and Language Models. He received the Best Paper Award at EMNLP 2011, an Outstanding Paper Award at EMNLP 2023, and an Area Chair’s Award at ACL 2023. He is the Editor-in-Chief of Computational Linguistics.

Accepted Articles

• Automatic design summary generation with generative AI by Daisuke Ikoma, Eisuke Aoki, Shinya Suzuki, Tomoki Taniguchi and Tomoko Ohkuma
• English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts by Patrick Barreiß, Roman Klinger and Jeremy Barnes
• Data Augmentation for Smishing Detection: A Theory-based Prompt Engineering Approach by Ho Sung Shim, Hyoungjun Park, Kyuhan Lee, Jang-sun Park and Seonhye Kang
• Prompt-Eng: Healthcare Prompt Engineering: Revolutionizing Healthcare Applications with Precision Prompts by Awais Ahmed, Mengshu Hou, Rui Xi, Xiaoyang Zeng and Syed Attique Shah
• Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems by Anuja Tayal and Aman Tyagi

Agenda

Organisers

• Damien Graux (Huawei Ltd., UK) is a principal research scientist at the Huawei Research Center. He has been contributing to research efforts in Semantic Web technologies: focusing on query evaluation and designing complex pipelines for heterogeneous Big Data. Prior, he had research positions at Inria (France), Trinity College Dublin (Ireland) and Fraunhofer IAIS (Germany).
• Sebastien Montella (Huawei Ltd., UK) is a research scientist at the Huawei Edinburgh Research Center. During his Ph.D., he specialized in Natural Language Generation and Knowledge Graph Embeddings research areas. Additionally, he has a keen interest in statistical learning, geometric deep learning, natural language processing, and computer vision. In the past, Sebastien as co-organized the 18th Workshop on Spoken Dialogue Systems for PhDs, PostDocs & New Researchers (YRRSDS) in Edinburgh, Scotland (2022).
• Hajira Jabeen (GESIS, Germany) leads the 'Big Data Analytics' research team within the Knowledge Technologies for Social Sciences (KTS) department at GESIS. Her team is dedicated to conducting research involving natural language processing, knowledge graphs, distributed analytics, and big data techniques. She has a background in both research and teaching, with previous affiliations at the University of Bonn, the University of Cologne, and ITU Copenhagen. She has previously organized several workshops and conferences.
• Claire Gardent (CNRS/LORIA, France) is a senior research scientist at the French National Center for Scientific Research (CNRS), based at the LORIA Computer Science research unit in Nancy, France. She works in the field of Natural Language Processing with a particular interest for Natural Language Generation. In 2017, she launched the WebNLG challenge, a shared task where the goal is to generate text from Knowledge Base fragments. She has proposed neural models for simplification and summarisation; for the generation of long form documents such as multi-document summaries and Wikipedia articles; for multilingual generation from Abstract Meaning Representations and for response generation in dialog. She currently heads the AI XNLG Chair on multi-lingual, multi-source NLG and the CNRS LIFT Research Network on Computational, Formal and Field Linguistics. In 2022, she was awarded the CNRS Silver Medal and was selected as ACL (Association of Computational Linguistics) Fellow.
• Jeff Z. Pan (University of Edinburgh, UK) is a chair of the Knowledge Graph Group at the Alan Turing Institute and is a member of the School of Informatics at the University of Edinburgh. He received his Ph.D. in Computer Science from The University of Manchester in 2004. He joined the faculty in the Department of Computing Science at The University of Aberdeen in 2005, where he later became the Leader of the Knowledge Technology group and the Director of the Joint Research Lab on Knowledge Engineering and Information Security. He joined Informatics from 2020 and is a member of ILCC.

Program Committee

Important Dates

• Submission (EasyChair): February 9^th, 2024
• Notification: March 1^st, 2024
• Camera-ready: March 8^th, 2024
• Presentation: May 14^th, 2024