Dr. Diyah Puspitaningrum

Welcome to my homepage!

I am an independent researcher in the Computer Science field with Data Mining Background. In the last eight years, I also get deeply involved in Information Retrieval researches. NLP (Natural Language Processing) is my specialization.

One of my latest researches (published 2022) is about abstract summarization using very recent techniques (ProphetNet, Pegasus, and T5).

Another research (June 2021) is a study about machine translations (Seq2Seq, Convolutional Seq2Seq, RNN, MHA) and their fine-tuning strategies. 

ABSTRACT SUMMARIZATION TECHNIQUES

A Survey of Recent Abstract Summarization Techniques
Diyah Puspitaningrum

Abstract 

This paper surveys several recent abstract summarization methods: T5, Pegasus, and ProphetNet. We implement the systems in two languages: English and Indonesian languages. We investigate the impact of pre-training models (one T5, three Pegasuses, three ProphetNets) on several Wikipedia datasets in English and Indonesian language and compare the results to the Wikipedia systems' summaries. The T5-Large, the Pegasus-XSum, and the ProphetNet-CNNDM provide the best summarization. The most significant factors that influence ROUGE performance are coverage, density, and compression. The higher the scores, the better the summary. Other factors that influence the ROUGE scores are the pre-training goal, the dataset's characteristics, the dataset used for testing the pre-trained model, and the cross-lingual function. Several suggestions to improve this paper's limitation are: 1) assure that the dataset used for the pre-training model must sufficiently large, contains adequate instances for handling cross-lingual purpose; 2) Advanced process (finetuning) shall be reasonable. We recommend using the large dataset consists of comprehensive coverage of topics from many languages before implementing advanced processes such as the train-infer-train procedure to the zero-shot translation in the training stage of the pre-training model.

Keywords: 
abstract summarization, T5, Pegasus, ProphetNet, train-infer-train, cross-lingual system, Transformers.
 

NEURAL MACHINE TRANSLATION

A Study of English-Indonesian Neural Machine Translation With Attention (Seq2Seq, ConvSeq2Seq, RNN, and MHA)
Diyah Puspitaningrum

Abstract

In recent years, Neural Machine Translation (NMT) with attention mechanisms has emerged in research and industry. This study discusses the essentials of NMT (Seq2Seq, Convolutional Seq2Seq (ConvSeq2Seq), Recurrent Neural Networks (RNN), and Multi-Head Attention (MHA)) while implemented in formal passages in English-Indonesian and Indonesian-English. The experimental results for ConvSeq2Seq achieve up to 38.99 BLEU sentence scores, 43.23 BLEU corpus scores, and 39.48 GLEU corpus scores over the Seq2Seq English-Indonesian. For Indonesian-English, the results for ConvSeq2Seq achieved as follows: up to 42.59 BLEU sentence scores, 42.91 BLEU corpus scores, 41.05 GLEU corpus scores, and 1356.65 WER scores over RNN and MHA. Thus, while ConvSeq2Seq tends to be the supremacy, this literature also describes the combination of architectures and specific fine-tuning strategies as a discussion. 

CCS CONCEPTS: Computing methodologies, Artificial intelligence, Natural language processing
Additional Keywords and Phrases: self-attention, RNN, convolutional Seq2Seq, MHA, machine translation
 

IMPROVING PERFORMANCE OF RELATION EXTRACTION ALGORITHM VIA LEVELED ADVERSARIAL PCNN AND DATABASE EXPANSION

Improving Performance of Relation Extraction Algorithm via Leveled Adversarial PCNN and Database Expansion

Diyah Puspitaningrum

Abstract

This study introduces database expansion using the Minimum Description Length (MDL) algorithm to expand the database for better relation extraction. Different from other previous relation extraction researches, our method improves system performance by expanding data. The goal of database expansion, together with a robust deep learning classifier, is to diminish wrong labels due to the incomplete or not found nature of relation instances in the relation database (e.g., Freebase). The study uses a deep learning method (Piecewise Convolutional Neural Network or PCNN) as the base classifier of our proposed approach: the leveled adversarial attention neural networks (LATTADV-ATT). In the database expansion process, the semantic entity identification is used to enlarge new instances using the most similar itemsets of the most common patterns of the data to get its pairs of entities. About the deep learning method, the use of attention of selective sentences in PCNN can reduce noisy sentences. Also, the use of adversarial perturbation training is useful to improve the robustness of system performance. The performance even further is improved using a combination of leveled strategy and database expansion. There are two issues: 1) database expansion method: rule generation by allowing step sizes on selected strong semantic of most similar itemsets with aims to find entity pair for generating instances, 2) a better classifier model for relation extraction. Experimental result has shown that the use of the database expansion is beneficial. The MDL database expansion helps improvements in all methods compared to the unexpanded method. The LATTADV-ATT performs as a good classifier with high precision P@100=0.842 (at no expansion). It is even better while implemented on the expansion data with P@100=0.891 (at expansion factor k=7). 

Keywords: relation extraction, database expansion, MDL, PCNN, classification