Advances of neural networks in 2020

Advances of neural networks in 2020

• 1.
  Advances of NeuralNetworks in 2020 International Journal on Natural Language Computing(IJNLC) ISSN : 2278 - 1307 [Online]; 2319 - 4111 [Print] http://airccse.org/journal/ijnlc/index.html
• 2.
  RESUME INFORMATION EXTRACTIONWITH A NOVEL TEXT BLOCK SEGMENTATION ALGORITHM Shicheng Zu and Xiulai Wang Post-doctoral Scientific Research Station in East War District General Hospital, Nanjing, Jiangsu 210000, China ABSTRACT In recent years, we have witnessed the rapid development of deep neural networks and distributed representations in natural language processing. However, the applications of neural networks in resume parsing lack systematic investigation. In this study, we proposed an end-to- end pipeline for resume parsing based on neural networks-based classifiers and distributed embeddings. This pipeline leverages the position-wise line information and integrated meanings of each text block. The coordinated line classification by both line type classifier and line label classifier effectively segment a resume into predefined text blocks. Our proposed pipeline joints the text block segmentation with the identification of resume facts in which various sequence labelling classifiers perform named entity recognition within labelled text blocks. Comparative evaluation of four sequence labelling classifiers confirmed BLSTMCNNs-CRF’s superiority in named entity recognition task. Further comparison among three publicized resume parsers also determined the effectiveness of our text block classification method. KEYWORDS Resume Parsing, Word Embeddings, Named Entity Recognition, Text Classifier, Neural Networks. Full Text : http://aircconline.com/abstract/ijnlc/v8n5/8519ijnlc03.html
• 3.
  REFERENCES [1] Peng Zhou,Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu (2016) “Attention-based Bidirectional Long Short-term Memory Networks for Relation Classification”, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16), Berlin, Germany, August 7-12, 2016, pp 207-212. [2] Xuezhe Ma, & Eduard Hovy (2016) “End-to-End Sequence Labelling via Bi-directional LSTMCNNs-CRF”, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16), Berlin, Germany, August 7-12, 2016, pp 1064-1074. [3] Kun Yu, Gang Guan, and Ming Zhou (2005) “Resume Information Extraction with Cascaded Hybrid Model” In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), Stroudsburg, PA, USA, June 2005, pp 499-506. [4] Jie Chen, Chunxia Zhang, and Zhendong Niu (2018) “A Two-Step Resume Information Extraction Algorithm” Mathematical Problems in Engineering pp1-8. [5] Jie Chen, Zhendong Niu, and Hongping Fu (2015) “A Novel Knowledge Extraction Framework for Resumes Based on Text Classifier” In: Dong X., Yu X., Li J., Sun Y. (eds) Web-Age Information Management (WAIM 2015) Lecture Notes in Computer Science, Vol. 9098, Springer, Cham. [6] Hui Han, C. Lee Giles, Eren Manavoglu, HongYuan Zha (2003) “Automatic Document Metadata Extraction using Support Vector Machine” In Proceedings of the 2003 Joint Conference on Digital Libraries, Houston, TX, USA, pp 37-48. [7] David Pinto, Andrew McCallum, Xing Wei, and W. Bruce Croft (2003) “Table Extraction Using Conditional Random Field” In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, Toronto, Canada, pp 235- 242. [8] Amit Singh, Catherine Rose, Karthik Visweswariah, Enara Vijil, and Nandakishore Kambhatla (2010) “PROSPECT: A system for screening candidates for recruitment” In Proceedings of the 19th ACM international conference on Information and knowledge management, (CIKM’10), Toronto, ON, Canada, October 2010, pp 659-668. International Journal on Natural Language Computing (IJNLC) Vol.8, No.5, October 2019 47 [9] Anjo Anjewierden (2001) “AIDAS: Incremental Logical Structure Discovery in PDF Documents” In Proceedings of 6th International Conference on Document Analysis and Recognition (ICDAR’01) pp 374-378. [10] Sumit Maheshwari, Abhishek Sainani, and P. Krishna Reddy (2010) “An Approach to Extract Special Skills to Improve the Performance of Resume Selection” Databases in Networked Information Systems, Vol. 5999 of Lecture Notes in Computer Science, Springer, Berlin, Germany, 2010, pp 256- 273. [11] Xiangwen Ji, Jianping Zeng, Shiyong Zhang, Chenrong Wu (2010) “Tag tree template for Web information and schema extraction” Expert Systems with Applications Vol. 37, No.12, pp 8492- 8498.
• 4.
  [12] V. SenthilKumaran and A. Sankar (2013) “Towards an automated system for intelligent screening of candidates for recruitment using ontology mapping (EXPERT)” International Journal of Metadata, Semantics and Ontologies, Vol. 8, No. 1, pp 56-64. [13] Fabio Ciravegna (2001) “(LP)2, an Adaptive Algorithm for Information Extraction from Webrelated Texts” In Proceedings of the IJCAI-2001 Workshop on Adaptive Text Extraction and Mining. Seattle, WA. [14] Fabio Ciravegna, and Alberto Lavelli (2004) “LearningPinocchio: adaptive information extraction for real world applications” Journal of Natural Language Engineering Vol. 10, No. 2, pp145- 165. [15] Yan Wentan, and Qiao Yupeng (2017) “Chinese resume information extraction based on semistructure text” In 36th Chinese Control Conference (CCC), Dalian, China. [16] Zhang Chuang, Wu Ming, Li Chun Guang, Xiao Bo, and Lin Zhi-qing (2009) “Resume Parser: Semi-structured Chinese document analysis” In Proceedings of the 2009 WRI World Congress on Computer Science and Information Engineering, Los Angeles, USA, Vol. 5 pp 12-16. [17] Zhixiang Jiang, Chuang Zhang, Bo Xiao, and Zhiqing Lin (2009) “Research and Implementation of Intelligent Chinese resume Parsing” In 2009 WRI International Conference on Communications and Mobile Computing, Yunan, China, Vol. 3 pp 588-593. [18] Duygu Çelik, Askýn Karakas, Gülsen Bal , Cem Gültunca , Atilla Elçi , Basak Buluz, and Murat Can Alevli (2013) “Towards an Information Extraction System based on Ontology to Match Resumes and Jobs” In Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications Conference Workshops, Japan, pp 333-338. [19] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean (2013) “Efficient Estimation of Word Representations in Vector Space” Computer Science, arXiv preprint arxiv:1301.3781. [20] Jeffrey Pennington, Richard Socher, and Christopher D. Manning (2014) “GloVe: Global Vectors for Word Representation” In Empirical Methods in Natural Language Processing (EMNLP) pp 1532- 1543. [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019) “BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding” arxiv:1810.04805. [22] Yoon Kim (2014) “Convolutional Neural Networks for Sentence Classification” In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) pp 1746- 1751. [23] Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao (2005) “Recurrent Convolutional Neural Networks for Text Classification” In Proceedings of Conference of the Association for the Advancement of Artificial Intelligence Vol. 333 pp 2267-2273. [24] Takeru Miyato, Andrew M. Dai, and Ian Goodfellow (2017) “Adversarial Training Methods for Semi-supervised Text Classification” In ICLR 2017.
• 5.
  [25] Ashish Vaswani,Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2017) “Attention Is All You Need” In 31st Conference on Neural Information Processing Systems (NIPS’ 2017), Long Beach, CA, USA. [26] Zoubin Ghahramani, and Michael I. Jordan (1997) “Factorial Hidden Markov Model” Machine Learning Vol. 29 No. 2-3, pp 245-273. [27] Andrew McCallum, Dayne Freitag, and Fernando Pereira (2000) “Maximum Entropy Markov Models for Information Extraction and Segmentation” In Proceedings of the Seventeenth International Conference on Machine Learning (ICML’00) pp 591-598. [28] John Lafferty, Andrew McCallum, and Fernando Pereira (2001) “Conditional Random Fields: Probabilistic Models for Segmenting and Labelling Sequence Data” In Proceedings of the Eighteenth International Conference on Machine Learning (ICML’01) Vol. 3 No. 2, pp 282-289. [29] Zhiheng Huang, Wei Xu, and Kai Yu (2015) “Bidirectional LSTM-CRF Models for Sequence Tagging” arXiv preprint arXiv:1508.01991, 2015
• 6.
  OPTIMIZE THE LEARNINGRATE OF NEURAL ARCHITECTURE IN MYANMAR STEMMER Yadanar Oo and Khin Mar Soe Natural Language Processing Lab, University of Computer Studies, Yangon, Myanmar ABSTRACT Morphological stemming becomes a critical step toward natural language processing. The process of stemming is to reduce alternative forms to a common morphological root. Word segmentation for Myanmar Language, like for most Asian Languages, is an important task and extensively-studied sequence labelling problem. Named entity detection is one of the issues in Asian Language that has traditionally required a large amount of feature engineering to achieve high performance. The new approach is integrating them that would benefit in all these processes. In recent years, end-to-end sequence labelling models with deep learning are widely used. This paper introduces a deep BiGRUCNN-CRF network that jointly learns word segmentation, stemming and named entity recognition tasks. We trained the model using manually annotated corpora. State-of-the-art named entity recognition systems rely heavily on handcrafted feature built in our new approach, we introduce the joint model that relies on two sources of information: character level representation and syllable level representation. KEYWORDS Myanmar word stemmer, Sequence labelling, Conditional random fields, Neural architecture, word segmentation Full Text : http://aircconline.com/ijnlc/V8N5/8519ijnlc04.pdf
• 7.
  REFERENCES [1] Pa. W.P.,N.L.: “Myanmar Word Segmentation using Hybrid Approach.”, presented at ICCA, Yangon, pp.166-170, 2008. [2] Win Win Thant, Tin Myat Htwe and Ni Lar Thein, "Grammatical Relations of Myanmar Sentences Augmented by Transformation Based Learning of Function Tagging", IJCSI International Journal of Computer Science Issues, Vol. 8, Issue 5, No 2, September 2011 [3] W.P.Pa, Y.K.Thu, A.Finch and E.Sumita, “Word Boundary Identification for Myanmar Text Using Conditional Random Field”, Springer, Switzerland, 2016 [4] Thu, Y.K., Finch, A., Sagisaka, Y., Sumita, E.: “A Study of Myanmar Word Segmentation Schemes for Statistical Machine Translation”. In Proceedings of 12th International Conference on Computer Applications, Yangon, Myanmar, pp.167-179, 2014. [5] Jie Yang, Shuailong Liang, and Yue Zhang. “Design challenges and misconceptions in neural sequence labeling”. In COLING, 2018. [6] Jie Yang and Yue Zhang. NCRF++: An Open-source Neural Sequence Labeling Toolkit. arXiv:1806.05626v2[cs.CL] 17 Jun 2018. [7] Nils Reimers and Iryna Gurevych. 2017a. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799. [8] Edouard Grave, Piotr Bojanowski, Prakhar Gupta1, Arma for 157 Languages"arXiv:1802.06893v2, 28 Mar 2018. [9] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arXiv:1702.01417. [10] Xuezhe Ma and Eduard Hovy. "End-to-end sequence labeling via Bidirectional LSTM-CNNsCRF". In ACL. volume 1, pages 1064–1074, 2016 [11] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching Word Vectors withSubword Information.arXiv preprint arXiv:1607.04606. [12] Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2014) (Vol. 14, pp. 1532–1543). Retrieved from https://nlp.stanford.edu/projects/glove/ [13] Zhenyu Jiao, Shuqi Sun, Ke Sun, “Chinese Lexical Analysis with Deep Bi-GRU-CRF Network”. arXiv preprint arXiv:1807.01882. Jul 2018. [14] Y. Shao, C. Hardmeier, J. Tiedemann, J. Nivre. “Character-based joint segmentation and POS tagging for Chinese using bidirectional RNN-CRF.” arXiv preprint arXiv:1704.01314. Apr 2017
• 8.
  SYLLABLE-BASED NEURAL NAMEDENTITY RECOGNITION FOR MYANMAR LANGUAGE Hsu Myat Mo and Khin Mar Soe Natural Language Processing Lab., University of Computer Studies, Yangon, Myanmar ABSTRACT Named Entity Recognition (NER) for Myanmar Language is essential to Myanmar natural language processing research work. In this work, NER for Myanmar language is treated as a sequence tagging problem and the effectiveness of deep neural networks on NER for Myanmar language has been investigated. Experiments are performed by applying deep neural network architectures on syllable level Myanmar contexts. Very first manually annotated NER corpus for Myanmar language is also constructed and proposed. In developing our in-house NER corpus, sentences from online news website and also sentences supported from ALT-Parallel- Corpus are also used. This ALT corpus is one part of the Asian Language Treebank (ALT) project under ASEAN IVO. This paper contributes the first evaluation of neural network models on NER task for Myanmar language. The experimental results show that those neural sequence models can produce promising results compared to the baseline CRF model. Among those neural architectures, bidirectional LSTM network added CRF layer above gives the highest F- score value. This work also aims to discover the effectiveness of neural network approaches to Myanmar textual processing as well as to promote further researches on this understudied language. KEYWORDS Bidirectional LSTM_CRF, Myanmar Language, Named Entity Recognition, Neural architectures, Syllablebased Full Text : http://aircconline.com/ijnlc/V8N1/8119ijnlc01.pdf
• 9.
  REFERENCES [1] Alireza Mansouri,Lilly Suriani Affendey, Ali Mamat, (2008), “Named Entity Recognition Approaches”, Proceedings of IJCSNS International Journal of Computer Science and Network Security 8(2), 339–344. [2] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa, (2011) “Natural language processing (almost) from scratch”, CoRR, abs/1103.0398. [3] Zhiheng Huang, Wei Xu, and Kai Yu, (2015), “Bidirectional LSTM-CRF models for sequence tagging”, CoRR, abs/1508.01991. [4] Thi Thi Swe, Hla Hla Htay, (2010), “A Hybrid Methods for Myanmar Named Entity Identification and Transliteration into English”, http://www.lrecconf.org/proceedings/lrec2010/workshops/W16.pdf. [5] Thida Myint, Aye Thida, (2014), “Named Entity Recognition and Transliteration in Myanmar Text”, PhD Research, University of Computer Studies, Mandalay. [6] Mo H.M., Nwet K.T., Soe K.M. (2017) “CRF-Based Named Entity Recognition for Myanmar Language”. In: Pan JS., Lin JW., Wang CH., Jiang X. (eds) Genetic and Evolutionary Computing. ICGEC 2016. Advances in Intelligent Systems and Computing, vol 536. Springer, Cham. [7] Guillaume Lample, Miguel Balesteros, Sandeep Subramanian, Kazuya Kawakami and Chris Dyer, (2016), “Neural Architecture for Named Entity Recognition”, Proceedings of NAACL-HLT 2016, pages 260–270, Association for Computational Linguistics. [8] Jason P.C. Chiu and Eric Nichols, (2016), “Named Entity Recognition with Bidirectional LSTMCNNs”, Transactions of the Association for Computational Linguistics, vol. 4, pp. 357–370. [9] Daniele Bonadiman, Aliaksei Severyn and Alessandro Moschitti, (2015), “Deep Neural Networks for Named Entity Recognition in Italian”. [10] Weihua Wang, Feilong Bao and Guanglai Gao, (2016), “Mongolian Named Entity Recognition with Bidirectional Recurrent Neural Networks”, IEEE 28th International Conference on Tools with Artificial Intelligence. [11] Shotaro Misawa, Motoki Taniguchi, Yasuhide Miura and Tomoko Ohkuma, (2017), “Character- based Bidirectional LSTM-CRF with words and characters for Japanese Named Entity Recognition”,Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 97–102,Association for Computational Linguistics. [12] Anh L.T, Arkhipov M.Y., and Burtsev M.S., (2017), “Application of a Hybrid Bi-LSTM-CRF model to the task of Russian Named Entity Recognition”. [13] Antonio Jimeno Yepes and Andrew MacKinlay, (2016), “NER for Medical Entities in Twitter using Sequence Neural Networks”, Proceedings of Australasian Language Technology Association Workshop, pages 138−142.
• 10.
  [14] Nut Limsopathamand Nigel Collier, (2016), “Bidirectional LSTM for Named Entity Recognition in Twitter Messages”, Proceedings of the 2nd Workshop on Noisy User-generated Text, pages 145–152. [15] Lishuang Li, Like Jin, Zhenchao Jiang, Dingxin Song and Degen Huang, (2015), “Biomedical Named Entity Recognition Based on Extended Recurrent Newual Networks”, IEEE International Conference on Bioinfonnatics and Biomedicine. [16] Zhehuan Zhao, Zhihao Yang, Ling Luo, Yin Zhang, Lei Wang, Hongfei Lin and Jian Wang, (2015), “ML-CNN: a novel deep learning based disease named entity recognition architecture”, IEEE International Conference on Bioinfonnatics and Biomedicine. [17] Zin Maung Maung and Yoshiki Mikami, (2008), “A rulebased syllable segmentation of myanmar text”, In IJCNLP, Workshop on NLP for Less Privileged Languages, pages 51–58. [18] Tin Htay Hlaing and Yoshiki Mikami, (2014), “Automatic syllable segmentation of myanmar texts using finite state transducer”, ICTer, 6(2). [19] Yoshua Bengio, Patrice Simard, and Paolo Frasconi, (1994), “Learning long-term dependencies with gradient descent is difficult”, IEEE transactions on neural networks, 5(2):157-166. [20] Sepp Hochreiter and J¨urgen Schmidhuber, (1997), “Long short-term memory”, Neural computation, 9(8):1735–1780. [21] Ye Kyaw Thu, (2017), “Syllable segmentation tool for myanmar language (burmese)”, https:// github.com/ye-kyaw-thu/sylbreak. [22] Taku Kudo, (2005), “Crf++: Yet another crf toolkit”, Software available at http://crfpp. sourceforge. net.[23] Jie Yang, (2017), https://github.com/jiesutd/PyTorchSeqLabel [24] Nils Remers and Iryna Gurevych, (2017), “ Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks”, EMNLP. [25] Xuezhe Ma and Eduard Hovy, (2016), “ End-to-end Sequence Labeling via Bi-directional LSTMCNNs-CRF”, ACL 2016, Berlin, Germany.