Shalini Ghosh
News (before joining Amazon in October 2020)
October 2020: I joined Amazon Science (Alexa AGI team) on as Principal Research Scientist.
October 2020: Gave an oral presentation in ACM Multimedia 2020 on our multimodal video processing paper
September 2020: Our paper was selected for Merit Award in Samsung Best Paper Award for 2020 -- within the top 4.6% of all submitted papers and within top 3 in the Multimedia track.
September 2020: Gave invited talk in Computer Science Lab at SRI.
August 2020: Gave an Invited Talk at KDD 2020 (Applied Data Science track): video.
July 2020: Our latest paper on multimodal AI got accepted at the ACM Multimedia 2020 conference as an Oral Paper.
June 2020: My interview on multimodal AI was featured on the list of Women in AI and Engineering: Pioneer Podcasts of ReWork 2020.
June 2020: Paper on Video Content Analysis has been short-listed for Samsung Best Paper Award 2020, awarded globally within Samsung for research with maximal innovation and impact (final selection of winner is pending).
May 2020: An article on 13 must-read AI papers that includes an interview with me was published by ReWork.
April 2020: Invited to be an Applied Data Science Invited Speaker at KDD 2020.
March 2020: My talk got selected as one the Top 5 AI talks in ReWork 2020.
January 2020: Interview with ReWork on multi-modal video processing went live (video).
January 2020: Gave invited talk in the Deep Learning Summit and interview at ReWork 2020 (full interview video can be viewed here).
December 2019: I got selected as one of the 30 Influential Women Advancing AI in 2019.
December 2019: My contribution to ReWork's AI Experts Predict 2020 Trends got published.
October 2019: My blog article on multi-modal AI got published.
March 2019: My interview on Deep Learning for Incremental Object Detection and Visual Dialog went live.
Overview
Currently I'm a Principal Research Scientist in Amazon Science, working in the Alexa AGI team.
Till August 2020, I was a Principal Scientist (Director) and the Leader of the Machine Learning Research team at the Smart TV division of Visual Display Intelligence Lab in Samsung Research America. Before this, from May 2018 - July 2019, I also served as the Director of AI Research in the Artificial Intelligence center in Samsung Research America in Mountain View, reporting to Dr. Larry Heck.
Before May 2018, I was a Principal Computer Scientist in the Computer Science Laboratory at SRI in Menlo Park, reporting to Dr. Patrick Lincoln.
I completed my PhD in 2005 at the Computer Engineering Research Center in ECE at the University of Texas at Austin. I worked with Prof. Nur Touba in the Computer Aided Testing (CAT) Laboratory. Previously, I did my MS from the Computer Engineering Department of University of California at Santa Cruz (UCSC). At UCSC, I worked with the Semiconductor Test Group. My MS Thesis advisor was Prof. F. Joel Ferguson.
In my undergraduate studies, I majored in Physics (Honors) from St. Xavier's College, University of Calcutta -- I minored in Mathematics and Chemistry.
I was invited to be a Visiting Scientist at Google Research in Mountain View, as part of the Google Visiting Faculty Program, for more than 1 year (July 2014 to August 2015). I worked on applying deep learning (Google Brain) models to problems in natural language understanding.
My contact email address is: "shalini DOT ghosh AT gmail DOT com"
Research Interests
I have worked on applying machine learning models to different domains. I am specifically interested in:
Speech understanding and Multi-modal learning (e.g., joint learning from speech, language, image, vision modalities)
Natural language applications (e.g., language models, paraphrase generation, machine reading)
Dependable computing (e.g., cyber security, fault tolerance)
Information Theory applications (e.g., error correcting codes)
Selected Awards and Honors
Invited to serve on the prestigious DARPA ISAT panel on LLMs in 2023, which advises US govt. on emerging technology.
Interview on multimodal AI was featured on the list of Women in AI and Engineering: Pioneer Podcasts of ReWork 2020.
Paper on Video Content Analysis selected for Merit Award in Samsung Best Paper Award 2020 (awarded globally within Samsung for research with maximal innovation and impact) -- within the top 4.6% of all submitted papers and within top 3 in the Multimedia track.
Invited to be an Applied Data Science Invited Speaker at KDD 2020.
Selected as one of the “Top 5 AI talks” of ReWork 2020.
Selected as one of the “30 Influential Women Advancing AI in 2019” by ReWork.
Invited to serve on the prestigious ISAT panel in 2020, which advises US government and DARPA on emerging technologies.
Best Paper Award at The 19th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) 2013.
SRI PoP (Period of Performance) Award in 3 successive years for outstanding performance: 2011, 2012, 2013.
SRI Spot Award 2013 for outstanding leadership and design innovation in the ARSENAL project.
SRI Spot Award 2012 for outstanding contribution to the ARSENAL project.
Best Student Paper Runner-up Award at GLS-VLSI 2004.
All-India Second Prize for undergraduate dissertation on radioisotopes in the Annual Contest of Atomic Energy Commission, Govt. of India, at the Bhabha Atomic Research Centre in Bombay, 1996; All India First prize for the written part of the contest.
Selected Talks and Interviews (recent)
Invited to present a tutorial on Resource-efficient and Cross-Modal Learning Towards Foundation Models at InterSpeech 2023.
Invited to give Keynote talk on “Speech Recognition in the Age of LLM” at the Amazon Science and Technology Summit (along with Bjorn Hoffmeister and Ariya Rastrow) at San Francisco in May, 2023.
Invited talk "Pre-Training and Multi-Modal Training" at the ICASSP 2022 Workshop "The new era of all-neural SLU: opportunities and challenges ahead": link. Was also one of the core panel discussion members.
Invited talk at Applied Data Science track, KDD 2020: video
Invited Talk at SRI 2020
Invited talk at ReWork 2020 Deep Learning Summit: video
Interview with ReWork, Jan 2020: video
Interview with ReWork Women in AI, Jan 2019: video
Invited talk on Conversational Vision at Deep Learning session of ReWork, Jan 2017: video
Invited talk on Contextual LSTMs at AI2, in Sep 2015: video
Invited talks at various universities, e.g., UC Berkeley (2014, 2019), Stanford University (2019)
Invited talks at various industrial labs, e.g., Google, Yahoo, LinkedIn, Nuance in 2018-2019
Publications
Papers
Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech [PDF]
David Chan, Shalini Ghosh, Hitesh Tulsiani, Ariya Rastrow, Bjorn Hoffmeister.
In Proceedings of ICASSP, 2024.
JAB: Joint Adversarial Prompting and Belief Propagation
Ninareh Mehrabi, Palash Goyal, Anil Ramakrishna, Jwala Dhamala, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta
In Proceedings of NeurIPS Workshop on Robustness of Zero/Few-shot Learning in Foundation Models (R0-FoMo), 2023.
Multi-Stage Multi-Modal Pre-Training for Automated Speech Recognition [PDF]
Yash Jain, David Chan, Pranav Dheram, Aparna Khare, Venkatesh Ravichandran, Olabanji Shonibare, Shalini Ghosh
In arXiv:2110.09890 [eess.AS]
Generative ASR Error Correction with Large Language Models [PDF]
Chao-Han Huck Yang, Yi-Le Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, Andreas Stolcke
In ASRU, 2023.
FLIRT: Feedback Loop In-context Red Teaming
Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta
In arXiv:2308.04265 [cs.AI]
Masking Latent Gender Knowledge for Debiasing Image Captioning
Fan Yang, Shalini Ghosh, Kechen Qin, Prashan Wanigasekara, Emre Barut, Chengwei Su
Submitted for publication
Multi-Modal Pre-Training for Automated Speech Recognition [PDF].
David Chan, Shalini Ghosh, Debmalya Chakrabarty, Bjorn Hoffmeister.
In Proceedings of ICASSP, 2022.
Tutorial on “Resource-Efficient and Cross-Modal Learning Toward Foundation Models”
Accepted at Interspeech 2023.
Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition [PDF].
David M Chan, Shalini Ghosh, Ariya Rastrow, Björn Hoffmeister.
Published in arXiv and In Proceedings of ICASSP, 2023.
Prune Then Distill: Dataset Distillation with Importance Sampling [PDF]
Anirudh S Sundar, Gokce Keskin, Chander Chandak, I-Fan Chen, Pegah Ghahremani, Shalini Ghosh.
In proceedings of ICASSP 2023.
Disentangled Action Recognition with Knowledge Bases [PDF].
Zhekun Luo, Shalini Ghosh, Devin Guillory, Trevor Darrell, Huijuan Xu.
In Proceedings of NAACL Conference, 2022.
Scalable and Accurate Self-Supervised Multimodal Representation Learning Without Aligned Video and Text Data [PDF].
Vladislav Lialin, Stephen Rawls, David Chan, Shalini Ghosh, Anna Rumshisky, Wael Hamza.
In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
Content-Context Factorized Representations for Automated Speech Recognition [PDF].
David Chan, Shalini Ghosh.
In Proceedings of InterSpeech, 2022.
One-stage Object Referring with Gaze Estimation [PDF].
Jianhang Chen, Xu Zhang, Yue Wu, Shalini Ghosh, Varsha Hedau, Pradeep Natarajan, Shih-Fu Chang, Jan Allebach.
In Proceedings of the Gaze 2022 Workshop, CVPR Conference, 2022.
Unified Modeling of Multi-domain Multi-device ASR Systems [PDF].
Soumyajit Mitra, Swayambhu Nath Ray, Bharat Padi, Arunasish Sen, Raghavendra Bilgi, Harish Arsikere, Shalini Ghosh, Ajay Srinivasamurthy.
Published in TSD 2023.
Multi-Stage Multi-Modal Pre-Training for Automated Speech Recognition [PDF]
Yash Jain, David Chan, Pranav Dheram, Aparna Khare, Venkatesh Ravichandran, Olabanji Shonibare, Shalini Ghosh
Submitted to for publication.
In-Context Learning for ASR Rescoring with Large Language Models [PDF].
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Andreas Stolcke, Shalini Ghosh, Ivan Bulyko.
Submitted to Interspeech 2023.
Masking Latent Gender Knowledge for Debiasing Image Captioning [PDF].
Fan Yang, Shalini Ghosh, Kechen Qin, Prashan Wanigasekara, Emre Barut, Chengwei Su.
Submitted for publication to ICCV, 2023.
FLIRT: Feedback Loop In-context Red Teaming [PDF].
Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta.
Submitted for publication to NeurIPS, 2023
Hierarchical Class-based Curriculum Loss [PDF of earlier Arxiv version]
Palash Goyal, Shalini Ghosh, Divya Choudhary
In IJCAI Conference, 2021 (selected for oral presentation, acceptance rate 13.9%)
Cross-modal Non-linear Guided Attention and Temporal Coherence in Multi-modal Deep Video Models [PDF]
Saurabh Sahu*, Palash Goyal*, Shalini Ghosh*, Chul Lee (*Equal contributions)
In ACM Multimedia Conference, 2020 (Accepted as an Oral Presentation).
Cross-modal Learning for Multi-modal Video Categorization [PDF]
Palash Goyal*, Saurabh Sahu*, Shalini Ghosh*, Chul Lee (*Equal contributions)
In arXiv:2003.03501[cs.CV], Mar 2020.
Exploiting Temporal Coherence for Multi-modal Video Categorization [PDF]
Palash Goyal*, Saurabh Sahu*, Shalini Ghosh*, Chul Lee (*Equal contributions)
In arXiv:2002.03844 [cs.CV], Feb 2020.
Class-incremental Learning via Deep Model Consolidation [PDF]
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C.-C. Jay Kuo.
In IEEE Winter Conference on Computer Vision (WACV), 2020.
Regularize, Expand and Compress: Multi-task based Lifelong Learning via NonExpansive AutoML [PDF]
Jie Zhang, Junting Zhang, Shalini Ghosh, Dawei Li, Jingwen Zhu, Heming Zhang, Yalin Wang.
In IEEE Winter Conference on Computer Vision (WACV), 2020.
IMOD: An Efficient Incremental Learning System for Mobile Object Detection [PDF]
Dawei Li, Serafettin Tasci, Shalini Ghosh, Jingwen Zhu, Junting Zhang, Larry Heck.
In Proceedings of the ACM/IEEE Symposium on Edge Computing (SEC), 2019.
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded [PDF]
Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, Devi Parikh.
In Proceedings of the International Conference of Computer Vision (ICCV), 2019.
Generative Visual Dialogue System via Adaptive Reasoning and Weighted Likelihood Estimation [PDF]
Heming Zhang, Shalini Ghosh, Larry Heck, Stephen Walsh, Junting Zhang, Jie Zhang, C.-C. Jay Kuo.
In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2019.
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression [PDF]
Jie Zhang, Xiaolong Wang, Dawei Li, Shalini Ghosh, Abhishek Kolagunda, Yalin Wang.
In arXiv:1902.00918, 2019.
Class-incremental Learning via Deep Model Consolidation [PDF]
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C.-C. Jay Kuo.
In CVPR Workshop on Visual Understanding by Learning from Web Data (WebVision), 2019 (workshop version of the WACV 2020 paper).
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded [PDF]
Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Dhruv Batra, Devi Parikh.
In Proceedings of the ICLR Workshop on Debugging Machine Learning Models, 2019 (workshop version of the ICCV paper).
A Unified Framework for Domain Adaptation using Metric Learning on Manifolds [PDF]
Sridhar Mahadevan, Bamdev Mishra, Shalini Ghosh
In Proceedings of the European Conference on Machine Learning (ECML), 2018.
Generating Natural Language Explanations for Visual Questions Using Scene Graphs [PDF]
Shalini Ghosh, Giedrius Burachas, Arijit Ray, Avi Ziskind
In Proceedings of the ICML/IJCAI Joint Workshop Explainable Artificial Intelligence (XAI), 2018.
Discriminative Visual Dialogue Bot
Heming Zhang, Shalini Ghosh, Larry Heck
Invited for presentation at the Visual Dialog Challenge Poster Session of the SiVL Workshop at ECCV, 2018.
Time Series Deinterleaving of DNS Traffic [PDF]
Amir Asiaee T., Hardik Goel, Shalini Ghosh, Vinod Yegneswaran, Arindam Banerjee
In Proceedings of the First Deep Learning and Security Workshop (DLS), co-located with the 39th IEEE Symposium on Security and Privacy (Oakland Conference), 2018.
Trusted Neural Networks for Safety-Constrained Autonomous Control [PDF]
Shalini Ghosh, Amaury Mercier, Dheeraj Pichapati, Susmit Jha, Vinod Yegneswaran, Patrick Lincoln
In Proceedings of the ICML/IJCAI Joint Workshop of Deep Learning for Safety-Critical Applications in Engineering (DISE1), 2018.
AEGIS: An Automated Permission Generation and Verification System for SDN [PDF]
Heedo Kang, Seungwon Shin, Vinod Yegneswaran, Shalini Ghosh, Phil Porras
In Proceedings of the Workshop on Security in Softwarized Networks: Prospects and Challenges (SecSoN), co-located with the ACM SIGCOMM Conference, 2018.
Model, Data and Reward Repair: Trusted Machine Learning for Markov Decision Processes [PDF]
Shalini Ghosh, Ashish Tiwari, Susmit Jha, Patrick Lincoln, Xiaojin Zhu
In Proceedings of the DSN Workshop on Dependable and Secure Machine Learning (DSN-DSML), co-located with the IEEE/IFIP International Conference on Dependable Systems and Networks, 2018.
Data masking for privacy-sensitive learning [PDF]
Anh T. Pham, Shalini Ghosh, Vinod Yegneswaran
In Proceedings of the NIPS Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments (NIPS-TIML), 2017.
Automated Categorization of Onion Sites for Analyzing the Darkweb Ecosystem [PDF] [Code]
Shalini Ghosh, Ariyam Das, Phil Porras, Vinod Yegneswaran, Ashish Gehani
In Proceedings of 23rd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2017.
Trusted Machine Learning: Model Repair and Data Repair for Probabilistic Models [PDF]
Shalini Ghosh, Patrick Lincoln, Ashish Tiwari, Xiaojin Zhu
In AAAI 2017 Workshops: (a) AI for Connected and Automated Vehicles (AICAV) (oral presentation), and (b) Symbolic Inference and Optimization (SymInfOpt).
ATOL: A Framework for Automated Analysis and Categorization of the Darkweb Ecosystem [PDF]
Shalini Ghosh, Phillip Porras, Vinod Yegneswaran, Ken Nitz, Ariyam Das
In AAAI Workshop on Artificial Intelligence for Cyber Security (AICS), 2017. (Oral presentation)
Trusted Machine Learning for Probabilistic Models [PDF]
Shalini Ghosh, Patrick Lincoln, Ashish Tiwari, Xiaojin Zhu
In ICML Workshop on Reliable Machine Learning In the Wild (WildML), 2016 (Selected for Oral Presentation).
Contextual LSTM (CLSTM) models for Large scale NLP tasks [PDF]
Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, Larry Heck
In KDD Workshop on Large-scale Deep Learning for Data Mining (DL-KDD), 2016 (Selected for Oral Presentation).
ARSENAL: Automatic Requirements Specification Extraction from Natural Language [PDF of extended arXiv version]
Shalini Ghosh, Daniel Elenius, Wenchao Li, Patrick Lincoln, Natarajan Shankar, Wilfried Steiner
In Proceedings of the 8th NASA Formal Methods Symposium (NFM), 2016.
Our ARSENAL work was a chapter in the Ph.D. thesis of Wenchao Li at UC Berkeley, which was awarded the 2015 ACM SIGDA Outstanding Ph.D. Dissertation Award.
Virus Detection in Multiplexed Nanowire Arrays using Hidden Semi-Markov models [PDF]
Shalini Ghosh, Patrick Lincoln, Christian Petersen, Alfonso Valdes
arXiv:1407.4490v1 [cs.AI], July 2014
Probabilistic Modeling of Failure Dependencies Using Markov Logic Networks [PDF] (Best Paper Award)
Shalini Ghosh, Wilfried Steiner, Grit Denker, and Patrick Lincoln
In Proceedings of the 19th IEEE Pacific Rim International Symposium on Dependable Computing (PRDC), 2013.
An Empirical Reexamination of Global DNS Behavior [PDF]
Hongyu Gao, Vinod Yegneswaran, Yan Chen, Phillip Porras, Shalini Ghosh, Jian Jiang and Haixin Duan
In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM), 2013.
MALCOM: Machine Learning towards Automated Law Compliance Monitoring
Selected for presentation at the IARPA Workshop (invitation-only) on International Law Compliance Monitoring (ILCM), 2012.
Combining Subjective Probabilities and Data in Training Markov Logic Networks [PDF]
Tivadar Papai, Shalini Ghosh, Henry Kautz
In Proceedings of ECML-PKDD, 2012.
Markov Logic Networks in Health Informatics [PDF]
Shalini Ghosh, Natarajan Shankar, Sam Owre, Sean David, Gary Swan, Patrick Lincoln
In Proceedings of ICML-MLGC, 2011.
Machine Reading Using Markov Logic Networks for Collective Probabilistic Inference [PDF]
Shalini Ghosh, Natarajan Shankar, Sam Owre
In Proceedings of ECML-CoLISD, 2011.
Dynamic LDPC Codes for Nanoscale Memory with Varying Fault Arrival Rates [PDF]
Shalini Ghosh and Patrick D. Lincoln
Proceedings of Design and Technology of Integrated Systems (DTIS), 2011.
Query Routing in Wireless Sensor Networks: A Novel Application of Social Query Models [PDF]
Shalini Ghosh, Patrick D. Lincoln
SRI Computer Science Laboratory Technical Report, 2012.
Dynamic Low-Density Parity Check Codes for Fault-tolerant Nano-scale Memory [PDF]
Shalini Ghosh and Patrick D. Lincoln
Proceedings of Foundations of Nanoscience (FNANO07), Snowbird, Utah, April 2007
Low-Density Parity Check Codes for Error Correction in Nanoscale Memory [PDF]
Shalini Ghosh and Patrick D. Lincoln
SRI Computer Science Laboratory Technical Report, CSL-0703, September 2007
Synthesis of Low Power CED Circuits Based on Parity Codes [PDF]
Shalini Ghosh, Sugato Basu and Nur A. Touba
Proceedings of the VLSI Test Symposium (VTS), Palm Springs, California, May 2005.
Selecting Error Correcting Codes to Minimize Power in Memory Checker Circuits [PDF]
Shalini Ghosh, Sugato Basu and Nur A. Touba
Journal of Low Power Testing (JOLPE), 2005
Detection Probabilities of Interconnect Breaks: An Analysis [PDF]
Shalini Ghosh and F. Joel Ferguson
Special Issue of Integration - the Elsevier VLSI Journal, Vol 38/3, pp 451-465, 2004
Reducing Power Consumption in Memory ECC Checkers [PDF]
Shalini Ghosh, Sugato Basu and Nur A. Touba
Proceedings of the IEEE International Test Conference (ITC), pp. 1322-1331, Charlotte, October 2004.
Estimating Detection Probabilities of Interconnect Opens using Stuck-at Tests [PDF] (Runner-up for the Best Student Paper Award)
Shalini Ghosh and F. Joel Ferguson
Proceedings of the Great Lakes Symposium on VLSI (GLS-VLSI), pp. 254-259, Boston, April 2004.
Low-power Weighted Pseudo-random BIST Using Special Scan Cells [PDF]
Shalini Ghosh, Eric McDonald, Sugato Basu and Nur A. Touba
Proceedings of the Great Lakes Symposium on VLSI (GLS-VLSI), pp. 86-91, Boston, April 2004.
Joint Minimization of Power and Area in Scan Testing by Scan Cell Re-ordering [PDF]
Shalini Ghosh, Sugato Basu and Nur A. Touba
Proceedings of the IEEE Computer Society Annual Symposium on VLSI (ISVLSI-2003), Tampa, FL, February 2003.
An Analysis of Detection Probabilities of Interconnect Opens [PDF]
Shalini Ghosh and F. Joel Ferguson
Baskin School of Engineering Technical Report UCSC-CRL-03-17, University of California, Santa Cruz, March 2004.
Joint Minimization of Power and Area in Scan Testing by Scan Cell Re-ordering [PDF]
Shalini Ghosh, Sugato Basu and Nur A. Touba
CERC Technical Report UT-CERC-TR-NAT02-1, University of Texas, Austin, 2002.
PhD Thesis
Selected Patent Awards and Applications
“Topic-based sequence modeling neural networks”, US Patent No. 10,08,3169-B1, 2018.
“Systems and methods for machine learning using a trusted model”, US Patent No. 2017/0364831-A1, 2017.
“Trusted neural network system”, Patent application, Docket No.: 1248-011US01 / SRI P170047.
“Multi-task based lifelong learning”, Patent application, Docket No.: CSI18-A135 / SAMS12-00512.
“Incremental learning without forgetting for efficient object detection”, Patent application, Docket No.: CSI18-A134 / SAMS12-00513.
“Microgenre-based Hyper Personalization with Multi-modal Machine Learning using Video Content Categorization and Analysis”, Patent application, DOI No.: WA-202002-003-1-US0.
“Cross-modal Learning for Multi-modal Video Categorization”, Patent application, DOI No.: WA-202002-014-1-US0.
“Synthetic Data Generation for Machine Learning Models”, Provisional Patent 63/521,267
Professional Services / Recognition/ Affiliations
Area Chair: NeurIPS 2022, ICLR 2022, ICLR 2021, ICML 2020.
Session Chair: Session Chair for ICLR 2021, IJCAI 2019, KDD 2017.
Program Committee member: NIPS (Neural Information Processing Systems) 2020, 2019, 2018, 2017, 2016 [PC Member/Reviewer]; KDD (ACM SIGKDD Conference on Knowledge Discovery and Data Mining) 2017, 2016, 2015 and 2014; ICML (International Conference on Machine Learning) 2020, 2019, 2018, 2017, 2016; SDM (SIAM International Conference on Data Mining) 2013; AISTATS (Artificial Intelligence and Statistics Conference) 2019 (refused), 2017; NAACL (North American Association of Computational Linguistics) 2012; IJCAI (International Joint Conference on AI) in 2020, 2019; NFM (NASA Formal Methods) 2016, 2014; VALID 2013, 2014, ACML (Asian Conference on Machine Learning) 2012. Invited to be on PC of Association for the Advancement of Artificial Intelligence (AAAI) in 2019, 2017, 2013 and 2014 PhD Symposium on Formal Methods and Analysis.
Reviewer for leading journals/conferences: ACM/IEEE Symposium on Logic in Computer Science (LICS) 2016, ACM Transactions on Reconfigurable Technology and Systems (TRETS), Journal of Electronic Testing Theory and Applications (JETTA), IEEE Transactions on Very Large Scale Integration Systems (TVLSI), IEEE Transactions on Computer, IET Circuits, Devices and Systems, Journal of Low Power Electronics (JOLPE), Elsevier Journal on Signal Processing (SIGPRO), Network & Distributed System Security Symposium (NDSS), Turing Centenary Conference in 2012, Information Processing Letters (IPL) 2016.
PhD Committee member: Tivadar Papai, University of Rochester (PhD Advisor: Prof. Henry Kautz)
Invited guest lecturer in EECS course at UC Berkeley: Gave lecture on ARSENAL at the EECS Department of the Universiy of California at Berkeley in February 2014. Course: EECS 294-98 (Formal Methods for Engineering Education).
Invited to write a blog article at ReWork on the state-of-the-art research in multimodal machine learning.
Member of Women In Machine Learning Program (affiliated to ICML), 2011.
Member of ACL (Association for Computational Linguistics), SIGKDD (ACM Special Interest Group on Knowledge Discovery and Data Mining), IEEE, Women in Engineering Program at UT Austin (2000 to 2005).
Invited panelist for various panels, e.g., AAAI-AICS in 2018, National Science Foundation panel for Emerging Technologies in 2008, VSLI Test Symposium panel at Berkeley in 2013.
Invited to give a keynote talk on ARSENAL at DATE 2017 in DUHDe, the Workshop on Design Automation for Understanding Hardware Designs (declined).
Invited talks at several leading research institutes (e.g., NASA) and universities (e.g., UC Berkeley in 2014 and 2019, Stanford University in 2019).
Invited to present whitepaper at IARPA Workshop on applying Natural language procesing for Automated International Law Compliance Monitoring in May, 2012.
Invited attendee at the BAST workshop at Bodega Bay, organized jointly by the Stanford Center for Reliable Computing and the IEEE Test Technology Committee, in 2000 and 2006.
Organized and led study groups and seminar series on (1) Statistics and Probabilistic Reasoning at CSL in SRI, 2007 to 2010, (2) Emerging technologies, 2005-2007.
Projects at SRI
ARSENAL 2.0 (Principal Investigator of project funded by DARPA) -- Converts stylized natural language requirements to their precise formal representations using state-of-the-art NLP techniques (e.g., domain-specific semantic parsing), which can be then analyzed by formal methods (e.g., formal verification tools, SMT solvers). The approaches used here are primarily based on machine learning.
Explainable AI (XAI) (Technical Lead of Explanation Model in project funded by DARPA, starting May 2017) -- Apply Contextual LSTM model (and variations) to come up with both answers and explanations for the visual question/answering task.
SETI (Key Personnel in project funded by DARPA) -- Applying different approaches (e.g., NLP analysis, deep learning) of generating paraphrases for conversational modeling in Twitter.
Trusted Machine Learning (Principal Investigator in project funded by SRI Internal R&D funding) -- Incorporating safety, liveness and other high-level logical properties into machine learning models.
MEMEX (Key Personnel in project funded by DARPA) -- Working on ATOL, an automated thematic categorization (both clustering and classification) of onion sites using machine learning and NLP techniques.
HIMALAYAS (Joint Principal Investigator of project funded by NSF SaTC Medium) -- Applying machine learning/data mining techniques for analyzing large-scale DNS query data, to detect malware domain sequences.
ARSENAL (Principal Investigator of project funded by DARPA) -- Converts stylized natural language requirements to their precise formal representations using state-of-the-art NLP techniques (e.g., domain-specific semantic parsing), which can be then analyzed by formal methods (e.g., formal verification tools, SMT solvers).
Machine Reading (Key Personnel in project funded by DARPA) -- Did probabilistic reasoning/inference using the Probabilistic Consistency Engine (PCE), an inference engine based on Markov Logic networks, to maintain/enforce consistency in the state of relations/entities extracted by different information extraction systems in the DARPA Machine Reading project. Worked actively for the DARPA evaluation of the machine reading system.
META (Key Personnel in project funded by DARPA) -- Used PCE for probabilistic modeling of failure dependencies in large complex cyber-physical systems.
Mole-Sensing (Key Personnel in project funded by DARPA) -- Used Hidden semi-Markov Models (HsMM) for detecting viruses using sensor arrays of nano-wires.
Mole-Computing (Key Personnel in project funded by DARPA, Principal Investigator on the NSF project on Emerging Technologies) -- Designed dynamic Error Correcting Codes (ECC) for building fault-tolerant architectures at the nano-scale level.
Robust routing -- Applied a probabilistic routing algorithm for efficient robust/resilient routing in wireless sensor networks.
Medical diagnosis -- Did statistical analysis of patient and control data from a novel Wii-based Parkinson's disease detection/calibration system.
Bootstrap learning -- Did fault analysis of the learning system in DARPA Bootstrap Learning project.
Universal spellchecker -- Wrote a hybrid (rule-based and statistical) universal spelling checker for multiple languages, as part of the DARPA Transdec project.
Radar mail -- Implemented a module to predict meeting tasks in emails for DARPA Radar project.
Scientific simulation -- Implemented a simulator for time-of-flight mass spectrometers and Fourier transform infrared spectrometers, as part of DARPA Chemist project.
Cyber physical systems -- Worked on developing resilience workbench for critical applications (e.g., medical remote surgery devices), as part of funded NSF project.
PhD Research (Computer Aided Testing Lab, University of Texas Austin, 2000 to 2005)
Studied the reduction of power consumption in concurrent error detection, for memory ECC checkers and low-power parity prediction circuits. Also studied how to reduce power in offline testing, for weighted pseudo-random BIST and scan testing.
MS Research (Semiconductor Test Group, UC Santa Cruz, 1998 to 2000)
Did MS thesis on fault modeling of interconnect opens using stuck-at tests, provided a statistical model for the conditions required for stuck-at tests to detect interconnect breaks in a circuit.
Internship Research during PhD
Test Group of Synopsys Inc., Sunnyvale, CA in Summer 2004 (Project Advisor: Dr. Rohit Kapur, Manager: Adam Cron): Worked on state-of-the-art scan compression method, launched as the Adaptive Scan technology.
Microprocessor Validation group of Intel Corporation, Austin, TX in Summer 2001 (Manager: Dr. Neeta Ganguly): Worked with Microprocessor Validation tools for functional testing.
Design and Test Group of Intel Corporation, Santa Clara, CA in Summer 1999 (Manager: Dr. Sreejit Chakravarty): Added pattern-dependant bridge behavior modeling to Intel's internal fault simulator software.
Official Mentoring/Hiring
Full-time employees:
Palash Goyal: Hired from University of Southern California, Ph.D and managed in SRA.
Saurabh Sahu: Hired from University of Maryland at College Park, Ph.D and managed in SRA.
Ajit Sinha: Managed in SRA.
Serafettin Tasci: Managed in SRA.
Dawei Li: Managed in SRA.
Jingwen Zhu: Managed in SRA.
Stephen Walsh: Managed in SRA.
Akhila Yerukola: Hired from Stanford University, M.S.
Wenbo Li, Hired from SUNY Albany, M.S.
Interns/International Fellows:
Gautam Krishna, University of Texas at Austin: Research Intern in 2019.
Niranjan Kannabiran, Georgia Institute of Technology: Research Intern in 2019.
Arda Senocak, KAIST.
Lipeng Ke, SUNY Albany.
Joena Wang, Arizona State University: Research Intern in 2018.
Heming Zhang, University of Southern California: Research Intern in 2018.
Junting Wang, University of Southern California: Research Intern in 2018.
Dheeraj Pichapati, University of California at San Diego: Research Intern in 2017.
Anh The Pham, Oregon State University: Research Intern in 2017.
Amaury Mercier, Ecole Polytechnique: International Fellow in 2017.
Jean-Baptiste Lamare, Ecole Polytechnique: International Fellow in 2017.
Ariyam Das, University of California at Los Angeles: Research Intern in 2016.
Yizheng Chen, Georgia Institute of Technology: Research Intern in 2014, co-mentored with Vinod Yegneswaran.
Wenchao Li, University of California at Berkeley: Research Intern in 2013. Currently at Boston University.
Hongyu Gao, Northwestern University: Research Intern in 2012, co-mentored with Vinod Yegneswaran. Currently at Google.
Jose Antonio Baena, University of Malaga: International Fellow in 2012, co-mentored with Grit Denker. Currently at Google.
Tivadar Papai, University of Rochester: Research intern in 2011. Also served on Tivadar's PhD Thesis Committee (PhD Advisor: Prof. Henry Kautz). Currently at Google.