Evaluation // Panel 1 // Large Language Models in Production Conference Part 2

// Abstract Language models are very complex thus introducing several challenges in interpretability. The large amounts of data required to train these black-box language models make it even harder to understand why a language model generates a particular output. In the past, transformer models were typically evaluated using perplexity, BLEU score, or human evaluation. However, LLMs amplify the problem even further due to their generative nature thus making them further susceptible to hallucinations and factual inaccuracies. Thus, evaluation becomes an important concern. // Bio Abi Aryan Machine Learning Engineer @ Independent Consultant Abi is a machine learning engineer and an independent consultant with over 7 years of experience in the industry using ML research and adapting it to solve real-world engineering challenges for businesses for a wide range of companies ranging from e-commerce, insurance, education and media & entertainment where she is responsible for machine learning infrastructure design and model development, integration and deployment at scale for data analysis, computer vision, audio-speech synthesis as well as natural language processing. She is also currently writing and working in autonomous agents and evaluation frameworks for large language models as a researcher at Bolkay. Prior to consulting, Abi was a visiting research scholar at UCLA working at the Cognitive Sciences Lab with Dr. Judea Pearl on developing intelligent agents and has authored research papers in AutoML and Reinforcement Learning (later accepted for poster presentation at AAAI 2020) and invited reviewer, area-chair and co-chair on multiple conferences including AABI 2023, PyData NYC ‘22, ACL ‘21, NeurIPS ‘18, PyData LA ‘18. Amrutha Gujjar CEO & Co-Founder @ Structured Amrutha Gujjar is a senior software engineer and CEO & Co-Founder of Structured, based in New York. With a Bachelor of Science in Computer Science from the University of Washington's Allen School of CSE, she brings expertise in software development and leadership to my work. Amrutha has experience working at top tech companies, including Google, Facebook, and Microsoft, where I've worked on a variety of projects including machine learning, data collection studies platforms, and infrastructure. She also had the honor of being a TEDx Redmond Speaker and receiving several awards and honors, such as the National Merit Scholarship Finalist and NCWIT Aspirations in Computing National Award Finalist, and Washington Affiliate Recipient. In Amrutha's current role at Structured, she is focused on unlocking the power of expert knowledge to supercharge language models. Prior to this, She spent four years at Facebook as a Senior Software Engineer, where she worked on the Community Integrity team building a Knowledge Graph to maintain policy designations of terrorism and hate organizations on Facebook platforms. Amrutha's passion for computer science and leadership is evident through her work and involvement in several organizations, including being a keynote speaker for the Northshore Schools Foundation and attending the Grace Hopper Conference. Amrutha is also a Contrary Fellow, and ZFellow, and has completed the YC Startup School. Connect with me on LinkedIn to learn more about my experience and discuss exciting opportunities in software development and leadership. Josh Tobin Founder @ Gantry Josh Tobin is the founder and CEO of Gantry. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel. Sohini Roy Senior Developer Relations Manager @ NVIDIA Sohini Bianka Roy is a senior developer relations manager at NVIDIA, working within the Enterprise Product group. With a passion for the intersection of machine learning and operations, Sohini specializes in the domains of MLOps and LLMOps. With her extensive experience in the field, she plays a crucial role in bridging the gap between developers and enterprise customers, ensuring smooth integration and deployment of NVIDIA's cutting-edge technologies. Sohini's expertise lies in enabling organizations to maximize the potential of machine learning models in real-world scenarios through efficient and scalable operational practices. Her insights continue to drive innovation and success for enterprises navigating the rapidly evolving landscape of machine learning and operations. Previously, Sohini was a product manager at Canonical, supporting products from Ubuntu on Windows Subsystem for Linux to their Charmed Kubernetes portfolio. She holds a bachelor's degree from Carnegie Mellon University in Materials Science and Biomedical Engineering.