Real Interview Q&A for Senior Data Engineer #2. Second round. | Surfalytics

This video is a second round for a Senior Data Engineer position, Dmitry Anoshin shares his experience across various data platforms like Snowflake, Databricks, AWS, Azure, and GCP. We discuss showcases in building data pipelines, ensuring data quality, and handling complex data challenges. Key Points: Data Validation: The candidate emphasizes the importance of data validation using tools like DBT and Monte Carlo, ensuring data freshness, accuracy, and schema consistency. Schema Evolution: The candidate shares strategies for managing schema changes, including schema-on-read approaches and careful communication with partners to avoid disruptions. Scaling Challenges: They discuss overcoming performance bottlenecks in Athena due to large datasets and complex queries, highlighting solutions like incremental loading and query optimization. Data Sharing: They touch on data sharing options like Databricks' data sharing capabilities, enabling seamless data access for partners without physical data movement. Thank you for watching our video "Real Interview Q&A for Senior Data Engineer #2. Second round. | Surfalytics" on SurfalyticsTV as part of a series of "Real Interviews records", click the link to watching the previous videos: https://www.youtube.com/watch?v=psuPhJtQmsE&list=PLNCDg7zJiXhM5Gshe5_Q2HAZM5vIOLpI1&pp=iAQB Subscribe for channel for more videos: https://www.youtube.com/@SurfalyticsTV?sub_confirmation=1 Interested in learning more about data engineering best practices and overcoming real-world challenges? Join our community at https://surfalytics.com/ to connect with other data professionals and stay up-to-date on the latest trends in the field. #dataengineering #interview #snowflake #databricks #aws #azure #gcp #datapipelines #dataquality #schemaevolution #scalingchallenges #datasharing #surfalytics Timecode: 00:00 - Intro 00:22 - Tell about yourself 01:35 - Data validation 05:18 - Schema evolution 07:28 - Scaling issue 10:15 - Python task 13:53 - SQL task ================= What is Surfalytics? Inspired by West Coast surfing spots 🏖️ and Pacific Ocean vibes 🌊. Created to help you start a new career in the data analytics space, and develop data engineering and analytics skills through coaching. It will teach you not just dry skills, but will keep your focus on delivering significant value to businesses in the analytics realm as well as help to get fair compensation 💰 for the work you’re passionate about ❤️‍🔥. The goal of Surfalytics is to assist you in achieving one of the following: 🏄‍♂️ Land your first job in the data industry with literally zero experience. I have accomplished this many times across the globe. 🏄 Advance from a middle-level role to a senior position (as an Analyst or Engineer). 🏄‍♀️ Transition from a non-technical Analyst role to a technical Engineer role. Moreover, we will focus on creating a highly competitive CV and securing top job offers. We will not consider any lowball offers, focusing only on top-tier companies and well-paid opportunities. Finally, Surfalytics is a results-driven community with a very narrow focus, resulting in a high return on investment (ROI). Here, ‘investment’ does not mean money but your time. I am literally fighting for your attention to encourage you to study and work hard, instead of watching Netflix or playing video games. This is the best YouTube channel for Data Analytics and Engineering. You will patch up a lack of knowledge and get new experience and tips to build a Data Analyst roadmap or Data Engineer roadmap for yourself. Want to be part of our growing community? Join on Surfalytics.com #surfalytics #dmitryanoshin #datacommunity #freecourses #dataanalysis #dataengineering #roadmap #careerpath #mindmaps #tools #overview #dataanalysttips