We spoke with IAS’ Rudra Ghosh about how his career in the automation sector is not the result of one single event.
When Rudra Ghosh, a machine learning operations engineer at Integral Ad Science, first began his career as a data engineer he quickly saw how manual deployment and the management of ETL pipelines were creating bottlenecks and were the source of many errors.
Which is how he recognised that his interest in this line of work was not “a single event, but more of an evolution fueled by necessity and a desire for efficiency”.
“I started scripting to automate repetitive tasks. Then, as a big data developer working with massive datasets and complex spark jobs, automation became non-negotiable for consistency and managing complex dependencies,” he told SiliconRepublic.com.
“This culminated in my work as a data architect, where I designed systems with automation in mind from the outset, thinking about scalability and operational resilience. Seeing the power of automation to free up people from mundane tasks and to build more robust systems consistently pushed me further down this path, eventually leading me to MLOps, which is arguably the pinnacle of automation in the data and ML space.”
What brought you to your current job?
My journey through data engineering, architecture and big data development provided a panoramic view of the data lifecycle. I saw incredible advancements in model development, but also a persistent gap in efficiently and reliably getting those models into production and maintaining them. My experience building automated, scalable data pipelines as a data engineer and big data developer, felt like the perfect toolkit for tackling the unique challenges of operationalising machine learning.
MLOps emerged as this exciting discipline that directly addressed these challenges, focusing on automating the entire ML lifecycle.
What were the biggest surprises you encountered on your career path in automation and how did you handle them?
One of the biggest surprises, especially early on, was the sheer complexity and interconnectedness when automating large-scale systems. It’s not just about scripting one thing, it’s about orchestrating many moving parts, handling failures gracefully, and ensuring idempotency. I learned to deal with this through meticulous planning, breaking down problems into smaller, manageable components, embracing infrastructure as code, and rigorously testing.
Another challenge, particularly as a data architect, was sometimes the cultural resistance or underestimation of the upfront investment required for robust automation. I addressed this by building proof-of-concepts, clearly articulating the long-term ROI and showcasing tangible benefits. Now, in MLOps, the challenge is the rapid evolution of tools and techniques, and the multidisciplinary nature requiring deep understanding of both ML and Ops. Continuous learning and collaboration with data scientists and devOps engineers are key to navigating this.
Was there any one person who was particularly influential as your career developed?
It’s challenging to attribute my development to a single individual, as I’ve been fortunate to learn from several influential figures at different stages. Early in my career as a data engineer and later as a big data developer, a senior technical lead was instrumental. They instilled in me a deep appreciation for rigor in automation, not just scripting for convenience, but engineering robust, testable and maintainable automated data pipelines. Their emphasis on operational excellence laid a critical foundation.
More recently, in my transition to and work within MLOps, two staff have been particularly impactful, a staff MLOps engineer and a senior staff data scientist. The MLOps Engineer provided deep technical mentorship on advanced automation techniques specific to the ML lifecycle, sharing best practices for building scalable CI/CD/CT pipelines, infrastructure-as-code for the systems and robust monitoring frameworks.
Simultaneously, collaborating closely with the data scientist offered invaluable insights into the practical challenges data scientists face, helping me understand their needs for streamlined experimentation, reproducible model training and efficient deployment pathways. Each of these individuals, in their own way, contributed significantly to my understanding and application of automation principles.
What do you enjoy most about your job?
What I enjoy most as an MLOps engineer is enabling and accelerating the impact of machine learning. I get immense satisfaction from building the automated systems that take a brilliant model developed by a data scientist and turn it into a reliable, scalable service that solves real-world problems. It’s a fantastic blend of software engineering best practices, deep data understanding honed through my previous roles and the cutting-edge world of ML. I love designing and implementing the ‘plumbing’ – the CI/CD/CT pipelines, monitoring, and infrastructure – that makes the magic of ML work consistently. It feels like being the crucial link that ensures innovation translates into value.
What aspects of your personality do you feel make you suited to automation?
I think my background as a math student instilled in me a deep appreciation for logical, step-by-step thinking. I find a similar satisfaction in automation, breaking down a complex process into its core components and then building an efficient solution. It’s a bit like crafting an elegant proof or solving a particularly satisfying puzzle.
The meticulousness that math often demands, where a small detail can change everything, feels very applicable too. Automation scripts often need that same precision. If I see a manual, repetitive task, my mind often starts thinking about how to automate it, not out of laziness, but for the challenge and satisfaction of building a better way.
Finally, as someone who enjoys learning and developing, the debugging process, while sometimes frustrating, is also a great learning opportunity. Each hiccup is a chance to understand something more deeply. And the fact that automation is always evolving, with new tools and techniques, keeps things interesting and feeds that desire to continually learn.
What can people expect from career progression in the automation industry?
Career progression in this field offers a wide range of evolving opportunities, driven by the rapid advancement of tools and technologies. Over time, one can deepen their expertise in areas such as system architecture or emerging platforms, with the potential to grow into roles like principal engineer. Leadership paths, such as managing teams working on innovative solutions, are always available, though the specific roles continue to evolve alongside the industry.
What sets my current employer apart is the strong passion for coding at every level. Everyone, regardless of role, has a deep love for development and it’s evident in the collaborative spirit of the team. IAS supports this enthusiasm by fostering an environment where individual contributors are valued at every level, offering opportunities to stay hands-on while growing in expertise.
What advice would you give to those considering a career in automation, or just starting out in one?
Start by building a solid understanding of fundamental engineering and operations principles. Learn key concepts like scripting, version control, automation and system management. For those with a specific interest in fields like MLOps, understanding the full lifecycle of projects, from data collection and preparation to implementation and ongoing evaluation, is also essential.
Don’t hesitate to begin with small steps, automate tasks in the current role or personal projects. Focus on understanding the reasoning behind automation, not just the technical execution. Learn to view systems as interconnected, considering how different parts work together and how to design them for reliability. Finally, remember that the field is constantly evolving, so maintain a mindset of continuous learning and be open to experimentation.
Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.
