Shobhit Shrotriya: Pharma is Moving from AI Pilots to Production

Ram Yalamanchili (00:07.394)

Hey, Shobit, how are you?

Shobhit Shrotriya (00:12.398)

All around doing very well. How are you doing?

Ram Yalamanchili (00:12.952)

I'm doing great. Thank you for your time. I'm really excited to have you here. You know, for our guest, Shobit is a leader who spent quite a bit of time in clinical operations. He certainly manages one of the larger groups of organizations which manage especially life science operations. And I thought his perspective would be excellent to have in our podcast today.

So Shobit, I'd love for you to introduce yourself for our audience and give a little bit of a color on your background and of course, where you are today as well.

Shobhit Shrotriya (00:53.262)

Thank you Ram. First of all, heartfelt thanks for giving me this opportunity to talk to you on this specific topic. Really honored and looking forward to have a very good and live discussion interaction on this topic. About myself, Shobit Shrotriya, been in the industry for 25 years. I'm an engineer by training, by the way. I didn't start my career in life sciences and it's been 23 years now, I would say, 23 years. No looking back once I...

landed up into the life sciences world. Being in the industry as I said, close to 20 plus years, started my journey in the life sciences world with Quintiles, which is now IQVIA. And that happened by chance. As I said, I was an engineer by training and one of my friends used to work in the CRO industry and that point of time.

the outsourcing in the life sciences world was just emerging, evolving, and India was becoming a magnetic hub in terms of setting up clinical data management services. I used to hear a lot about how clinical trials are run, how we are giving back to the society, how we are caring for the patients, and enabling quality of life for them. That inspired me quite a bit, and I happened to then leave by engineering so-called job and land up into this world.

No looking back from that time on. Been with Accenture for 15 years, close to 15 years, and I love what I do. I'm pretty much engaged in doing everything and anything that helps the patients get to the better quality of life. Working with multiple global biopharma, responsible as a global lead and managing director for the life sciences business.

which means all the work that we do under the umbrella of clinical, regulatory, and pharmacovigilance. I'm responsible for service delivery and also responsible for bringing in transformation and innovative solutions to the frontier. So that's my day job, but I'm also very passionate about learning more and more. I do contribute to the industry. I'm also a part of one of the

Shobhit Shrotriya (03:08.629)

leading industry consortiums, is SCDM, Society for Clinical Data Management. I'm actually part of the board. I've got nominated as the vice chair last year. So this year I will be actually working with the global leaders across the biopharma and our industry leaders from the CRO and the FSP world to see how we can take our clinical data management, now the clinical data science practice to the next level.

Looking forward for the exciting year ahead and yes, continue to enjoy what I am doing. So that was a long introduction. I'll pause.

Ram Yalamanchili (03:43.542)

No, I love it. I want to say we share the commonality that both of us are engineers. I want to start with that. the other thing is timing-wise, I think it's interesting that your nomination as the vice chair comes at a time where it feels like engineering is eating into the world. That's what's happening on the technology side. So I wonder if that has something to do with it. the thing you raised on the nominated you. Great. No, thanks for the introduction. And I think we'll.

Let's start with the topic which I was really curious to hear about from you. You've been in this industry for a very long time. You've clearly scaled these operations to an extent where I think very few of us have seen the scale which you probably operate at and manage right now. So what I'd like to understand is, what's the journey been like? You said 23 years. So how did this industry look maybe early part of your career? And I'd love to see

you explain what the transformation looked like over the past couple of waves. I'm sure there's been multiple of them. So yeah, maybe just getting a perspective of that would be helpful.

Shobhit Shrotriya (04:54.606)

Sure, and that's a very interesting question. If I go back 2006, 20 years back, when I was at Quintiles, everything in the clinical trial world was paper-based studies, right? We used to get tons and tons of paper, all the CRF.

Ram Yalamanchili (05:09.912)

This is like carbon copy type of paper. that what you're talking about? Yeah.

Shobhit Shrotriya (05:15.244)

right, cartons and cartons of paper, which we used to scan, upload, and then you used to have two screens, right, for data entry. So we had a function dedicated for data entry. So the source data was the paid paper CRF transcribed by the investigator. And then it comes to us from the site. We scan it, make it available for our data entry people. They had two screens looking at one side on the

physical copy of the CRF and then in the system Oracle was one of these systems at that point of time Oracle RDC remote data capture it used to be called and then there were two steps for data entry right you used to do a data entry so for example show bit is a data entry associate one I'll do the data entry and then ROM is a data entry associate two so the way to do prevent any quality errors to creep in you will

do the entry twice to ensure that there are no entries when the data is into the system, it is clean and is available for a data manager to then restart reconciliation. So that was fun. We had targets of 24 hours, eight hours, hours, depending upon the type of the CRFs and timelines depending upon if you are hitting a milestone, interim lock is coming on, analysis is coming or database log is coming, right? So everybody's running and ensuring that

how things can be expedited. And then slowly, year into this, I started seeing a lot of talk about electronic data capture, EDC coming. And Inform was one of the first platforms that came in. Inform is now a part of Oracle, Oracle acquired Inform over a period of time. And when EDC came in, the thought process was, everything is going to get automated now. I don't require a

data entry, even a data manager will not be required to do so much of manual reconciliation, et cetera, et cetera. And fear also crept in at that point of time. I'm saying now, everything was human-based. And now when we say electronic, that means now technology will take over the human and jobs will become redundant. I will lose my job. Those kinds of fears start coming in. However, even after...

Shobhit Shrotriya (07:41.091)

five years, 10 years down the line, nothing changed, right? And when I say nothing changed, yes, EDC just really enabled everybody in terms of ensuring that we are harmonizing the way the processes are run, how the data is entered, collected, curated, cleaned, but it doesn't just transform everything in terms of how things were being done really, right? So the manual reconciliation will continue to be out there.

using its fabric in that sense. When you talk about analyzing the data, is outside the EDC domain, say external data types, a lot of data coming from labs, ECG. Again, the manual reconciliation continues to be there. We used to program listings using SAS, for example. And I remember sometimes people didn't used to trust the outcome output from SAS.

I'm using functionalities, automated systems to say, OK, I need not review the data row by row manually because the output is there. It's already finding out the discrepancy between the system and telling me this is the action that you have to take. I couldn't believe it. I will still go ahead and redo the same work again. So rather than making it more efficient, the process became a little bit more inefficient because you're putting an extra effort to do it. So that's how I would say the

Ram Yalamanchili (08:52.248)

Mm-hmm.

Shobhit Shrotriya (09:08.28)

things evolved over a period of time. And now we're talking about AI, but I'll pause there, but this is how the evolution has been. I see over the last 20 years, we've made a significant stride, but there are still some lacunas that we need to address.

Ram Yalamanchili (09:21.814)

Right. And I, as you're saying this, right, one of the things I was thinking about is it's so true that when someone says, you know, as a society, we tend to overestimate the impact of things. You know, like especially technology, right? We overestimate the short-term impact and we underestimate the long-term impact. And I'm sure there were all sorts of concerns which came up when we went from remote data entry to electronic data entry. And clearly, you know, we found new problems. The industry has evolved to.

Shobhit Shrotriya (09:40.462)

Correct. Correct.

Ram Yalamanchili (09:51.853)

hopefully a better place than where it was 20 years ago. And we're doing more, right? I'm assuming the amount of work which has started off in 2006 to now has gone up dramatically. The industry has grown. This industry remains to be one of the more healthy, like, know, long-term CAGR industries, would say, in clinical research, like these long-term charts. I think that sort of tells a lot.

Yeah, it's really fascinating. So a couple of questions on that. You said there's this new technology called EDC which came in and obviously there's been a shift. Was the shift dramatic or did it take, know, what was the journey like from going from this one technology to another technology paradigm?

Shobhit Shrotriya (10:47.726)

Yeah, so I think I'll break it up into three elements right or three tenets right whenever there is a change the change is across people process and technology right technology being the enablers underlying foundation stone. When I look at the people I think there was a lot of mind shift mindset shift required at that point of time because you have to let go of your old ways of working.

Ram Yalamanchili (10:57.367)

Mm-hmm.

Shobhit Shrotriya (11:16.27)

to adopt to the new ways of working and get accustomed to it. The example that I was giving that though the technology was there, the ability to trust the technology and say, OK, I can trust the outcomes, outputs given by the technology to me. And I can not QC and spend a lot of time to rework on what has already been given. So that was a.

First of mind shift change and second was a cultural change, right? I'm so used to working in one operating model one way is a working so for me to tune to the new operating model new ways of working and also then propagating it within my teams Right. It has to if you look at it as an individual within a team When I want to make a shift, I want my other peers to also do the same I expect my leader to have that mind shift change. I expect

Ram Yalamanchili (12:11.256)

Mm-hmm.

Shobhit Shrotriya (12:11.338)

a top down kind of propagation to happen, which I would say took a lot of. Yeah, so from the people's side, it required a lot of change management in terms of coming to terms. So people were excited also to see what is the possibility, like what can technology offer to us? And then it was not limited to one EDC tool, right? Then metadata came up. So you had more variety.

Ram Yalamanchili (12:14.264)

Right, it's just like an organizational culture almost, right? You have to kind of go a whole estate for not just one person.

Shobhit Shrotriya (12:40.074)

and then more features and more functionalities coming in, which were also exciting, but at the same time of point of time, intimidating as well, right? A human doing something, what's the technology doing something? So that was one most important thing. The second dimension, which I was talking about the process, which we still continue to talk and we can talk about process more, but it also required a process change, right? Now you have a ways of working which...

a person executes in a certain way, but then you have your standard operating procedures, processes, work instructions, which define how the work needs to get done. That needs to be rewritten because in the earlier world of paper studies versus the new world of electronic data capture, you have to redo the entire process design. So that took a little bit of time as well. So it was a slow steady progress, but I think over a period of five years from 20...

2006 to 2010, I think things stabilized and people started to adopt and adapt to the new ways of working, it becoming a new norm. And then you say, okay, this is the way I would operate going forward now. So those were the two important aspects I would say that really took a little bit of time, but then things settled down and everybody then in the industry. So you're talking about being a pioneer versus a follower. Then when organizations started to see, okay,

Should I start moving all my studies from paper to electronic or a new study should I start in EDC? So that piece also started settling down. So sponsors became more comfortable saying that, okay, this technology is enabling us and enabling us in the right way.

Ram Yalamanchili (14:19.948)

Makes sense, yeah. I think one of the things which I'm curious about always is there's certainly been different innovations which came in in our industry. There's more to be earned for, right? Like it's not like we are done with all the problems and we're in a great shape. So in some ways, how do you look at the waves of technology which came in? Do you feel like they've accomplished what we intended to when we first started?

You know, like going back like 15 years ago, if you were to say, in the next 15 years, we're going to achieve all of this based on this new wave. How do you look at that?

Shobhit Shrotriya (14:59.223)

Yeah.

Yeah, so I would say when we started off from doing everything manually to starting to automate things with the help of technology. So you're saying, okay, you added an automation layer. Now from the automation layer, you started to look at, okay, I've automated an activity, but have I automated a workflow? Maybe yes, maybe no. It depends, But what did automation do it?

automated the way you were doing the transaction manually. Now, we were still away, far away from making it intelligent, right? So actually drawing insights from the data and started looking at it, okay, what is the informed decision that I can take? So from automation to intelligence, that's the first step I would say. Now, when you reach the intelligence, the data is the data unless you really curate the data and draw meaningful insights out of it. So,

And then again, data, if you look at it from the point of view within the sponsor organization, clinical data, safety data, regulatory data, everywhere the data is there, it is labeled in a different way, but it's all isolated. How do you bring the orchestration to bring the data layer together? So from intelligence and making use of the intelligence to orchestrate the data. That became one of the most important things.

some organizations started doing it in a right way and I saw roles like Chief Data Officer coming into picture, for example, which never existed, right? And the prime role of that data officer was to integrate and ensure that there is harmonization of all the types of data that is being collected across functions, right? Not that we have solved that problem even today completely, but that effort started to happen. And then...

Shobhit Shrotriya (16:52.896)

After the orchestration, now, if you look at it, the data has been orchestrated. Now, how you can make use of that legacy data or the data that is available for me to really plan for the future? And that's when I would say machine learning at that point of time, when AI started coming in, we started implementing that philosophy or that logic of now I can derive meaningful insights from the data and make the past data

train my new models, which I can use for my future planning. So that's how I would put it, from automation to intelligence, intelligence to orchestration, and from orchestration to more AI-led transformation.

Ram Yalamanchili (17:37.465)

Interesting. So one of the things you mentioned, I like this. You said with automation, we automated an activity, not a workflow. And I think that's a profound statement in the sense that I guess you need to do a certain number of activities before you can call it a workflow. Is that true? You essentially look at it as a string of events, and maybe automating a handful of things really doesn't get you to the end goal. I see.

Shobhit Shrotriya (18:05.656)

Correct. Correct. So if you look at it, when I talk about an, let's take a simple example of data cleaning activity, right? Now, or the value chain of the data cleaning itself, right? Data cleaning, you can break it down into small pieces. Now, I can do manual data reconciliation for the different types of external data that is coming in, say, serious adverse events, right? Then I have...

ECG data, I lab data. So I started to look at this data, but all the data in isolation. I wanted to see, how can I automate cleaning of SAEs, right? How I can automate the cleaning of ECG data? How can I automate the data ingestion for some other data type, right? But so these are piecemeal problems, right? So we started digitizing the inefficiencies caused by these activities rather than looking at

Ram Yalamanchili (18:55.373)

Yeah.

Shobhit Shrotriya (19:04.578)

how I digitize the entire value chain and redefine the workflow if I'm looking at any external data, right, coming in irrespective of the data source. I'm not looking at the entire workflow. I'm looking at only one part of the workflow, which is again inefficient, right? You may be doing a point solution, but you are creating inefficiency because other part of the workflow does not integrate with the previous part of the workflow. So that's what I meant when I'm giving that example.

Ram Yalamanchili (19:28.77)

Yeah.

Shobhit Shrotriya (19:33.624)

we have to look at as a value chain in entirety rather than a piecemeal automation.

Ram Yalamanchili (19:37.953)

Right. And this is also sort of aligned with how I've been thinking about this problem, right? It's like, what is the difference between machine learning and traditional automation? There's many forms of automation, know, RPAA, machine learning, like, know, stochastic, microchains. Like I've spent a lot of time in the prior era of like machine learning, just, you know, from a career perspective. And what's fascinating with the AI era is we have a new form of intelligence or new skills in intelligence, I would say.

particularly things like reasoning. We've never had reasoning up until, I would say, a couple of years ago in a box. So reasoning is such a fascinating thing, which has been uniquely human for many, I mean, forever. We've never had a strong reasoning kind of a framework out of a machine. And I think we've kind of.

come to a point where that is there. I think everybody can see that if you just go on ChaiGPD, it will reason through certain aspects and then that's how it's coming up with answers. And apparently if you can reason well, you become intelligent, or at least you do well in terms of what you can achieve. And so there's all these fascinating discoveries cognitively which I sort of have studied and it makes a lot of sense. What I'm curious about is given that we have some of this missing ingredients now.

looking forward to the next five to 10 years. And you mentioned there's a technology problem, people problem, process problem. We have to sort of adapt all of this together. We have these new tools now. So as someone who sits at an advantage point of potentially scaling the type of large operations you have today,

What are the things you're thinking about? How do you think the next five years will look? Actually, maybe I should step back and say, what is your time frame of change, which you're looking at when you're looking at investment right now? That could be a technology process of people. And what is your expectation from that time?

Shobhit Shrotriya (21:37.561)

Yeah, so maybe I'll look at first understanding what barriers we have today. For example, we are where we are. Now, I spoke about a little on the fragmented data ecosystem that we have. So the first and the foremost thing that every organization needs to do is to how to break the silo and bring the data ecosystem as a one integrated data ecosystem. I think that's the first most important priority.

that goes without saying whether we are talking about the AI-led transformation or we are talking about transformation which is driven through automation. Even RPA is example that you gave. So that's one important idea that I would like to break. The second important one I would say is once you have that ecosystem laid down, how do we actually standardize? In today's context, there is

And there are lot of efforts being made in terms of standardization, right? Both metadata and the data standardization, right? So that's the second important step, which is a subset of the first one, right? Then comes the second important thing, which I would say, and specifically I would say it applies not only to the AI world, to any kind of program governance, right? Do we have the right governance, which is top down, right?

Do we have frameworks that really enable us to take informed decisions at all levels? And is there a governing body which actually directs us? Given that we work in a very regulated environment around, FDA, MHRA, you name all the different regulatory authorities, the expectation of adoption of AI and approvals of medicines, drugs, vaccine based on the data that comes in.

which is using AI as a technology unless and until you have a framework on AI, which we call it in today's context, say a responsible AI framework or governance that has to be in place. The third one is everybody is wanting to experiment on different, they're picking up a workflow, they're picking up a problem and running a lot of POCs and pilots. I would say we are fatigued with the pilots.

Ram Yalamanchili (23:37.303)

Mm-hmm.

Shobhit Shrotriya (23:58.093)

We have to really, really look beyond pilots and industrialize those things rather than just continuing to do those POCs. So that's another thing I would like to look at. Then comes what I mentioned earlier is the people part, which is more to do with the cultural resistance that we have. In today's context, for example, when we talk about, again, taking an example from the data management world, you're no more going to succeed

the way you are doing the current role unless and until you upskill yourself and move, make a transition from being a data manager to a data scientist. And nobody is expecting a data scientist should start programming, right? That's a different expectation, but aligning to the new ways of working and ensuring that you understand the nuances of what is expected in the new world is important, right? And I think a lot, many industry consortiums like SCDM itself, right? We have...

priority and we published a few papers that talk about transition from data management to data science and how do we make it happen. So every organization needs to have that program. Every service provider needs to have that program, whether it's a technology service provider or a services like professional service provider. Everybody needs to have that program to bring that change happen. And finally, I think

Ram Yalamanchili (25:01.068)

Mm-hmm.

Ram Yalamanchili (25:18.242)

So.

Shobhit Shrotriya (25:23.134)

overarching on all top of this is have you defined your what are your ROI models, right? You're investing so much in technology. Have you really defined how the return of investment should look like? Right? Sometimes the models are there, but they're also again, fragmented, not structured. Sometimes we just put numbers for the sake of putting numbers. I don't see a real ROI model that really defines end to end value. And at the end of it, we are just trying to crunch numbers to say, okay, I did a POC.

what's an MVP or a POV, what's the actual benefit. So these are the four or five things I would say, not only me, anybody and everybody should look at if we have to embrace the technology or the changing ways of working.

Ram Yalamanchili (26:05.152)

Makes sense. there's several points you brought up, which I'd love to double click on. So first is we're talking about governance, right? And governance is a policy. I mean, it's a process. It's a people. I think it involves multiple things. It also feels like it's a new thing, which we will have to now learn on how to govern an AI. I don't know if you agree with that, but to me, it feels like a new science, new area, new exploration. That's how to get that right.

So maybe I'll start there, right? I'm certain you're adopting investing quite a bit in these types of technologies we're talking about. What's the framework or how do you think about it? I mean, this is all new territory, right? And I'd be curious on how you're thinking about governance. My second point, I think if you can lead into that is the fact that you brought up POC fatigue. We have heard about this quite a lot of it has been written.

I think the one which often is quoted as this MIT publication from last year, which says 95 % of pilots which enterprise are adopting are failing, which is fascinating, right? Because I could give you 95 % of the cases where when we adopt AI, we've basically just gone full tilt. mean, it's like, code gen, for example. There is no going back. This is like the future. It works. It's amazing.

Shobhit Shrotriya (27:10.442)

Enjoy.

Ram Yalamanchili (27:27.736)

very specifically tuned our organization to find the right type of people to make it work and of course upskilling them and you know all that's all that good stuff. So I think I'll be curious on governance plus sort of how you look at the future of the adoption cycle as well because pilots I agree are not where things are I think 26 feet is different but I'd be curious on your perspective.

Shobhit Shrotriya (27:50.265)

Yeah. Yeah. Yes. On the governance side, I think it's very simple, right? So it's not governance in terms of its terminology is not a new word, right? Pre-AI era, we used to have a governance, the way we are running operations, the way we are running our organization, the way we are governing our people, the way we are governing technology, infrastructure, processes. So governance always existed. Now,

Ram Yalamanchili (28:08.536)

Mm-hmm.

Shobhit Shrotriya (28:17.112)

you need to have an additional layer of governance because you're saying, OK, AI has come in. You need a special layer of governance because you want to ensure that there is ethical and responsible use of AI. That's how we term it. And when I say that, just as an example, so I'll digress for a minute to quote an example of what I mean by responsible use of AI. Like a human, AI can also be biased.

So we as human have our own biases, AI will have its own biases. And I'll give you an example, which may be a decade old, but which is very, very relevant still. I'm talking about say 2016, 2017, I was reading one of the papers, some research was done in one of the German universities and they were looking at building a CNN for detecting skin cancer, right?

And there are lot of data scanned images, all modalities coming, whether it's a mammogram to PET scans to other modalities being collected. Once the model was created, the researchers fed in all that data to see the accuracy of the model and test it against oncologists or radiologists' assessment. And the outcome was 95 % accuracy.

What it means is the radiologist assessed malignant versus benign and the model assessed malignant versus benign and 95 % of times the outcomes matched. Brilliant outcome, right? Isn't it? Then they said, okay, let's do one more step and do a little bit more of experimentation and testing to validate that the model is working fine. Another set of images were fed into the model.

Ram Yalamanchili (29:55.49)

Mm-hmm.

Shobhit Shrotriya (30:14.582)

and the results were alarming. The accuracy, which was, 95%, has dropped to, 50%. And guess what? What's the answer? The answer is as simple as the first set of images that were fed in for all white skin-colored people. The second set of images that were fed in were all brown skin-colored people. So the model failed because it was trained. It learned from

Ram Yalamanchili (30:30.252)

Yeah.

Shobhit Shrotriya (30:43.082)

a training data set that was all white skin color people. So this is a bias, a machine having a bias and giving you a bias. So now do I say that the model was inappropriate, incorrect, made inaccurate decisions? And if I would implement it on a clinical trial, will it give me the right outputs? No. And that's why this element of responsible and ethical AI is important.

Ram Yalamanchili (30:44.63)

Yeah, it's like overfed to a particular data set essentially in this case.

Shobhit Shrotriya (31:12.64)

and all organizations, I'm not talking about your organization or my organization, in general, the sponsor organization or the technology service providers or product developers or the services organizations who are using these as end users, everybody needs to have that framework to ensure that you have proper governance before any technology, AI-led technology gets implemented. You test it in a way where you're also testing an element of ethical and responsible use of AI.

So that's the first part of the question that I would like to and yes, there are various frameworks to it and those are all available in public repository. I will not drain in terms of how the framework would look like or should look like. I also actually wrote a paper on ethical and responsible use of AI for SCDM that white paper is available on the portal in the public domain. People can have a look at that as well.

Now, the second part of your question, is very interesting, the fatigue with the pilots, right? Now, I think the approach has to change from what I can do to solve a one-point solution to a more process-led approach. And that approach cannot be in isolation. So what I'm trying to say is, when I'm working for a particular sponsor organization, that we need to lay down

First, let me take a step back. What are the industry problems that we have? We have to look at holistically. These are the industry problems that we have today, which we would like to solve for. And in today's context, agent-tech AI or gen-AI or AI-lad or machine learning-lad solutions are the most appropriate ones. And that cannot be, again, done in isolation. And you cannot just pick up a problem statement and start working in isolation.

I as a service provider, if I develop a product, if I don't have the right amount of data to train and test my product, I do not know what the outcome of that product would be, what the output would be. So I think to move away from that problem, I think we have to come together as a community and solve four problems together. consortiums like Translarate, for example, where so many sponsor organizations are coming together and saying that,

Shobhit Shrotriya (33:40.633)

This is a common problem that we have and we have to come together to solve for that problem together. Right. And we would be committing to make the data available to the product developers, the technology service providers and the FSPs to help us train, test and figure out what the best optimal solution could be. Right. think so that's again, a mind shift change. That's a governance change. That's the way you will go away from the pilots and getting fatigue with the pilots to

Ram Yalamanchili (34:05.079)

Mm.

Shobhit Shrotriya (34:07.394)

concentrate and pick up relevant problems which are there in the industry and then work together as a collaborative unit in a community form rather than working as an independent entity. That's how I would put it would be the best possible way.

Ram Yalamanchili (34:21.622)

Gotcha. So much to break down there, but maybe one perspective I'm curious about is, have you also noticed this type of I guess, like readout, which we have seen in the past papers, like 95 % of pilots failing? I mean, obviously, that's a broad question. Why did they fail? What exactly were the failures and things like that? But I'm just curious, in your experience, would you say that's kind of been the...

the norm over the past couple years and why. If it is true, I'd love to understand from your perspective why does the wait us.

Shobhit Shrotriya (35:00.194)

Yes, so I think what is important to understand is when we say the word fail, right? What does fail mean? Right? Sometimes that word is loosely used, right? I went ahead and wanted to solve a problem. Maybe it did not solve the problem in entirety, but did it address a part of the problem statement or the opportunity area that I had? Right? And the failure is not about the technology. The failure is also about two other components, which I will continue to say people and process.

You went ahead and solved up, tried to solve a problem, right? But did you understand what the as-is process was and what the to-be process should look like? And maybe I did not think that before I start implementing a technology to start solving the problem, let me first make an attempt to redesign the process and ensure that the redesign process should have, if there were 10 steps, 30 % of the steps become redundant. I need only seven steps.

what we did we fitted the technology to also look at those three steps which are redundant actually or should be redundant. Right. So.

Ram Yalamanchili (36:05.954)

So you had to eliminate them almost, but that was not part of the calculus when you looking at it from that lens.

Shobhit Shrotriya (36:15.052)

Right, so we jumped to the solution, but we did not understand that before I jumped to the solution, I need to look at my process and see I need to change my workflow itself if I have to solve the problem. Right, and you started to solve a problem which did not exist. Other example, I was talking to one of somebody yesterday and this is fresh coming out of my memory.

Ram Yalamanchili (36:23.457)

Mm-hmm.

Shobhit Shrotriya (36:39.16)

trying to develop a data entry bot, right? Okay, sounds very simple, nothing big about it. And so, and while you're developing a data entry bot, you have to test the outcomes being developed. So you're doing programming for a particular study, you program the edit checks. Now somebody has to manually enter the data to test the edit checks, all the different scenarios, positive, negative, all permutation combination.

As a human, can write n number of test cases, right? But a machine will write hundreds and thousands of test cases, data entry, and you're sorted.

The product that was developed was like, and for a particular study, like in today's context, I'll just make it up and say, if it takes three days to do a manual data entry to test, I would expect it to take three hours when I have an AI enabled data entry bot, right? That's the efficiency you would be expecting. Here is somebody or a set of team members who built a product, right? And the product is super efficient. may do the...

data entry work in three hours. But the time required to configure the bot study by study is three days. Right. So every study if you are spending three days to configure and then run the bot, it will give the output in three hours, you are actually becoming more inefficient. So if you don't fix the problem upfront, so the problem is not about the data entry. The problem is how would you configure it?

universally for any study, nobody looked at that part of the problem. They went ahead and solved the problem for how do I automate the data entry. So, and in that context, if you say 95 % of the implementations will yes. Right. So,

Ram Yalamanchili (38:31.862)

Yeah, it achieved what it was supposed to in a, I guess in a certain perspective, but it didn't actually solve the whole problem, right? Like that's where the failure rate came in. see. That's another fascinating thing. think, you know, it's, as you said, one of the interesting aspects of the new class of AI technologies is also the fact that they have some amount of ability to learn or quickly learn, I would say.

Shobhit Shrotriya (38:43.978)

Yeah.

Ram Yalamanchili (39:00.498)

know there's three four percent learning, there's continuous learning, there's all these new areas we're exploring. I think to a certain extent, we've also never had a system which could actually learn as quickly as they are today, which is also fascinating from a future five years perspective. So one of the things, before we wrap up, I'd like to touch on is you're describing quite a bit of reality which needs to catch up.

when you have these kinds of technologies. There's clearly process, there's people, there's technology itself. I feel like right now a lot of the focus on the new cyclism, just one part of it, which is technology. And I think technology has made strides. We are at a place where these technologies are quite efficient, quite powerful. There's a lot going on there. The other two, think, need to catch up. Not enough leaders are talking about that. And that's where I think more innovation, more thought process, more learning.

Hopefully you'll work with SDTM that, you these sort of areas that I think like clearly like lagging. We see that for sure. And many times we're educating as well. We're in front of customers telling them, hey, this is the policy or this is the model you'll have to start thinking about, right? Because these processes which you're thinking about may not make sense. I know you're saying that, but if you think about it and step back, this does not make sense. And we have to basically eliminate this process. I'm curious when you're looking at all the different technologies in front of you,

Shobhit Shrotriya (40:12.397)

Absolutely.

Ram Yalamanchili (40:23.682)

You know, there's a lot there. It can feel overwhelming in terms of like, do you actually like evaluate what makes sense, what doesn't make sense? you know, I do see you as somebody who's thought a lot more about the non-tech pieces as well out of the ecosystem, that you have a kind of a leadership position there. What's your framework been in terms of just saying, okay, like given I've made progress on people and process, does this technology work with the rest of my ecosystem?

How do I evaluate it? How do I know if this vendor is actually going to work? Some of the problems you've just mentioned, like biases and things like that, which are common, have been common in the machine learning world with overfitting and data distribution issues, things like that. Do you have any learnings or anything you can share there for others who are probably going into the same cycle where they're evaluating and potentially going through this possibility to roll out?

Shobhit Shrotriya (41:19.96)

Yeah, so that's a very interesting question and important one as one important one as one Ram. The way I would look at it is it's not only about one particular technology, it's about its interoperability. It's also about fit for purpose and the way you would like to implement that technology, right? So those specific

requirements continue to remain the same if I keep the infrastructures aside right it's more about how do I make use of the technology available at hand in the most effective and most optimal way right which includes your see again come going back we work in a very regulated environment so GXP becomes the foundation layer so compliance to regulatory standards that goes without saying in our context we cannot

bring in a technology which is not working in the regulated environment. So it has to meet all the GXP requirement. That's the fundamental base. Then the second important piece would be when we are looking at system integrations. Is it able to help me integrate different systems together and I am able to seamlessly do data ingestion and able to then ensure that it

it's able to give me the desired outcome which I can take it downstream. upstream sometimes the upstream integration becomes easier but the downstream integration is poor right and then comes the third important dimension which is the scalability. may one piece of technology may work pretty well in a controlled environment but the moment you start scaling it up and industrializing it you see a lot of

Ram Yalamanchili (43:12.94)

Mm-hmm.

Shobhit Shrotriya (43:17.006)

So that's the third dimension. And the fourth one I would say is if it is agent-tech AI, AI-lad, the framework that we just spoke about becomes one of the overarching umbrella in terms of does it fit the larger bill of ethical and responsible use of this new technology governed by AI, right? So these are the four fundamental principles I would implement or would like to

take it as forward as an evaluation criteria for assessing any kind of technology. That's in a very general broader sense, but then there are nuances to each one of these aspects as well. And we can drill down and go into data protection layers to security layers to how the performance of the system is when you scale up, et cetera, et cetera.

Ram Yalamanchili (44:07.832)

You know, I think there's too much to unpack perhaps with this particular podcast, but I think one area I would be curious about the final topic is you mentioned when you scale up the systems have to be performing the way they were in a non-scale approach, right? Which is a very, very, I think like really reasonable approach or reasonable requirement.

Particularly in AI, one of the things I notice is because AIs are stochastic in nature, they're property-based. And anyone can kind of like double-check this by just going on chat GPT asking the same question across time. And there is no guarantee that it'll answer the exact same what-do-what answer, right? It is a feature. It's not a bug of that kind of system. And hence, that's just the way these systems work.

Shobhit Shrotriya (44:53.132)

Absolutely.

Ram Yalamanchili (44:59.67)

I think if you think a bit more about that type of a system, then you think of it as, that means the system is not consistent by design. It's not potentially reliable in a traditional sense by design, if it's all property. But it tends to be very, very much aligned for a large enough number of bets. It's more or less accurate. The reason I'm pointing this out is, in a small scale, guess you could tune the system to be right.

but how does it behave in a large scale is up to another question. It's an entirely different question. Is that how you think about, like, you know, essentially when you think about deploying AI, like there are several gates to essentially how to evaluate on, right, in this case.

Shobhit Shrotriya (45:44.409)

Yeah, so I think the interpretation part, which you just gave as an example, I can say the same thing in different ways, right? That does not mean that as a human brain also, right? If you ask me the same question five times, Shobit will answer in five different ways or five different statements, right? The essence of it has to be the same. And that's why we say human in the loop is the most important thing that we cannot do away with in the entire context, right? What I mean by that, there are

Ram Yalamanchili (46:08.29)

Mm-hmm.

Shobhit Shrotriya (46:14.548)

areas where you have to say, for example, take medical judgment. Now, AI-led transformation technology can enable you to give a recommendation. And I'm using the word recommendation carefully, but it's not making a judgment call. And the same recommendation can be given in five different ways or 100 different ways. It's left for the medical judgment or a human to make the judgment.

There can be safety considerations, for example. There can be regulatory requirements, for example. So the human in the loop should facilitate the scalability to ensure that we are not rubbishing the output given by the AI model. So that's how I would put it. So it has to be a combination of both human plus machine working in tandem to ensure that if

I'll consider AI as my another teammate, right? The way you call it, right? It cannot be. So if Shobit and Ram are working on a project, if Shobit gives a recommendation and Ram is validating it, and tomorrow it is not Ram is another teammate, machine teammate, I would expect that machine teammate to do the same, right? And that's how the concept of scalability does not go away without the human in the loop not getting away, right? So that's how I put it. So I'm not.

Ram Yalamanchili (47:14.183)

Mm-hmm.

Shobhit Shrotriya (47:42.03)

undermining the importance of scalability and industrialization, but with the caveat that we should not undermine the capability of an AI-led output just by saying that it is answering in five different ways.

Ram Yalamanchili (47:53.025)

No, that makes a lot of sense. It's like coming back to that idea of how does it integrate and not just into other technologies. I think what you're describing the way I look at it is how does it integrate with the person working with it. Right, like the human in the loop is so key to this whole unlocking of the puzzle.

Even, I mean, for example, we've spent a lot of time in figuring out what are the right patterns to introduce the human in the loop in these journeys, right? Because if you were to do every single call has to be validated by a person, then what are you really solving? It goes back to what you I think earlier said. It's like your SAS programs are, you know, throwing out your TLFs and, maybe doing some analysis on top of it. But if you don't trust it, you want to go still one by one, like every single line, then there's no efficiency to begin. Right. So designing the system in a way where you can sort of

Shobhit Shrotriya (48:38.275)

Absolutely.

Ram Yalamanchili (48:40.248)

know, be intelligent about where the human loop comes in and how do you like essentially do that well and showing I think a strong mathematical foundation on why you've taken these, you know, these steps and, you know, heuristics on what's the data look like and the evals look like. So I think there's a whole lot of fascinating questions and answers which need to be figured out.

Which I'll again go back to what you were saying, which is like, what's my governance policy? And in this case, it's an AI governance policy, It's specifically for this particular field.

Shobhit Shrotriya (49:17.038)

So I wrote something down as a closing remark from myself, right? I wanted to, so I wrote it down purposefully and I wanted to read it again, because you cannot recite it the very same way all the time in those words. So I wrote down, AI will not transform life sciences because it is intelligent. It will transform life sciences when it becomes trusted, standardized and embedded into our

Ram Yalamanchili (49:23.788)

Mm-hmm.

Shobhit Shrotriya (49:46.286)

trial designs, ability to manage the data the right way and serving the patients. So the winner won't be those who are experimenting the most. The winners will be those who are industrializing, underlying responsibly. So I think that's how I would summarize what I wanted to convey.

Ram Yalamanchili (50:01.238)

Mm-hmm.

Ram Yalamanchili (50:06.422)

No, that makes a lot of sense. It's a very deliberate process of how you go about this rather than, like you said, a pilot, a proof of concept, which is let me take a small problem, try to solve it, and may or may not work ultimately on a broader scale.

No, this is great, Shobit. Thanks for sharing. I've learned a lot. So this is always the fun part of my day when I do this. So I appreciate you spending the time with us. Thanks for being here.

Shobhit Shrotriya (50:40.308)

Thank you for the opportunity Ram. Likewise, the feelings are mutual. I also enjoyed the conversation, a good conversation and this can go on for hours and hours actually.

Ram Yalamanchili (50:47.916)

Yeah, that's the fun part, right? We've got to stop at some point. But yeah, no, that's great. Thanks again and we'll talk soon.

Shobhit Shrotriya (50:54.446)

Good. Cut it.

Ram Yalamanchili (50:59.949)

Yep.

Shobhit Shrotriya (51:01.186)

Thank you so much, Ram.