Cloud Reset – The Podcast | Episode 2: AI – Promise, Potential, and Pitfalls
Show Resources:
Here are the resources we covered in the episode:
Follow Naran McClung on LinkedIn
Contact us at enquiries@macquariecloudservices.com with any questions, suggestions, or corrections!
A great no-cost way to support us: Rate/Review!
Episode Summary:
- The rise of AI washing and how businesses can cut through the hype to identify real AI value amidst exaggerated vendor claims.
- Balancing AI investment and outcomes, with examples from Microsoft Copilot and Macquarie’s SOC Digital Twin, highlighting cost-effective wins and evolving ROI.
- Productivity as a priority—the fastest way for businesses to see returns from AI is by automating workflows and reducing operational costs.
- Data governance as a foundation for AI success, stressing the importance of data readiness and ownership to avoid inaccurate outputs and compliance risks.
- Building AI talent organically, encouraging businesses to foster tech pioneers through innovation frameworks, competitions, and collaborative environments.
Episode Transcript:
All right, welcome back everybody to cloud reset where your host Jonathan staff and Naran McClung straight talk real solutions maximizing the ROI for your IT investment Today should be a pretty interesting episode. We promised to talk about AI. We promised to talk about AI washing We promised to talk about whether or not AI lives up to promises.
What’s AI washing Jono? Well, I think we’re going to get right into that. It’s everywhere, right? You can’t read a news article now without reading something about AI. AI is in every product, seems to be in every news article. At once upon a time, it was security, right? We’ve gone from talking about security all the time to talking about AI.
AI powered security? What about that? Let’s get into that. Okay.
AI is in every conversation. It’s in every product, uh, it seems to be a promise that’s linked to outcomes for just about every SAS provider in market at the moment. Our dear friends at Microsoft have a copilot for everything. If we look at copilot for M365, it’s fair to say, um, there’s a healthy mix of businesses that, uh, just flat out tell me they can’t live without it.
Now, it’s doing such wonderful things for them with productivity, particularly with workplace services, with Outlook, with meeting summaries, etc. There seems to be a lot of value to be added there. I would say, uh, there’s a, there’s a degree of evolution of product and capability as we see Copilot for all the other products come to market.
There’s a Copilot for Sentinel, there’s a Copilot for Fabric, etc. Uh, I know that when Microsoft kicked off their financial year, they’re really interested in, Azure and AI particularly wins. So what is an AI win? And that’s really where there’s been a use case or a problem solved with a customer, and there’s a material benefit being offered by the use of things like Azure OpenAI.
Now, as a perspective, as an MSP, what are MSPs like us, Jono, trying to do? We want to do more with less too. So if our people can be working on more meaningful things and if they can be doing more value add, that’s a good thing. So we had to reflect on our own business with AI. I know we looked at our security business.
There’s a couple of key metrics within our security business that are hugely important to us. Uh, our mean time to respond and our mean time to closure. And I know that AI has helped us wonderfully in that space. Um, and Equally, uh, when we look at our customers and our customer journeys, when we look at customer success and, um, all the different things that we do with our customers, whether we’re tracking net promoter score, maybe we’re tracking movements of customers as they would perhaps move from one business to another, et cetera.
There’s a whole raft of intelligence around insights and things that we can learn from our own customer base. And I know that’s a project internally for us as well. So it’s had a big impact on our business already. I would say I’ve been pleasantly surprised with the cost of those AI technologies for us.
If I look at just our security capability and the development of our SOC digital twin, to be really honest and transparent, this is what this podcast is, it’s costing us less than 500 bucks a month to essentially eke out an open AI insight that allows us to drive down mean time to respond and mean time to closure to almost record levels.
It’s had a wonderful impact in our business for very low cost. Equally, then when I look at the cost of copilot across our whole business, it’s more expensive. Um, and I would say, um, the benefits for us, um, we’re still getting to know them. Um, certainly I’ve enjoyed meeting summaries and, and things of that nature.
Um, but, If I look at the disparity of value there, uh, and how it’s improved our security business, like, there’s a huge difference in cost there. I’m not saying Copilot wasn’t value for money at all, but I’m just saying for 500 bucks a month, we’ve been able to, uh, deliver service improvement to each and every security customer we have today.
So, massive difference there, um, and I think maybe this is a sign of the market still levelling out. Um, what are you seeing in market? I’m seeing a lot of interest, a lot of interest and, um, and, and not, not so much clarity around what the practical use cases are for this technology. You know, um, I know, uh, I know in the, in the early stages of this, there’s a lot of interest in large language models and, you know, how that can help, uh, particularly professional services organizations.
Um, A lot of the conversations we’re having around, do I have the right skills and how do I take advantage of this technology and how can it make a difference for my business? Yes. I’d say most of, and not, not too dissimilar to us. Most of the real practical use cases that are getting real investment or attention from business owners at the moment where they’ve got clear line of sight to a return around productivity.
Yes. Definitely productivity. Yep. Uh, anything that can save you time, do more with less. The net effect of that is, again, I know in the last episode, we talked a lot about driving down cost and risk, uh, anywhere where AI can be leveraged to do one of those two things or both. Yes. I think there’s a lot of interest.
So. Um, we saw that we saw that within our security business. I think it was something like 500 man hours of sock analyst time that it saved us. But I think better than that, it was, yes, there was a time saving, but it was also our sock analysts are then able to work on higher value things. Uh, and it means that it drives down that mean time to respond and it drives down that mean time to closure.
So it has a material impact on the service levels that we’re able to offer our customers, which was a good thing. Now Internally, I know in our own customer success project, um, speaking to the guys, the models are evolving so quickly, right? Like one week they’re working on a model and it’s giving them reasonable results.
Next week, there’s a new model that was available. It was the next iteration of Llama, for example, that came out. All of a sudden they’re working on that and they’re getting demonstrably different insights out of it, and it completely flips their engineering on its head and they start again. Now, is it that they’re going to work for another two weeks before another model comes out?
I mean, it’s, this whole space is moving quickly, and it’s knowing which models to use as well. Uh, I know our team have been working on this in real time. Yeah, I agree. I think it’s definitely moving at pace, hugely disruptive to our industry. Um, like I said, a lot of the customers I’m talking to still in the early days.
I know though, if you say go to North America, spend some time there at Dell Technologies World, uh, earlier in the year. markets a bit more advanced, uh, more investment. There’s a lot more commercial off the shelf, uh, AI powered options available for you in productivity suites. And we’re starting to see that come out in Australia.
I think as well, there’s a lot of scale up and startup. Activity in this space, this, you know, and this comes back to that AI washing concept that was sort of born off the back of greenwashing. Um, you know, and you’ve seen a couple of organizations, uh, around the world be sort of prosecuted for this and misleading shareholders and this sort of thing.
I think, um. really is uncharted waters. It feels a little bit like com, you know, and, and everybody knew this tech was going to be great. No one was really sure how everyone, you know, we’re in this period of everybody trying to figure it out. And, um, I think it’s, I think it’s going to be increasingly hard for, Australian companies to discern, you know, where is the right investment here?
Maybe I can get some productivity gains. Okay, I’ve got all these options. Who’s telling the truth? Yeah. Will the tool work? No one knows. There’s not really a, you know, there’s, there’s nothing around that’s got a five year track record of delivering AI powered outcomes. No, that’s right. At scale. This is, so how do you, how do you decide what’s good and what’s bad in terms of where do you make an investment?
Look, I hope that, um, Our own examples as an MSP, and you will know this, Jonno, obviously, that we’re not precious about our intellectual property. If we produce something, we produce it with reuse in mind, and I know the team have been building modules, for example, of customer and data obfuscation models, which is hugely important, right?
If we’re going to take advantage of these publicly available LLMs, we need to protect customer interests and identity and intellectual property and things of that nature. So there’s a body of work and best practice around that already, um, but. More, more than that is the problems we solve for ourselves may well be applicable to our customer base, and I think it’s going to be easier for us to have a conversation to say, well, here’s something that we did.
Here’s what the data repository looked like. This is the data model that we needed to produce to get an outcome. These are the different, uh, Uh, models that we had available to us as well in terms of LLMs, et cetera. Uh, I know the Graph API piece that the team were taking advantage of. These various aspects of the capability that solve specific problems for us.
And I, I like to think that in the fullness of time, we’ll have a catalogue of capability based on our own experience that we can then say to a customer, maybe this use case will apply to you. Uh, you have the same problems. I mean, mental health issues. Most of our customers have customers would be interested in customer insights.
I don’t think we’ve got a single customer that doesn’t interface with teams. You know, all of that work I have to assume would be valuable to someone going forward. Yeah, I think so. I think sharing and openness is going to be really important, um, especially in the tech community as, as we try and figure this out.
Um, there’s definitely a lot of traps. Um, And I think for me, uh, if, if I’m a business owner looking to make an investment, you know, put real money into something I’m looking for a gain right now, the most obvious ones are productively productivity related because it’s measurable, you know, and if I’m, if I’m looking at that and customers are saying to me, well, what, what, what should I do?
We’re looking at co pilot, we’re looking at this, a commercial off the shelf thing, we might build our own thing. There’s a service now integration, whatever it might be. Um, that productivity gain is the easiest thing to measure, right? So you can’t, you know, measure what you can’t, you can’t manage what you can’t measure.
Of course. Right. So, um, like our one with the man hours and mean time to resolve. Yes. 500 man hours. It’s a real thing. That’s such an easy business case. That’s right. I’m sort of encouraging customers to look for those. Yes. Don’t, don’t bite off more than you can chew. So
every episode we go to a listener question. Uh, this one’s actually. Pretty interesting. They want to train a large language model on their knowledge base. They’re a professional services organization. And like a lot of potential customers that we talk to, they’re interested in productivity gains. So this is, you know, how do I really quickly answer a question that I’ve answered before?
Um, they’ve got a pretty reasonable knowledge base, but they’re looking for a really clever way to enable their people to move a lot faster at scale, uh, in terms of answering. customer questions. Now, they’re actually curious about what to do next. And the reason they’re asking this question is, well, they thought the right answer would be to get some consulting.
Um, the quotes that they’ve been given for consulting, uh, pretty eye watering, frankly, and there’s no real practical outcome at the end of that consulting exercise. So they’re wondering what should they do next? Uh, for me, I think just the first bit of practical advice, I’d be very interested in. data governance, you know, and are you aware of where all of that data is that you need to train this model on?
You know, the data sets that you need to actually provide the sample sizes for an effective LLM are actually massive and they can grow beyond the knowledge base that you have. So you can start to pull data in from multiple sources. I know even in our own business, The other day we were looking at our CRM and if you go back far enough, I’m sure everybody’s CRM is like this.
It’s tens of thousands of contact records as an example. So there’s a hygiene factor and a data governance factor that you’ve got to get started on first. And it’s not really until your data house and backyard is in order that you can start to do anything meaningful in terms of developing your own LLM.
I wonder whether there’s a hierarchy of needs piece here and it’s similar to the security conversation that. Was up until recently, the most popular conversation still is very popular. Um, I wonder if there’s a base level of maturity that an organization would want to be satisfied with before they feel like they’re in a position to really take advantage of a homegrown organically built AI capability.
Let’s talk about that. Um, you know, there is so much to, uh, getting, getting ready to do it yourself. Right. You know, particularly around data governance. Yes. Um, actually understanding where your data is, who owns it, um, getting it structured in a way such as that you can take advantage, um, of training a model on it.
That’s right. Um, being confident that, that when you do, you’re not going to get, you know, false positives or hallucinations. Yeah, there’s so much to that. Um, and then, you know, there’s a pretty big cost associated with that. I haven’t really met many, uh, Australian organizations, unless they’re in the business of data, you know, and that’s how they make money, um, who have the level of maturity required around data governance, who would be honestly ready.
to take that step and do something meaningful for themselves. Yeah, I think that that’s, isn’t that just the next evolution? You know, if we think about, like, the security story again, and I keep sort of going back to it because it feels like there was a level of maturity and evolution that happened there that will happen in the same space with data and data governance and modelling.
And I’m wondering whether, within an organisation, We need to start thinking about who are those data domain custodians? Who are these people who are responsible for data sets? And it’s not, move away from, you know, IT or service provider being responsible. I mean, ultimately there’s systems involved there that have really nothing to do with that logical understanding of data and what it can be done or how you can take advantage of it.
There’s probably a huge body of work there, I think. Um, organizations are worried where their risks are, clearly. So, you know, that sort of classic thing of who has access to the data, um, let alone who’s responsible for it. And then coming up with a meaningful and useful data model, piecing these things together such that you can draw out an outcome.
I think it’s a, it’s a new body of work and thinking and skills and capability. Um, and I think, look, Nothing speaks better than experience and use case. And I’m hoping that, you know, today we’ve got two or three examples within our own business. Um, hopefully, you know, three to six months time, I’m talking 20, 25 examples, things that we can say, look, in real terms, in a real practical way, this is how AI helped our business.
Does this relate to your business?
Yeah. So important. You touched on something really interesting there. And, um, yeah. It’s, uh, it’s a, it’s something that’s very, uh, common theme right now in terms of current events. Talked about skills, you know, just the, the skills and the type of person that is required to get your data into shape, right?
And whether or not, you know, you think about, um, how fast this is coming at us, you know, Australian businesses are going to need to adapt. Yes. And overcome to stay competitive. I don’t think it’ll be long before Um, if you’re not leveraging AI in a meaningful way in your business, even if it’s just for productivity, you know, you, you won’t be able to compete with people who are.
Yeah. It’s interesting. You’re like within our own business, uh, there was almost like a natural gravitas towards this for those people that if you had have said to me now and pick the 10 people in the business, they’re going to lean into this thing on their own. Um, I probably would have guessed at least eight out of ten of them within our own business because they were the same tech pioneers that led us on a journey with infrastructure as code, for example, you know, they were the same tech people that gave us product ideas for modules on our security offering, like, you know, breach attack simulation and things of that nature.
So they’re already thinking. Automation, they’re already thinking programmatic methods. They’re already thinking along those lines. And those people, at least with our business, had a more natural affinity to embracing AI models. They were working in code, for example, already. Like that was a, um, a discipline that they were comfortable with and Um, I don’t think you could ever really force that on anyone within a business.
I think the interest probably has to be there. It’s like, it’s a little bit like trying to change someone’s religion. You know what I mean? Like, good luck with that. Yeah. Well, you have, um. Yeah, I think, I think that’s right. The industry right now is relying on, uh, those naturally talented people who, uh, who will gravitate towards this tech and find innovative ways to, to make it meaningful for business.
You know, um, I think the challenge ahead of us is at a grassroots level. How do we make more of them? Yeah, true. How do we, how do we be deliberate about it? Yeah. Like, um, I guess I’m hoping within our own business that our, our sort of AI pioneers will create real interest for those around them. Um, I know that it’s a topic of conversation when we hire now and our graduates coming out of university, you know, and we obviously do a lot of that with our hosting management center.
We take on. You know, people that are fresh in industry, they’re probably looking at their own future and they’re probably looking at us really as a service provider to provide the learning paths and the opportunities for them to embrace things that may just be interests, right? So maybe for us, at least it’s a case of propping up our, our AI subject matter experts for lack of better description and putting people around them that may be interested to learn.
And then hopefully from that we’ll get real organic, organic swell of skills interest. Um, within our business. Yeah, I know. Um, you know, implementing frameworks to encourage those people is important. Um, I mean, in our business, we, we stood up obviously camp AI and identified all of those interested people and we’ve got them all in one place and it’s rewarding.
And, you know, um, there’s a lot of hype around. I think that’s been really good for us. It has. And I know out of that, um, that was where sock digital twin came in. We’re sort of born, maybe you could tell us a bit more about that. Well, yeah, so, so Camp AI, like, you know, again, as an MSP, we look for, we’re looking to obviously surface up the skills and interests in AI.
So it certainly did that. And you run a competition, you know, with some, some decent prizes out the other end of it. And people managed to find time, those that are at least interested in racing. Amazing what people would do for a prize. 100%. But I mean, it starts with the interest and the acumen. You know, and the desire to sit there and hash out a solution.
SOC Digital Twin was amazing for us because it’s obviously live now. It’s something that we’ve deployed that just started out as an idea with one of our lead or top SOC analysts in the business who, um, really wanted to embrace the technology and capabilities. And, um, Um, as a project took learnings from the community as well.
It’s probably one of the best aspects of being a security provider is that there is a community out there that talk to each other and are happy to share learnings, which is great. Um, for you and I, it doesn’t help, Jono. Like, so for every security incident we’ve managed, we want to stand from rooftops and tell stories.
Clearly, we can’t do that on behalf of our customers because it’s a very private situation. But at that technical level, There is a fantastic community that share information and ideas and within our business that did help to spawn this idea of SOC Digital Twin where the desire was very simple. Like we have SOC analysts that we want them doing meaningful work.
They want to lean into the most interesting work too and they want to know their work has a material impact on customers and reducing risk. So, We put the technology to work and it was a learning model based on our own experience and incidents that we’ve worked on as well and we fed that through the machine.
We had to obviously obfuscate the right type of data to keep our customers safe to then deliver an outcome and now we’ve got mean time to respond down to under seven minutes, mean time to closure under three minutes and if those metrics mean anything to you, you’ll know that that’s world class. And the technology is really helping us do that.
And our SOC analysts are working on the things that we would want them to work on. And even better than that, there’s pride in the solution. There’s pride. So our security practice is hugely proud of what they’ve built. Um, they know that you and I are going to get out there and spruik it and talk about it like we are, which they love, which helps.
Uh, bring more customers to us as well, uh, which they love. So the whole thing is self fulfilling. Now we’ve heard a lot about, um, you know, how we’re, how we’re identifying these skills and, and sort of growing them probably within Macquarie. Um, and I know, I know that’s been a journey for us. Yes. And I know it’s resulted in already some interesting stuff on the roadmap.
And, you know, we’re, we’re obviously pretty focused on, on stuff that really makes a difference. So productivity is a big, uh, interest area for us. Uh, if you had any advice for, um, the people who are listening, whether they be, you know, other providers like us or potential customers, uh, firstly, You know, how do you, how do you spot those talented people?
Yes. And, and, and what advice would you give to them around how to nurture those people so you can start to get, see some homegrown skills development in this area and start to make a difference? Yeah, sure. So I think, look, fortunate that Macquarie was prepared to put make time and resources available for our staff.
It’s almost like a hackathon, wasn’t it? It was like a, a call to action to say, Hey, listen, we’re looking for the best AI ideas, implementations. We will make resources available. We will give you time to work on your solutions. And we’re seeking the best homegrown, organic AI idea within our business. Um, and that, With enough tech interested people in Macquarie, right, that sparked the interest of a surprising number of people, actually, that were already playing around.
Um, in fact, so many tech people and engineers in our business were already playing with ChatGPT, for example, and doing programmatic things. Um, Building little apps, things of that nature. There’s a bit of that going on already. And yet we, we made this available in a work context to say, Hey, listen, give us a productivity game.
Give us something that’s going to materially help our business. Gamify this thing, make prizes available, et cetera. We put people’s names up in lights, which we like to do and promote good work. And through that created great opportunities. So I’d encourage any organization to do the same thing. If you’re going to build something, Make that available to them.
Make sure they’ve got the resources and the time to do it too. I know we’re all very busy. Um, but certainly Macquarie did that and the benefits for us already with SOC Digital Twin and the Customer Insights Project, fantastic improvements and, and gains afforded to our business off the back of that. Um, and more than that, I think we, We showed our business a desire to double down and really invest in things too.
That was a good thing. It’s super positive. And it started with, Hey, listen, give us your AI ideas, AI ideas, sorry. Um, and let us turn it into something material. So I’d encourage any organization to do that. And, and this SOC digital twin thing, just to, um, just to wrap up on that. Yeah. Uh, there’s obviously an offer around sharing and openness around what we’ve learned.
Tell us more about that. If our listeners want to learn more, what should they do? Yeah, look, so there’s a lot to it. Um, there is a whole architecture, um, that we can talk through. We can talk through our thinking on the modules and the information exchange and the various AI models that we work with to produce those results.
We can talk through the text outputs, uh, not just the metrics on MTTR and closure. Um, I think demonstrating does better than talking to it. I’m doing my best describing something that’s arguably more complicated for my feeble mind brain to explain, but we’d love to show it right. Like we, we, we will put anybody who’s interested in front of a screen to demonstrate how it works, what it does, the inputs, the outputs, the information flow, et cetera.
And it might, it might get you thinking, I wonder if I could apply this to a use case within my own business. Um, So yeah, I think an offer of us, or from us, sorry, to demonstrate and talk through the capability in depth is there for anybody who’s interested. I think that transparency is important. 100%. We started off the conversation about AI washing.
Yeah. That’s about the, this idea of people talking about stuff. Yeah. Um, but not actually doing it. No, that’s right. Um, and for me, uh, if I was going to give any advice to, uh, you know, business leaders who are trying to navigate this conversation, you know, everyone’s looking for a piece of this action and, uh, you’ve only got, you know, so much money to invest.
Of course. And you need it to work. Yes. Um, I think I would encourage them to just be looking for a provider that is very transparent Very willing to engage and demonstrate and show the actual outputs. That’s right. You know, and be willing to go on that journey with you and show you the proof points. I think, I think you need to go digging for the proof points.
You can’t rely on, you know, the other five customers that they have or five years of credibility. It is brand new. There is gold out there. You’ve got to go digging. Yes. Um, you gotta, you gotta put in that due diligence. And look, I think you and I, we’re, we’re, we’re lucky in that. Our business doesn’t seek to monetize this thing.
Like this is just literally an improvement of service for us, which means the actual product itself, the capabilities, um, it’s open. It’s transparent. If it can help your business, we’re not precious about it. We would make that intellectual property available to any customer of ours to solve any problem.
Yeah, exciting space for sure. All right, so look, to close, Jono, we’ve spoken about a lot, right? There’s every SaaS provider under the sun promising an AI outcome. Um, we’ve offered some real world insights into what it means for us as an MSP and some of the problems we’ve solved. I think our own personal journey has only really just begun.
I know there’s going to be many more use cases and scenarios that we’re willing to talk through. Um, I would say, look, just in closing, um, Reach out to us. Um, everything I’ve spoken about, we prepared to be open and transparent with. We’ll talk you through our journey, what we’ve done, how we’ve built it, um, where we think it’s going as well.
Again, this whole thing is very fluid and models are changing all the time, so I would encourage anyone listening, if you’re interested and you like the sound of it, reach out and we will make it open and transparent and we’ll talk through exactly how we’ve done it. I think that’s a pretty hot offer, Naren.
Pretty hot offer for sure. And look, um, yeah, very interesting conversation for me. You know, I’m actually excited about where the industry’s going, the prospect of all the new jobs and, you know, where this could land and what it means for, uh, the next generation of talented people entering into our industry.
And I would encourage, uh, every organization, you know, looking, To make investments in people, to be just putting the effort in to identify those talented people and nurture them, put frameworks in place, give them the space to explore this. It’s uncharted territory, you know, and it’s quite exciting and there’s real gains there to be had.
