Diederik van Liere: Simplifying Tech - Lessons on Scaling, AI, and Data Anti-Patterns

This transcription is AI-generated and may contain errors or inaccuracies.

Matt

My name is Matt and I will be talking with Diederik van Liere about simplifying system architecture, data modeling and leadership challenges. Hey, Diederik, good morning. I'm really happy to have you here. The person with a lot of years of experience today. I really like your experience with the Shopify and now you run. You are the CTO for the Wealthsimple in Canada, which is pretty successful investment vehicle, I would say. But you have only a mobile phone and I don't like the intros.

I already said it to you. So if you don't mind, I have the first question and you caught my attention when we recently talked about the questions and the topics we discussed and we were talking about the architecture and simple simplification of it. So the microservices, I feel that we have having a bit of different trend now and you have a bit of experience with that. So maybe you could tell about your experience and your journey here with the Microservices monolith and how do you see it at the web? Simple.

Diederik van Liere

Yeah, for sure. And thanks, Matt, for having me on your podcast. It's really, really awesome.

I think this is. We, as technologists, we sort of always are caught in our own hype cycles. And I think microservice architecture went through a similar hype cycle where this was like the obvious thing to do. Like, if you were growing fast, then you should not have a monolithic code base, but you should split it up in microservices so you could go faster. I think that's sort of like the key, the core promise. But it sounds very easy to do, but it's actually very, very hard. And because the hard thing is that when you're growing really fast, it actually means that you don't quite yet know where your business is going.

You don't have yet really good understanding of the products that you're building, you're going to build. And so I think it's really. You really run the risk of like premature optimization where you split it out too soon and they actually get it wrong. And I think that's happening in the industry and I think we got it a little bit wrong at times. Right. And so my other two companies I've worked at, both Wikipedia and Shopify, we did have monolithic code bases. And you can make it work.

It has its own challenges, but there's no fundamental inherent reason that those things will slow you down. Would have to make more investments in tooling and developer experience. But you can make it very effective way for developing if you don't really understand the way your business is and how it's going to evolve. You're going to most likely cut it up the wrong way. And so what you then often see happening is like a distributed monolith. Right. Like the idea is that each service is independent of each other.

But in practice, um, if you get it wrong, they become highly coupled. And if you live in that world, it actually means you're going to struggle with shipping, with confidence, troubleshooting, understanding the behavior. Um, and so we went very quick from a monolithic code base to about 150 microservices and actually didn't go allow us to go faster. It actually slowed us down. So for the last two years since I became cto, I've been really beating the drum on simplifying and I strongly believe that fewer components, fewer things make things simpler. So we have been consolidating and merging code bases into fewer services. We're not going to go back to a single monolithic code base.

That's not the intention, but I think an order of magnitude fewer. So from 150 to about 15, it feels about the right number, approximately. It's not an odd number but. And so we've been unshipping services, merging them, consolidating them, and within the first 12 to 15 months we went from 150 to 120. And I don't think people believed it was actually possible. Right. So we had to come up with, with rituals and ways in the culture to actually celebrate and encourage this kind of behavior.

So one thing we did, we created an unshippered Slack channel where we celebrate people deleting stuff, small things, big things. And it's a very fun channel. And basically what you do, you take a screenshot of your GitHub PR. The screenshot focuses on the number of lines added and lines removed. If the lines removed is bigger than the lines, edit, it's, it's great. You add a single or two sentence description of what it is and then we just celebrate. So it's an extra, extraordinarily feel good channel.

Pick and ship it are welcome, small ones are welcome. And I think that's sort of like just celebrating and recognizing this kind of work built momentum. And then after the momentum happened, people start seeing that it became easier in those areas to reason, to ship with confidence. Right. So teams actually start really liking the benefits that they actually were experiencing from these more domain aligned services. So that's where we are on the journey. We're not done, but we've made some really Good head start here.

Matt

So today what is the number of the, of the microservices? Do you remember? Maybe?

Diederik van Liere

Yeah, I think it's about 120 that are related to the, to the different products. It's probably a little bit more if you also include internal services ones, but I exclude those are fine. So for the products, different products, it's about 120 right now.

Matt

Yeah, but definitely less so. I really like the idea because encouraging people like to simplify and to make the things that from the outside maybe looks impossible at the very beginning. Right. It's a really cool idea. With the like simple Slack channel. This would be even a thing.

Diederik van Liere

You need mechanism, good intentions, as Jeff Bezos said, don't get you there. You need a mechanism. And our mechanism was creating an unshipping channel. We also created emoji. It's the inverse rocket.

Matt

Cool stuff. And Diedrich, you work a lot with the data like so you have like a strong background here. And we were recently talking about the anti patterns in the data products. So what, what like are the common anti patterns? Do you see like in the data products?

How to avoid them? What would would you tell the other leaders, like where to put the emphasis?

Diederik van Liere

That's a great question. I would take a step back. I think they exist at different kinds of levels. But probably, I'm sorry, if I would start at like an architectural level. We have really lost, I think the art of data modeling. This was a very popular thing to do in the 80s and 90s and this also really dates me. But the art of data modeling is really about.

You have take the raw data out of the, out of your operational databases and transform and clean and model that in a way that actually maps to your business. And so because of a whole bunch of technological innovations, we stop doing that. Right. Like we start basically modeling at query time. Right. Like the transformation happens at query time. But there's no longer a source of truth around your metrics or very, very rare.

Very few companies have that. And so what happens then is that people, we democratize access to data. Everybody can query basically everything in principle, but. But it is really people's own understanding and which is probably too narrow of the data. So they can come up with metrics that are probably incomplete or even misleading because we don't put in the effort of building and maintaining a metrics layer modeling the data.

Like we don't. There's few teams or companies that, that still use methodologies like from, from Kimball or, or Inman. Right. From the 80s and 90s and so I think that is a massive, massive anti pattern and it's only going to get worse because now of course we want to use the same data for all our generative AI applications and everybody knows garbage in is garbage out and like but somehow we are not able or willing or I don't know what it quite is to put in the effort. It is not super sexy work, it is not visible work. Right. It's also very hard.

But it is so incredibly important is that you make your investments in modeling data. Clean, accessible data has never hurt you. The inverse has hurt you many, many times over. So that I think is probably the biggest anti pattern.

Matt

And how about the organization structure for collaboration? I mean do you have your own approach like how to organize the team to support, you know, better collaboration on decisions on the architecture, on the data?

Diederik van Liere

Like yeah. So the way we run our data science teams, they are basically in verticals which map the organization. So we have a marketing data science team, a finance data science team, an operations data science team and so they mirror or they are one to one paired with their functional counterpart. And these are long lived teams. And the reason for that is few, few actually like one. It takes time to become a subject matter expert. But Im so if you want to do proper and good data modeling you really need to understand that vertical very very well.

So people are basically trained in this particular vertical like as a marketing data scientist it takes a while to become really, really good in attribution and churn and LTV and cac. Right. It doesn't happen in one day, it takes a long time. So that's one, two. We also really believe that you need to have trusting relationships between the data scientists and the business counterparts. There's another very important reason why these are long live teams. So the supplemented expertise in building long lasting relationships and so they have a really good understanding of the problems they are business trying to solve the questions they have.

And so that's how we have done it. So I think other models are more like a shared service center where you just sort of like first in, first out but then you don't get the benefits of specialization, you don't get the benefits of building long distinguishing relationships. So this has been a model that I think is working quite well and it also gives you some sort of predictability. People don't have to worry that they have to compete with other teams. So the marketing group doesn't have to compete for data science resources with the finance group because you get your own team. And so so then if there's a prioritization challenge. It stays within your own vertical. Right. Instead of like you have these cross interdependencies like oh my God, now I need to stop arguing with another group.

It doesn't exist. So vertical decoupled pillars that are long lived has been a, I think a pretty effective strategy.

Matt

And recently we talk about the single vendor or single tool dependency. Right. You were talking about why do you avoid committing to one tool or one vendor and how do you approach it? Maybe you can elaborate on that.

Diederik van Liere

Yeah, I think that requires. It really depends on like the kind of how deep in your, in your stack. Right? Because there are, there are, there are benefits to like having a single vendor. Like for example at your, at the lowest level, I think at your, at your, at your cloud provider I think it will be insanely complex to run multi cloud, right. Like, and I actually think it's often because of misunderstood concerns around resiliency and reliability. So I think you can solve that in a single cloud provider.

But if you for example go more up in like in your stack and you have like other providers, for example like we rely on different order execution brokers, then of course you want to have some kind of redundancy that in case one goes down you still are able to function and can and can redirect your orders to another execution service. So I think you have to sort of really think about where in a stack how deep are you and do you need, how are you going to organize for resilience and redundancy. So sometimes that can be done within a single vendor. Sometimes you will have to have multiple vendors. It's also always good to have multiple vendors. I mean you from a price leverage point of view, understanding the roadmap like but they also come cognitive cost associated with. So I think it's really, it's a, ultimately it's a, it's a risk appetite and ROI decision that you have to make.

Matt

And now is my favorite topic. I'm always asking my guests and everybody likes to talk about AI.

Diederik van Liere

But I.

Matt

Wanted to talk to you in a more pragmatic kind of way. So what do you do on, on what do you do? What kind of initiatives do you have and how do you measure the impact? Do you measure the roi? What is your approach?

Diederik van Liere

I think we, we were from day one at ChatGPT and I came onto the market like very excited about this as a, as a new revolutionary technology like I think is going to be as big as the introduction of mobile, introduction of the GUI back in the 60s like, I think it's a truly transformational technology and we don't quite yet know how it's going to play out, but I think it can fundamentally redefine how we interact with machines. That's my belief, but it's also very much day one, day two. We're not nowhere near that yet, but I think you can sort of start seeing the outlines. So given that we, I think we realized from the get go the potential of this technology, how then do you create an environment in which people can experiment and learn with some guardrails? And so the immediate guardrails that we thought of like we want people to use ChatGPT and other models, but we also want to make sure that they don't accidentally share either IP with those third party providers or PI. So we created a, what we call LM Gateway. It's basically a proxy in between OpenAI and other ones.

And it inspects prompts and makes sure that nothing that shouldn't be shared is shared. That was very successful. We open sourced it. It's available on GitHub under the Bellsimple account. It's quite some forks and some contributors, which is really amazing because it's a general problem, not the only ones that face this challenge. But then as we did that, we also started to learn quite quickly that actually the most powerful use cases are actually the ones that leverage our own proprietary data. And so we were sort of like preventing that.

So then the second step in our evolution was okay, we have the gateway, but for these very powerful use cases we are going to be running open source models on our own infrastructure. So we don't need to worry about people sending any data that it shouldn't because everything stays within our own vpc.

It's walled off. So that was step two, so people can internally use Llama Mistral. It's hard to keep up because the models keep coming and coming. Right? So that was the phase two and phase three was then, okay, we have made the investments in the foundations, in the tooling. Now we have to start looking at really exciting use cases. So the first thing we did was let's give all our engineers access to GitHub Copilot because that seems like a pretty easy way to get people comfortable and experiment with it.

And right now the final step, the fourth step is actually like we are in a very, in very specific areas introducing gen in workflows. And so it's starting to like being used, being leveraged against our own infrastructure using our own guardrails. So I think we have very good handle of that. One way I think we are managing the risk is by like if you use it in your back end, back office processes, then you don't need to provide an open prompt and so you sort of really can significantly reduce the risk because open prompts are I think the area where we are seeing like the most challenges. Prompt injection attacks, right. Potential of data leakage, cost contamination, all those challenges exist in that world. If you actually keep it in your back office, you are actually able to see like efficiency, efficiency gains without sort of like the risks.

And I'm a kind of guy that loves to eat his cake in F2. And so you were asking also about like what are the ROI? So actually the GitHub Copilot API comes with a lot of instrumentation and so we have a pretty good understanding of how it's being used and how effective it is. So I think the last time I checked this data was only a few weeks ago and about 30% of the suggestions offered by GitHub Copilot were accepted by our offers. So that's best offers engineers.

That's good, right? That's a lot. And so if you then look at the impacts on, on PR cycle time, for example, there's a market reduction, I think it was about 20% faster. So, so that, so that's so very good instrumentation where I actually do know the roi. I mean I, I could, I mean don't ask me for a dollar value right now because I don't have that. But, but I can definitely see it's, it's helping, it's making us more efficient on the back office side. We are still much in the build phase right now, so I don't yet have ROI figures for, for that, but I'm pretty convinced it's going to be meaningful. Nice.

Matt

But like if, if I'm, I'm hearing like 30%, 20%, this is definitely something. And you see the impact like long term, it's like a huge benefit. So it's not like a 5% or 2%.

Diederik van Liere

Right.

Matt

It's like a huge benefit.

Diederik van Liere

Huge benefit.

Matt

And because you have a lot of years of experience, for me it's always interesting to see like what are your aha moments. Like what was the hardest things, the hardest thing that you have done in your career? Because I think this has, has like a huge impact and shape you as a leader at work.

Diederik van Liere

Everybody's journey is very idiosyncratic, is very unique to them and so I don't know if my journey is a recipe at all. But maybe there's one or two principles that might be more broadly applicable. But it's hard for me to assess that. But what I can offer is that particularly when you're a bit younger and you have fewer demands and responsibilities, it's really important. Important to, to say yes to things that, that, that are very scary. And, and I did it multiple times and right like the very first time, a bit of big one was my decision to stay in Canada. I originally was going to be here for two years and go back, back to Holland.

And I decided to stay here even though there was very little in front of me to make me stay. Right. So that was a big, big, big decision. So I think that's an important one. Like say, say yes. Raise your hand for opportunities that seem a little bit crazy. It's very important.

It teaches so much. It gives you grit and resilience. Might not always work out necessarily in how you expect to work out. That's not a bad thing either. And so you're, you're creating your own life journey by these kind of decisions. So I think that's, that's very important. And be easy on yourself as well.

Like when like sometimes there's going to be setbacks, sometimes things don't, don't work out and that's also okay. It's really part of the, of.

Of your journey. And sometimes we can be really harsh on ourselves. Most of often we are the most harsh on ourselves.

The biggest critics. So reflect and learn. Don't beat yourself unnecessary over the head. I would say that probably my second piece of advice.

Matt

Great thing. I think I had the similar ones like to say yes when you are younger and especially when you don't feel. And you don't see the boundaries because I feel the older you get, the more you see the kind of obstacles. And when you are young you say like hey, why not? Right.

Diederik van Liere

And it's something I don't realize getting yourself into which is really an amazing feature. Right. Like don't fully understand what could go wrong. And listen, there's naivety often that you should use to your advantage. For sure, for sure.

Matt

And how about now? Because like I feel like each year at the. There's something new for us. You learn something new. And what are your pain points? What are your challenges? You know what, what do you type in Google or Chat GPT to help you out?

You know, if something.

Diederik van Liere

I think this, I think we. You can always be more decisive. I think, I think at least personally like you're, you're always trying to make the best decision possible. And you're always think you need more data, you need more time but on the biggest ones you actually biggest decision you actually know what you need to do. It is about having faith and confidence if you're taking the right thing and it's better to be decisive and, and if, and even if you get it wrong which or it doesn't work out as you in hopes it would be a fast for fast course course correction learned by like it's not a mistake is not most mistakes are revertable in a way. Right. So I think that that's for me always I keep learning and relearning that lesson to be, to be more decisive.

I think it's probably for all of us. There's some truth in that.

Matt

Now how about the current role as a cto? What is the, the most stressful thing about it and how do you manage it? How do you deal with that?

Diederik van Liere

Making sure you keep the ball, the eyes on the right ball. There's so many balls in the league and making sure you check the right ones. And that that is a. In a fast growing company like Gumball that is very hard because there's so many balls coming at you and, and so you're sometimes overloaded the information overloaded with information from Slack. People want you. People need you for very good reasons almost always. But in that sort of deluge information it's easy to actually get sidetracked of the really important things.

So I think in any company that's growing very very fast there's just so many things on the go, so many opportunities, so many distractions, so many small fires, bigger problems. Making sure you yeah you keep the eye on the right balls at all the times is probably the biggest challenge.

Matt

And but how about the managing the calendar? Because I'm like the longer I'm managing the company and I'm running the company as a co founder it's like managing the calendar to find a time slot for three people and even more it's impossible. I mean like the people have, they have like packed calendar and I assume in your case it's similar. Do you have your like kind of approach here?

Diederik van Liere

Well I'm, I'm, I'm fortunate to be helped with an executive assistant on this one because. Exactly. It is, it is. Otherwise I would be just spending half my days calendaring which is not at least to happen but I'm in a very fortunate position to have a lot of support on that. And it's also a way to make sure I Keep my eye on the right walls. But that's, that's not of course available or applicable to most of us and so but I think it's really also here, I think AI is some of the really promising like where you can sort of start delegating to an AI agent that does the calendar scheduling for you. There's some really interesting tools out there that do this for you or quite well.

So then you have your, your, your, your AI ea. It's quite interesting.

Matt

This would be a next step. I think so. Yeah, I agree.

Diederik van Liere

Right.

Matt

And how about the, the, the recession? I mean the last year in Attack it was like a huge recession. This year it's like bumpy road. I would say The S&P 500 was getting up. I feel we, I feel that we are on the right course and I see that 2025 should be more optimistic for, especially for the tech and startups rising the capital. How do you see it? Do you, have you failed it?

Did it change like something in the way you operate and how do you want to operate in the future?

Diederik van Liere

Yeah, I think 24 is really a transition year. I think it started really slow early this year. We were not really growing but that sort of started to change a little bit mid year and we started to hire again a little bit. But also very much not forgetting the hard learned lessons from the 2020. 2021. Right. Like when, when money was very cheap and we all went a little bit crazy in hiring.

So I think we have learned our lessons. It needs to be very good business reason business case you be mindful. It's also very important to pay attention to onboarding. I mean people part of your culture and getting them up to speed like those kind of things. We also sometimes forgot a little bit to do back in those days. So I think we are in general we are a lot more mindful and deliberate in our hiring. But there is definitely some growth happening and I, and I do expect that to carry on in next year.

At the same time also it seems though that the world is getting increasingly volatile and unpredictable and so I don't think we are over committing. We are just really like keeping a close eye and sort of like decide on a, on a monthly cadence. What is responsible, what do we need? So some growth but with eyes wide open.

Matt

We are running out of time. And my last question is that I asked all of my guests, you know, any books or conferences, podcast resources that have been, you know, particularly influential for, for, for yourself and you remember this aha moment. I don't know after reading it or listening to it.

Diederik van Liere

Oh yeah. Well, I mean unfortunately I don't have so much time to read books anymore these days but I think some books that that, that really stuck with me but this was when I was a teenager I I read quite ferociously but out of Control by Kevin Kelly was an incredible book about how bio how how you how much we can learn from from biological systems in how we engineer our and the Art of Zen and the Art of Motor Cycle Maintenance was an incredible book that I I I I about the the value and the practice of being mindful so those are those are two books that have been a long time ago but they they are very mean a lot to me.

Matt

Awesome. Dietrich, thank you so much for today for all those insights. I really appreciate it. Wish you luck in the future.

Diederik van Liere

Thank you. Thank you Matt for having me and looking forward to the outcome of this podcast. Follow Matt and Leshek on link.

‍