Luke Rotta: Optimizing Non-Production Environments for Better Developer Productivity

My name is Matt, and I will be talking with Luke Rota about challenges in the Fintech industry and the efficiency and safety of software releases. I'm looking forward to our conversation today.

A lot of interesting topics to cover. But a person with whom I talk works in a trading company from a tech perspective. So that's an interesting thing. You work for a Chicago Trading Company and CME Group. I'm just wondering from the outside, because this is a really specific industry, what are the challenges of this business from a tech perspective that we don't see from the outside? Absolutely. Financial services, Fintech, it's definitely an interesting space.

It's a highly regulated space. So depending on what your firm provides, if it's a bank or a financial exchange or a trading firm that's participating in the marketplace, there's different regulations. So the SEC, CFTC, Securities Exchange Commission, Federal Trade Commission, they have a lot of regulations. And so those can be hard to navigate at times. They can be roadblocks to putting kind of more modern development and operational practices in place, DevOps for example. So DevOps is meant to bring ops and dev closer together. A lot of regulators say they'd like to see that separated.

They want the developer in a specific role and they don't want them to be able to touch production. So those things can be hard to navigate. I've been able to overcome some of those challenges in my career. So that's something from the experience that can be a little surprising when you kind of walk in the door. Do you have any specific, I don't know, routines, processes that you have implemented for your dev team to handle this kind of challenges that you have mentioned? Yeah, absolutely. So while the regulators like to see things separated, that can kind of force human driven processes and humans make mistakes.

So one of the examples of what I've been able to do is to put some automation in place that does a lot of checks that an operator would do when they're doing a release or a deployment. So a lot of times the operator doesn't always have all of the knowledge of what has gone into that software, what were the changes. And so a lot of times they could be doing things kind of blindly, which could be actually more dangerous. So being able to tell that story is important as well as building some automation to take the humans out of the process and put some more rigor and actually reduce the risk of the software release. So that also has empowered the developers to take on the releases themselves. So you can do things like put specific time constraints in place or a specific version. Maybe some versions are higher risk and so you can put guardrails around that in your automation.

But ultimately this has allowed developers to get closer to production. They have more skin in the game and therefore the quality of software actually becomes better. So it actually creates a really nice situation and a nice story. And you can document and show the automation and show the guardrails and that usually is satisfactory to any kind of regulations or audits that you're subjected to. Recently when we talked, we discussed the mission that you had to transform the department. Because you went into the department and you had to transform some best and implement some practices to improve the processes. And I'm just wondering, could you elaborate on that experience and what worked for you?

What are the lessons there for you on that? Yeah, absolutely. So yeah, I just actually completed the transformation in February of this year. So it was like maybe 16 months or so, less than a year and a half transformation, 14, 16 months, somewhere in there. But ultimately moved the team from a traditional application support, IT operations or technical operations team where they're doing a lot of support and releases and moving that into a development function and turning it into a site reliability engineering team. So definitely, definitely lessons learned there. Some of the things that I did to make it successful, because I think at least the feedback that I got, it was pretty successful from a leadership perspective, as well as those that were involved.

So really establishing what is the new mission of the team, evaluating skill sets, what is the current skill set of the team and providing a training plan to fill the gaps so that people had the opportunity to build the skill set. Hiring from the outside with people that have experience in site reliability engineering, that was a key success point, right? Helping bring in that knowledge and that culture, that discipline really, really helped as well. And then making sure that your partners within the firm are on board, that they understand your direction that you're going. They're supportive of it. I did a lot of roadshows talking about what is the current state, what is going to be the future state. Certainly keeping the team updated as to the progress, getting their feedback, trying to iterate on their feedback as well.

And then some responsibilities needed to be moved to other teams. And in some cases, we just cleaned up responsibilities. They weren't really needed servicing any value anymore. So the lessons learned really for me are just get as much feedback as you can. You can never get enough feedback when you're doing a transformation and get it from all levels, right? People that are hands-on, junior engineers, all the way up to principal engineers, staff engineers. And then certainly your leadership, your peers, you know, the heads of areas, right?

So really getting their feedback and keeping them in the loop. And then also trying to not push too hard, right? Too fast. I think as I got towards the end, I started to maybe kind of rush it a little bit. And so I had to just slow down, do a little bit extra communication to leadership and to the teams that were involved. And just do some additional checkpoints with them, which I think was really beneficial to make sure that it was truly a smooth process and that people had a good experience through it. And I'm wondering if you would get the same kind of mission, right?

To transform a department in the future, what would you do differently? Is there anything that you would change in your approach? Yes, I think I would hire people sooner. I think I waited a little bit too long in the process to hire someone. I ended up hiring, it was at least two people to start and then kind of built from there. But yeah, so I would hire sooner. I would do, I did some of this, but I don't think enough getting out just people outside of the firm, right?

People in my network, talk to them, understand, you know, if they've gone through a transformation, what were their challenges? So I could have done a little bit more of that. I think overall it went pretty well. And I think the last thing that I would do is maybe just concentrate a little bit more on the final details at the end. I mean, it's hard to catch and predict everything that people may think about or run into. I think it was pretty thorough there. But I think what I could have done is kind of once the, kind of in the final weeks or month of the transformation, especially at responsibilities around, just check in, you know, after the first week, hey, how's everything going?

Are there any hurdles? What can I do to help? So those are the items that I think I could do differently to make it, you know, like the perfect transformation, if there is such a thing. I don't think there is no transformation. So yeah. And you mentioned a lot of about communication, communication, communication. So it's related to people.

And usually this is the hardest thing in the role as a leader, to like figure it out and realize the task together with the whole team. And I'm just wondering, do you have some, your own strategies that work for you, like in regards, how do you lead the people? How do you deal with the communication? Maybe some processes you have to follow. Yeah. So I did a few things. I'd, one, I, so I concentrated on the team, explaining to them, you know, everyone kind of understood the current state, right?

I think at some time, at some points it was kind of that clear to people on, you know, what is the future state. So making sure to have, you know, whether it's through your weekly team meetings or kind of how often you do it. I set up some ad hoc things as well, but for the most part, just trying to communicate on a regular basis and get the team's feedback as well as meet with people setting, right. And so also taking that into consideration helps. So I met, you know, many times with the team, the entire team, and then met with individuals as well as solicited, you know, as I brought new people in, solicited their feedback, you know, how, how have you seen this work? Where do you see the challenges thus far? And then, so, so that was kind of the team aspect.

And then I also focused on talking to others outside of the team, right? Getting their perspective, okay, what, what do you think the current team provides and what do they do? And if that team wasn't doing that anymore, would, you know, how would you feel about that? Would you miss that? What, where do you think that function would move? How else could we solve it, right? So, so that was really insightful to get an idea because the team function had been around for a decade and a half, right?

So it was really ingrained into the firm and people really had expectations of it. And so some of it was shifting responsibilities onto other teams or onto the developers themselves, right? So I wanted to understand how they felt about that and how that was going to impact their work. So that was very helpful for me to, you know, really understand what I could do to smooth out the transformation, as well as it also identified some really cool pieces of automation that we built, which really smooth things. It actually decreased developers concerns, right? Because they didn't have to learn things that got extrapolated for them in automation. And so, and, and the team that was going through the transformation built that automation.

So they got to be empowered to be like, okay, yeah, we don't have to do this work anymore. That's great.

I can do new and different work. So that, that was, that was really powerful to be able to put kind of the team's destiny in their hands and be able to shape it themselves as well. Speaking of the automation, which is, I think, absolutely necessary in today's work and engineering work. But there is a topic of AI that I always really like to tackle. And I'm just wondering, do you see any impact of the AI of the work that you're doing? Maybe you're working on some concepts already? Yeah, definitely.

There, there's definitely a focus from my perspective in the firm overall on AI. We do have actually like an R&D team dedicated to, to looking at various AI solutions and how we can apply that to business problems or just to our own technology challenges or problems, right? So I think there's a couple areas where AI can really help. And I think we're, we're seeing that already. One is people have questions, a lot of questions, right? Especially new joiners.

How do I submit an expense? Or, you know, hey, I need a new laptop, or can I get a new laptop? And there's plenty of bots out there now that can, you know, read your knowledge articles, maybe even your code base, right? And they can help point the person in the right direction without a human having to answer that. And so that's been decently successful. It's still building. I think we're still in early days there, but it's, it's definitely helping.

It's also kind of a fun challenge to have like, this is the bot or the human smarter here, right? Like who can answer the question the best. So occasionally, you know, occasionally the human still wins, but so, so that's been, that's been really cool to see. I think that's definitely a way that can reduce kind of the, the administrative overhead or the servicing overhead of, of just, you know, employees in general. I think it's also helping increase people's knowledge set quicker and more efficiently. For example, if I'm a junior developer and I'm, I'm trying to figure out how to do something with a pandas data frame, you know, a lot of times I would turn to my senior engineer and maybe sit down with them and ask them questions. You know, I could throw that stuff into chat GPT or something like that and get a pretty good answer back. Right. It's not going to be a hundred percent.

You know, I think a lot of times it's about, you know, 20% of the time maybe chat GPT is wrong, but those things are kind of caught in a code review. Right. But I think in general, it's just helping build people's knowledge set and making them more self-sufficient rather than having to interrupt someone to ask them a question. You know, sometimes maybe people don't feel comfortable asking the question, you know, maybe they, you know, so, so I think this is really, really helping. Right. And then the real kind of face-to-face or virtual interaction is really kind of more of a deep dive, like, Hey, these are the things I've already tried and this is where I'm stuck. So, so I think it's been very powerful for that.

And then, you know, I think still remains to be seen exactly how we will apply it to like business problems. But I, I think certainly it, it has that capability of, you know, maybe even kind of being able to describe like a trading strategy that you want to try out or something like that. And it kind of, kind of give you the framework for that. So lots of application here. I think ultimately it's really about increasing the efficiency of your firm and allowing you to, to really get your features, your products to market quicker because you don't have all these other things slowing you down. You can get quick answers and you don't have to rely on another team or another human to sort of kind of help you, help you along. Cool stuff.

I think it's a really useful application and really simple. So it reminds me like six years ago, you would say like to somebody who is asking you the question that is pretty straightforward, like Google it. And you use the internal chatbot, right? Yeah, exactly. Yeah. Yeah. Yeah. Yeah. It's, it's, it's very cool.

It's very cool stuff. I think ultimately it's going to make engineers even better. And it's, it's less about maybe replacing, I mean, replacing jobs and humans and things, and really just allowing them to work at a higher level, right? Because they don't have to deal with some of the nitty-gritty that, that maybe isn't super beneficial to them. Yeah, I fully agree with that. And I'm just wondering, I'm always like to ask this question, like what is the hardest thing in your career that you have ever done and what are the lessons learned out of it? So the hardest thing that I've done is having to, I mean, we've talked about transformation that I did, right?

But prior to that, I actually had to, it was less about a transformation and more about a complete rebuild of the team, right? So sort of like a sports team where you kind of have to tear everything down and kind of rebuild it from scratch. Right. And so that was very challenging. The team at the time had a lot of tenure. There were definitely performance issues within the team. The team didn't get along well with other teams.

So there was a lot of challenges there to, to smooth all of that out. You know, ultimately had to part ways with some people, bring in others. Some people were actually able to kind of re-skill and, and improve. And then spent, you know, a lot of time working with other partner teams and rebuilding that relationship and trust. So, you know, that was at least a year and a half or so of, you know, hard work each day, you know, just trying to, to build relationships and improve things and rebuild the confidence in the team and, and try to build autonomy within the team and, and get the team to feel empowered. So that was, that was very, really challenging work. I think some of the lessons I learned or the takeaways are, you know, everyone should always know where they, where they stand, right?

You don't hold back or wait, right? They should always know where they stand. You know, if you're raising the bar, you know, they need to know where the current bar is and where the next bar is, is, is, is going to be right. So that's really important. And then really taking time to, to listen to the other teams that, you know, your team is struggling with. And it's important to kind of listen to understand and, and not necessarily, necessarily listen to respond. It's a slight subtlety, but it's really important to really try to be empathetic to understand what are the challenges they're, they're, they're having with your team.

And just, you know, work to, to try to improve that, you know, come up with an action plan to, to improve that, you know, each day or each opportunity that you can. And so those, those are the big items that, that I learned from that. Definitely, definitely challenging, but made it through. And, you know, in the end, the team was rebuilt and, you know, a much better performing team and interacting a lot better with the other teams at the firm. And we are now 2024. And what are your biggest challenges right now? What do you have on your plate? Yeah. So in my current role, I'm heading up our testing and integration environments.

And so the biggest challenge there is trying to satisfy everyone, you know, every developer's testing use case, right? Everyone wants production level capability, but it not to be production. Right. So there can kind of, there can only be one production that, that makes the money, right. Especially when you're, you're interacting with the financial markets. So, so the challenge there is, you know, what data do they need? How can we, how can we replicate that data?

How can we strip out the data that we don't need? How can we keep the environments reliable and not breaking, right? Because these are non-production environments, software is changing. Software can be, you know, by nature of it being how do we create automation to create, you know, allow some stability in the environment. So I've been trying to take a lot of my production experience over the years and applying that, that rigor and discipline to non-prod to create some stability so that these developers have an, we can meet their expectation of the environment being ready, you know, each day for them to use. And then really just all the dependencies and complexities that we have in our environment where we're kind of in between re-architecting large pieces of our environment. You know, things are straddled between on-prem and cloud.

And so just trying to find ways to automate our manual processes and reduce the complexities that we have within, or at least try to work within, you know, within the complexities that we have and reduce them where we can. So those are the big challenges right now. Ultimately, you know, to increase the developer productivity and satisfaction through, you know, more reliable and available and fully functioning, you know, non-production environments. And the last one, I believe the question that I wanted to ask you is, could you recommend any books, podcasts, or maybe resources that have been particularly helpful to you as a leader? Yeah. So I'm a big fan of networking, you know, getting to know others. It's, it can be uncomfortable at times, right?

To reach out to someone that you don't know and try to start establishing a relationship. So there's a book called The 20 Minute Networking Meeting. I've found that it's very helpful in, you know, helping establish a framework on how you approach networking. How do you go through a meeting? How do you start to build a relationship with someone efficiently and make good use of their time and your time as well? So that's something I've been recently focusing on and that book has really helped. I think also there is a book out there called Good Power by Ginny Rometty.

She was the ex-CEO of IBM for many years, right? So it's a book about leadership, you know, and how to build teams, treat teams, you know, empower them. So, you know, as well as identify your own blind spots, and how to fill those with people around you, right? Because as a leader, you don't have it all, right? As much as you want to think you do, right? Not everyone is perfectly rounded. So I found that helpful.

You know, the Google SRE book certainly was helpful, you know, as I was going through the SRE transformation. And then I think lastly, I spend a lot of time, maybe not time, but I spend a decent amount of time reviewing other tools, right? Just trying to keep an eye, you know, a beat on what is happening in the various spaces, observability, you know, CI, CD, developer tools, IT tools. And so that's been really helpful to understand what are the challenges that those tools are trying to solve. A lot of these kind of grow out of someone's experience that they had at another firm. So it's very interesting just to kind of see how people are experiencing the same type of challenges, and how they could be solved. So that's been, I think, very helpful to me as well. Great. Thank you very much, Luke, for all the answers and insightful thoughts here.

Appreciate our talk today. Yeah, absolutely. I really appreciate you carving out the time. I really enjoyed the conversation. Great. Thank you. Thank you.

Follow Matt on LinkedIn and subscribe to the BetterTech Leadership newsletter.

‍