How one European bank embraces open source

I sat down to talk about open source and development with Jindrich Kubat, the Head of Development and the Chief of Engineers (COE) at Komerční banka (KB) in the Czech Republic.

Ben Rometsch: What is your official role at KB?

Jindrich Kubat: My official role is Head of Development, COE (center of expertise) at Komerční banka.

Ben: What exactly is your area of responsibility and day-to-day?

Jindrich: As the Development COE lead, I consider the expertise, structure and capabilities needed to make software development successful across hundreds of developers. We have roughly 600 people in the broader development team, but I'm not directly responsible for managing them. I'm responsible for defining how they will do their job. So for development best practices, standards unification and so on.

Ben: What’s your main focus today at KB?

Jindrich: Our transformation project. We have the current bank, which is built on typical legacy banking systems that are huge, distributed monolithic architecture systems. That is where most of our developers (around 400) are working day-to-day as it generates the bulk of the revenue for the bank.

Alongside the legacy applications, we decided to completely modernize to something that we call "Digital Hub" which is replacing the bank's current infrastructure. The front-end applications are native mobile and web applications.

The Digital Hub and new front-end applications are built entirely on microservices so we have to redesign a lot of things and introduce new guidelines for people. All of this together is what we call the "New Digital Bank". It's a massive project!

Ben: Does this transformation stop at some point?

Jindrich: We think about the transformation in two main parts:

1. "The New Digital Bank" initiative is planned for 5 years and we are about 2.5 years into it already. It started small with a few teams that were ready to adopt the most modern development approaches and it’s growing every year. Today we have about 180 engineers split among 50 teams. Those teams have been hired externally or transferred over from the legacy systems. They are dedicated 100% to “The New Bank” effort.
2. The other side is culture transformation. We call this “Agile 2.0” which hits the whole team (both legacy systems and new systems teams). Adopting Agile and DevOps is all about adopting a new way of working. This shift is something that we are driving with our legacy system teams as well. While the building will come to an end at some point, the hope is that the new culture will take hold and be the new way of working at KB.

[ Read next: Agile adoption: 6 strategic steps ]

Ben: Beyond agile, have you adopted any new approaches?

Jindrich: Yes, there are three new approaches that we adopted:

1. Moving to OKRs which aligns well to Agile.
2. We adopted a collaborative chat application for all meetings, which has helped with remote work.
3. We have functional "tribes" containing people and skills that each tribe needs to develop and run its own projects or its own applications. This aligns well to the microservice architecture we're moving towards.

Ben: What about tooling, languages and infrastructure? How do you manage that across so many people and teams?

Jindrich: As the COE, we are mostly responsible for the unification of how people work. We are pretty rigorous in this. We developed our own KB framework, which we call Speed which is divided into three parts:

1. First is a Java SDK, built on top of Spring Boot, which is a framework for building cloud applications.
2. Then CI and CD Pipelines are used to automate deployment. This is built on Jenkins and Argo CD.
3. Kubernetes is used where we are deploying and running our applications.

We built our own cloud infrastructure, so everything needs to be on-premise today. If a team needs something, they need to ask for it and it is reviewed by my team. If we have something that solves their problem, then we ask them to use that. If not, I’m open to adopting a new technology.

We started with the Speedy Framework two or three years ago and it is still evolving. The first version was built for monolithic applications and monolithic works. We are on version 4 now. The microservice approach was implemented in version 3.

The disadvantage is that we have to follow some product updates and life cycles which is really painful for us. Kubernetes, for example, has three major versions per year.

Going forward we want to make it more independent of these cycles and automate upgrades in the future. Interestingly, our parent company adopted OpenShift, which we are keeping an eye on. We’ll see if we unify the frameworks or not in the future.

Ben: It sounds like you are using a lot of open source technologies in the new platform. Have you always done that?

Jindrich Kubat: No, we used a lot of the typical products like Oracle and Sun Microsystems. For the new digital bank we decided to adopt more open source technologies like Kafka and Flagsmith. This is great because they are free, but we are responsible for keeping it up and running. This is a completely new mindset for teams.

Ben: Did you have to fight to use those products internally?

Jindrich: No, they sort of just accepted that because they know the reasons. If we show them the contract with Oracle, how expensive it is, there’s a big business case to adopt open source. Even if you have to have a few more people to manage the infrastructure, it has a positive impact on our costs.

Ben: What about feature flags? Was that something that you knew you needed before you started?

Jindrich: That's a good question because, you know, even in current banks (legacy systems), people are using feature flags, but they’ve developed their own systems. They’re very simple systems that manage the configuration file. And it was just because the legacy system was only deployed 3 times per year. So you can imagine how difficult it was to keep all of the promises from other teams. So they were really just there to keep the enterprise release management and testing on time and coordinated. It was easy to switch on or switch off some features that weren't ready for deployment.

But for the New Digital Bank we have microservices and continuous integration. We deploy on a daily basis to non-production environments. Interestingly, we only have two non-production environments. One of them is only for testing during the build and the other is the final non-production environment (staging).

Ben: Was it a conscious decision to have fewer environments?

Jindrich: Yes, this is an approach I brought to the bank where we completely redesigned some environments, and how we deploy them. The goal is to get to production as fast as possible to keep production and non-production as close as possible.

Ben: The "holy grail" is to have just one environment— production. Surely that's not possible in your business?

Jindrich: I have some experience with that, where you just test everything in production. But in banking, that's impossible. A lot of our developers can't even access the production environment. There are regulations. For example, one regulation says that you can't have a testing account in production because you are operating with real money. Everything must be "real" in production.

Ben: What do you do to mitigate that?

Jindrich: You have the smallest number of non-production environments as possible.

Ben: When you were selecting a feature flagging system, did you consider SaaS services or was that something that you immediately ruled out?

Jindrich: Yeah, we decided to evaluate three systems. Flagsmith, LaunchDarkly, and building it ourselves. We did three “proof of concepts”. First was with LaunchDarkly, then Flagsmith and then we looked at our own homebuilt system. We decided on Flagsmith not just because of the system’s flexibility, but also the great support. The fact that you guys are open source and the great documentation also helped in our decision.

Ben: How are things going today with feature flags?

Jindrich: We have been using feature flags for one year in our non-production environments which has been working great. As for the production environment, it took a while from a security perspective to implement feature flags in our production environment. The reason it took so long is because the person originally responsible for bringing feature flags to KB left. Then the security team changed their criteria for evaluation. Finally on the third time, we got it into production after passing all of the penetration testing and requirements!

Ben: Making changes at a bank is not for the faint of heart!

Jindrich: Yes, because of the security standards the entire banking industry is really sensitive to this. Now we are up and running, focusing on the permission models, and logging everything so it is auditable.

Ben: Has it changed your workflow?

Jindrich: Yes! We decided to start with the frontend, especially mobile teams. People have been waiting for this capability because they’ve used feature flags in previous roles. We have people on the team pushing to adopt because without a feature flag system, everything has to wait until it is fully finished. Now they can deploy to production and more teams are onboarding to The New Bank. For those teams in a microservice set-up, it is crucial for teams to be more independent.

So far, we have about 85 engineers working with Flagsmith in production. That is almost half of The New Digital Bank already!

Ben: Are you using flags on server-side as well or just the frontend?

Jindrich: So far just on the frontend, but we are getting ready to roll that out with the backend systems that leverage microservices. That will be very soon!

Ben: Are there any ways you want to use Flagsmith in the future?

Jindrich: Yes. We're looking at integrating the Flagsmith API into our bug tracker, being able to drive things upstream, and be able to keep the QA teams aligned with the product owners.