Transcript
This transcript was autogenerated. To make changes, submit a PR.
Morning everyone. My name is Rupesh Kumar. I'm the
principal DevOps working with Andrew Foster who is the
director of DevOps Engineering at Humana. And we
are very excited here to bring forth our journey
of DevOps and how
we have done it. And we would like to share some insights which
you can take it back to your organization and hopefully you will
find some good value in it. So I have
been working in this field for over 20 years and I've been
working with both the federal sector and the commercial sector.
And as of recently we have been doing some
good, amazing work in DevOps outside the
industry. Now Humana has leading that effort and Andrew
Foster certainly has been leading this from past few years.
And I have joined his team and we are kind of making
DevOps at a scale. So that's where I come
in and I would hand it over to Andrew to kind of introduce and
take it away from there. Thank you, Rapesh. Great intro
sir. Welcome everyone to our session again.
My name is Andrew Foster, I'm the director of DevOps Engineering here at
Humana and my team is responsible for all
of the DevOps platforms, whether it's open systems, mainframe or salesforce.
For Humana, we care for the engineer experience as well as the engineers
tooling. So everything around engineering software. We are
the it team for the it teams in which we take great
passion in. I've been at Humana since 2011 holding
various technical roles and been doing the DevOps for the
past four years. And I love it every day.
Rapesh, could you hit the next slide, sir? Sure.
Thank you. Before we get into much about our journey,
as Rupesh mentioned earlier, we wanted to tell you a little
bit about Humana. So Humana is a Fortune 41
health and wellness company. We view ourselves as more than just
traditional insurance and we are what
we call a human care company. We take a unique approach to
healthcare that's completely centered on people and all the
things that they need to feel whole for just a little bit of context.
We're the fifth largest health insurer in the United, started among publicly
traveled insurers, and our primary business is serving
our 5.1 million individual and group medicare advantage
members. We also serve our active duty and veteran military
families, offer group insurance and other standalone wellness products.
All right Rapesh, hit the next slide, sir. And so as
you can tell at Humana, we really value
our members and everything that we do. All of the software that we deliver
is in support of our members to make their experience better while they're using
our products and using their healthcare benefits. So how does DevOps fit into
that? Well, our mission is to,
as you see on the slide, give the engineers the most compelling
development experience possible, and we take that seriously.
We aspire to get out of the engineers ways,
allow them to focus on software delivery only,
and give them the best experience possible while they're doing that.
Repesh, I'll hand it to you to get into how our Humana
values have inspired the DevOps transformation here at Humana.
Thank you, Andrew. So, as Andrew said, our DevOps journey started
at Humana in 2018 with the leadership team wanting
to actually drive change. Now, as you can see,
our core values define our culture. Now,
culture is what makes people understand each other better.
You may be wondering why culture has to do with DevOps.
Well, we say that culture is pretty much everything to do with DevOps,
because everything is driven by the people, for the people,
and of the people. So that's where DevOps actually fits in very
perfectly. Now, our strength is in developing
a uniqueness and to kind of always question
the status quo. And that's pretty much what we always do as
part of this team. We have a diverse workforce, and we
are always able to bring in their unique strengths and
even the weaknesses and kind of see how we can complement each other.
Now, with the different thinking styles of people we have in our workforce,
it always good to look at the problems from different perspective.
And that has always helped us to actually look
at the status quo and keep innovating, if you will.
Now, as Leonardo da Vinci once said, I remember that very
clearly in one of the book which I read. It said,
simplicity is the ultimate sophistication. So one of the things which we
do at Humana with respect to DevOps is to make things simple,
as Andy was saying earlier, because we do it for our developer community
and at the end of the day, for the customers. So we need to make
the process simple, which can be understood and kind of like easily
scalable as well, at the same time. Now, one of the other
things which we also do at Humana is focus on
health. And certainly, needless to say, because we are the healthcare
company, so health is one of the utmost things.
So we inspire that the person is always
emotionally, physically, and spiritually available, and they are inspired
to work in a community.
And in order to do that, we actually have lot of
different online communities, channels, talk shows,
where we talk about the overall health of the
individual, because a happy employee makes the happy customer.
And that's what at least I believe in. So that's where I would say our
company culture has trickled down into our DevOps,
and that's what we see every day, and we live and breathe that. So with
that said, I will actually take you on to the next screen where
we talk about how we have developed this Humana DevOps
model and which has been working for us pretty
efficiently. Now, in order
to make things happen incrementally and
iteratively, it's very important for us to kind
of like listen to the customers. So one of the things which we have
started doing is we are paying more attention to what the customer
needs and wants are by doing active listening. Most of the time
you may feel, hey, that something is a second nature. We always do
that. But you will be surprised how much we can learn when we are
actively listening to our customer. And even for things where they are not even
saying it, but they may actually be
talking around it. But then the good thing is to listen to the
customer, pay attention to what they are saying, and kind of
identify where their pain points are. And that
actually has helped us tremendously to build on some of the
value which we introduced, which created a big
win for us, or some may call
it as a low hanging fruit. So certainly active listening is
certainly the big one. And then the other one is
kind of making sure that you are repeating
yourself to the customer to understand that you have understood
it right, because that's something which I have seen many a times where we
miss on the expectations of what customer really is
saying versus what we are kind of understanding.
Next thing is empower. Now, how do we make
sure that the customer is providing
us the requirements and what he wants to get done and who is
making it happen? So we certainly believe in
that. The team is,
I mean, it takes a village to drive a
change. So the same thing goes in our DevOps. So we have a
culmination of diverse team members who come from different backgrounds,
and we empower them to actually innovate and even to
fail smart fail rather quickly than later.
So that's where we provide all the capabilities to the developers to
come in on the platforms to actually
talk and experiences and see how things can happen.
So empower is one of the key things which we do,
and then certainly the guide. So making sure that when
you are bringing on DevOps to a large organization like Humana,
you have to have a set of defined
principles, standards,
patterns, policies, because it
could very well go in many different directions. And certainly we want
to make sure that there is a standardized
process for doing DevOps and in order to do that,
one of the things which we have started working on is bringing
principles, standards, policies around, how to do the
branching strategy, how to do progressive delivery, how to do Gitops,
how to do environment strategies and microservices.
So quite a few things which are out there. We have standardized
it so that the developers can feel that they are part of the community and
they can work together in
not only digesting it, but also helping out
other development communities, which we already have in Humana.
So we have a team of over, we have over
400 plus teams within Humana. And I think it takes a
village, again, to kind of make things happen. And that's where this
guide certainly helps in doing that. The other important
thing as part of this model is the
data. How do you collect the data? Like making sure
that in order for you to know that is
my DevOps improving the user experience, is it
helping the developer do the
right way of doing DevOps? We call it as realistic DevOps.
Now, in order to gather that metrics, we certainly have to
make sure that everything is flowing into data.
Lakes and tools like big data
superset comes to rescue, where we can mine
the data, we can analyze the data, we can run
simulations on it and see how it goes. And we build our okrs,
the objective ski and results around it as to how we
can iteratively and incrementally improve
the overall experience of DevOps for the developer community
and eventually for the customers as well. So gather
the data and then certainly measure the data depending upon different
tools which you have and kind of make that happen. So I guess
with that, I would say that currently this is our model.
I would not say that this model is the only model,
but this is the model which currently works for us and we are using
this every day to make it better for the life of the
developer. And Andrew will now talk
about what we have done so far and the
other aspects of how we are scaling DevOps for a
large organization like Humana. And then he will also share some of
the key personas which exist in Humana.
And we'll wrap it up by talking about some of the benefits
and some of the return on investment which
we have seen with DevOps. With that, I will hand it over to Andrew.
Thank you. Rapesh. Great job summarizing our approach. I think
the emphasis on the engineer and how we can get feedback
and support that experience, that's really the key.
So let me get into a little bit of specifics right
about how do we do the DevOps in a large enterprise
like Humana. And I'll take you guys all the way
back to 2018 and kind of the approach that we took as
we move forward, there are really three key steps
and three key kind of methodologies, if you will, that unlocked
our ability to really scale DevOps.
The first one I will talk about is alignment and feedback.
It's hard to really do anything in a large enterprise, as many
of you know, I'm sure many of the folks watching here work in large corporations
without alignment and a go forward strategy.
And I would say in 2018, Humana was in
a unique position, organizationally, where we achieved
a significant amount of alignment across a broad spectrum
of the engineers community, which allowed us to really build
momentum and take significant ground towards DevOps
adoption. So that alignment across the organization,
across every segment, delivering software,
that is probably one of the key, most foundational things that
has unlocked everything that we've done as we've moved forward.
The other piece of it is once you get that alignment, it's really tight feedback
loops. So regular updates to leaders, regular feedback
sessions with engineers to make sure that we are, in fact,
building and delivering the capabilities that they, in fact,
need. So those regular, really tight feedback loops helped us
stay on track and continue to stay in that, I would
say, state of alignment across the organization. The second
key was to focus on platforms and tools
that are engineer focused. And the key to that was
we kind of set a rule at the beginning, is that in order to release
software to production, we don't want an engineer to have
to leave their native platforms. So our
goal was to have our Humana engineers stay within the DevOps platforms
and manage their change through our change process using
their pipelines. And for them to never have to leave that interface,
they should never have to go to a CMDB tool, to some kind of change
management tool. They stay within their native tooling and get all
the feedback they need, and the processes are integrated there at that point.
So that was really key. And then the other thing
is just platforms that are engineer focused,
right? Platforms that help engineers understand
what their posture of their software is that they're creating, whether it's secure or whether
it's quality software. These things need to be integrated into their workflow.
And just not only that, but just the core platforms should be easy
to use. Right. And engineer first. And we
not only did that for our open systems, but we also took an approach
where we have a unique experience for our mainframe engineers.
Humana still has quite a bit of mainframe activity or
development, as well as Salesforce. So we have a unique set of
tooling for each of these personas.
And then the third step, I would say that was most important. And I'll talk
about this a little more detail on the next slide. But Humana is
a highly regulated, we're in a highly regulated industry within healthcare
and primarily our Medicare business, working with
the federal government. And so it's really important to us from a
business perspective that we adhere to
change best practices and that we have traceability and that we're passing our audit
and compliances and not only to our business, but it's important to our members,
right, that we're operating our software in a secure way and that we're running
our business in a secure way from a software perspective.
So we were able to come up with a very unique,
actually now patented solution to
automate Humana's SDLC. Not yet, Rapesh. Go back to
the other side, if you don't mind, sir. You're fine. Jumping ahead.
You get excited. I love it.
But we took a very unique
approach to automating those checks into the pipelines. Again, keeping that
approach that we want the engineers to stay within their native tooling.
And then I would say the final bit is smaller releases,
right. In order to do DevOps, we have to release smaller
in order to iterate and move faster. And that's actually safer and healthier
for a software development team. And so driving our
releases and creating a strong mapping between application components
and repositories not only allowed us to categorize our
source code in a unique way, but also to
help have an enterprise push to drive towards those smaller componentized
releases. So I would say those three key things
really, and there's a lot in that, right? But those three high
level things really helped Humana unlock DevOps. And we've been on a
really fast ride ever since we've done that. So Rapesh
with that, if you don't mind. Jump to the next slide, sir. Thank you.
So let's talk a little bit about Greenlight API. So this is
our patented solution to incorporating Humana's
SDLC into the DevOps pipelines. And so
what you see around kind of the traditional DevOps
Infinity sign that we've all seen,
you have the Humana SDLC, right? The phases of
that. So plan, deploy,
sorry, plan, build, deploy, accept,
feedback. You have the phases of that worked
in to the pipelines in such a way that
Greenland API, which is again, kind of our homegrown Humana
naming, but Greenland API is a task that lives
in the DevOps pipelines, it's required and in fact enforced to
be there. And it gives the developers feedback as
they go upon their software development journey. So as soon as they make a
commit into their repo and they run their build pipeline, Greenlight API
is telling them which parts of the SDLC they've adhered
to or which parts they haven't. Not only that, but it's enforcing
not just SDLC, but also things like security scanning, code quality
scanning, a tight relationship between the source code
repository and the change management or CMDB
database. These things are really
important in the way that we
stitch the releases together for the engineers and stitch that
feedback together, right? So that by the time the engineer gets
to the point where they're ready to release to production,
they've adhered to the Humana SDLC and all the requirements that Humana
has to deploy software just by using
their pipeline, right? And then an integration with our change
system, which is in service now, as you see in the diagram,
automatically opens change orders for those developers and so
sends them through the approval process. So literally,
DevOps do not have to leave their pipeline interface to
release software in a certified and trusted
way at Humana. And it's really helped us to get to a repeatable process where
again, engineers are first, right? Because while yes, we're a large enterprise and
we have rules and regulations and things that we have to follow
to our jobs, that doesn't mean that we can't do it in an engineer
first way. And so that's what we've done here with greenlight API.
So we're very excited about this, very proud of it. And it's
one of the things that has, as you saw in the last slide, that has
really helped us unlock our DevOps journey. So let's
talk a little bit about our developer experience.
Again, I mentioned earlier, we have a unique diagram
and kind of experience for each of our
personas. So we consider open systems, which would be our traditional net,
Java, Python, et cetera, programming languages,
to be open systems. And then of course, mainframe has its own experience.
And Salesforce, we won't get into those as much today.
We've actually done other talks on our mainframe dev experience at BMC exchange
last year. You guys can go find that presentation if you're interested in it.
But for today, we'll focus on open systems. So again,
you can see that we have the tooling aligned
to the phases of our SDLC and which
tool is used, at which point I won't just drain this slide
right I think the audience can get a general overview
and kind of understanding, especially since you're all DevOps practices.
I think the most important thing I would call out here though is that all
of this tooling is integrated, right? It's integrated in such a way that
developers can self service, right? So if
the platform didn't offer it, we've built automations in
the thing that we call our developer control plane where engineers can provision their sonar
queue projects or set up their artifactory JFrog repos.
Right, to manage open source software. So we've taken great pain to
integrate these solutions in a way that are easy
for developers to use. But we're also, I would
say, really passionate about a sense of ownership. So while
as an enterprise team we offer these capabilities, we integrate them,
we make them available and we provide tremendous guidance
on when and how they should be used. We really are pushing towards
a sense of ownership for development teams to build and own their pipelines
because they are part of the application itself.
And again, that's where our guardrails come in, right? That's where we have
the governance of Greenlight API to ensure that while
teams have ownership, they also have some
oversight. Right, and that Greenlight API is going to ensure that before
a release makes it to production, before that release is allowed
to run and deploy to the production environment.
All the checks have been followed, right? All the scanning happened in the build phase
the right way. By the time in the accept phase
we ensure that all the testing was completed in the right way, whether it was
performance testing or manual testing. And then once they get
to deploy, we make sure that the change management process was followed using Servicenow.
And again, all that's automated. But there's some governance there to
make sure that while teams have that ownership,
they're still empowered to do it themselves.
So very excited about this. It's been
kind of the culmination of several years. So Rupesh, I'll kick
it to you man, and let you talk a little bit about some of the
value that we've realized from the work that we've done.
Sorry Andrew. Thank you. I think I might have hit the slide.
You were ready to quit too early, rapesh. We're not done yet.
I guess this is where everything comes to
the fruition of all the hard work, Andrew, you and your team, and done from
past four years, I would say so, I guess. How did we
do? So to put things in perspective, Humana has over 1600
applications and components with over 300 teams,
which are members from around the world. Who are
actually working day in and day out, except for the weekends and
evenings, certainly, but certainly across different time zones.
And we have over 10 million lines of code
with over 17,000 CI CD pipelines.
And these are pretty much managed and powered by over
24 tools which are owned by the DevOps
platform team now. So I guess one of the high level statistics
which I want you to take away from here is that we have seen over
56% of automation
releases over manual releases. And when Andy
was talking about the glappy, the green light API, which is one of the
SGLC policy check kind of a tool which has been patented
by this team. So this actually has allowed us to automate the
releases to production, where it builds the trust with the change
management and enablement team to actually make the releases
go much quicker, much faster and with
much quality. So 56% automation we have done.
And there is an 80% of a lead time which has been experienced
with respect to the dura metrics, certainly with the lead
time, the wait time, the cycle time, the process time.
And certainly this is the figures which we have
run a couple of weeks ago. I'm sure this
might be going up as we speak, because most of the
teams are actually on the DevOps journey.
And the teams which have taken on the DevOps adoption now,
they are actually trying to mature DevOps and they want to do the right thing.
So certainly you will see a bit more numbers
going up as we speak. And then the overall we
have, 97% of our releases are eligible for
releases in production. What it basically means is that the entire CI
CD stages of the pipeline, all the infinity loop which
Andrew was showing earlier, all the different stages of the pipeline, they all
meet the SDLC Humana standards.
And that is a pretty big number for us. And I
hope that in the future we keep exceeding and keep adding
more metrics to capture more streamlined processes and
kind of make it happen for the team. Now with that, I would say that
certainly, Andrew, let me just take it back. Do you have
anything else you would like to add here, Andrew, before we wrap it up?
No, you did a great job. Rapesh. Thank you, sir. Thank you,
Andrew. So I guess, as we all know, DevOps is certainly a
journey and not a destination. So hopefully next year
Andrew and I will come back again to share new
challenges and opportunities which we came across and we
overcame them in Humana. And we would like to
kind of share our stories and certainly learn
from your experience and stories as well. So with
that, I would say thank you for tuning in and
joining our session and hope you have benefited with our
key insights and experiences which we laid.
And hope you have a wonderful evening and other
sessions with that. Thank you so much,
Andrew. Thank you, everyone. Thank you.
Bye, everyone. Take care,