Video size:

0:09 Miko Pawlikowski

Hello, and welcome to Conf42Cast Episode 5, Galactic Bees. My name is Miko, Pawlikowski. And today with me, our guest Charity Majors, the co-founder and CTO at Honeycomb. Charity, really glad to have you here today. How are we doing?

0:26 Charity Majors

Thanks for having me. It's great to be here.

0:29 Miko Pawlikowski

We're starting this little tradition, that we start every guest off now with the same question. If you could have any animal at all as your pet, what would it be?

0:41 Charity Majors

I think it would have to be a monkey. They're just so clever, and always getting into trouble. And I have a soft spot for creatures like that.

0:49 Miko Pawlikowski

Oh, that's a great answer. Do you have any specific kind of monkey or any monkey will do?

0:54 Charity Majors

A small one, I think. The big ones are just too hard to keep. I mean, they eat a lot. And they're big and raaaar. But something that I could physically overpower, I think would be a necessity. You don't want them to be bigger or smarter than you are.

1:06 Miko Pawlikowski

I like that. Would it be a chaos monkey? Do you think?

1:08 Charity Majors

Oh, yeah. I mean, is there any such thing as a non chaos monkey?

1:14 Miko Pawlikowski

That's probably true. So I've been following Honeycomb for a while, you know, we're probably gonna get into that. But one of the first things that I did notice when I was, you know, scanning your LinkedIn before you got here is that you actually hold a few patents. Can you talk about that a little bit? That's pretty amazing. That sounds serious.

1:33 Charity Majors

They're mostly bullshit, honestly. I mean, I think one of them is for, like, it's almost embarrassing. One of them was for like, running JavaScript remotely in the cloud, I think. It happens when you work at early startups where, you know, there's pressure from the VCs, to have IP, like intellectual properties as the company was more valuable and stuff. So people patent things, there is one that I'm, I'm kind of proud of, though, which is the one that we got it Honeycomb. And, first of all, like, I don't believe in software patents, full stop. I just don't and I think that they're an abomination. But it's kind of those things where if everyone else is doing it, you have to or else you get ***. And I am kind of proud of the one that we made at Honeycomb. So it's basically like a hybrid between on prem and, and cloud, right? Because you want your data to be inside of your private network. But you still want the convenience of having somebody else take care of it in the cloud. And so what we did is we made this this proxy, where you can run the proxy inside your secure network, stream your events through it. We compute the in store, the hash and map it to the raw event. And only for the hash Honeycomb, the result is that you get your data, it never leaves your premises. Yet, when the JavaScript in your browser connects to Honeycomb, like we store all the hashes, so you can slice and dice and you can do all the normal things that you would. But when you get the response, your browser knows to look up the raw event from the proxy. So it feels like a cloud service. But your data never leaves your premises. So it's kind of cool.

3:09 Miko Pawlikowski

Yeah, it does sound like it. So you basically tokenize it before sending and so that never leaves the original. So how does it feel to be like a patent holder? Or do you like, feel good about that?

3:21 Charity Majors

Well, the most ridiculous thing is, at Facebook, they gave us these, these very serious, like engraved patents, and a friend of mine took it, just put down hold it ridiculous. So I kind of love that.

3:38 Miko Pawlikowski

Okay, I want all of this on my desk now.

3:40 Charity Majors

I know, right? They want you to have ridiculously, like the dazzled patent lamp.

3:46 Miko Pawlikowski

Okay, one more thing that I did notice is that you've done like a fair amount of, you know, all this infra stuff. And I work with infrastructure quite a lot myself, and I share the pain. So how did you end up starting Honeycomb? What was the moment you felt like, okay, we need to do that?

4:05 Charity Majors

I mean, like everything else in my life, it came from pure rage. It was born from many, many, many, many nights of getting woken up in the middle of the night and sorrow. I was the first infrastructure engineer at Paris, and it was going down constantly, you know, like, it's a multi tenant platform, right? We had over a million apps running on it by the time I left. Which means one of nitsa, iTunes, top 10 or whatever. And everything goes down because we're using like a single pool for workers, a single database. And you know, when something gets slow, everything gets slow. And it's like a chicken and egg problem, right? It's like, well, what's slow? Well, everything's slow because it's all waiting on the same lock, right? So like, I tried every tool out there and just nothing helped. This meant that I was just throwing bodies at the flames including my own. And then there was one tool that finally helped us get a handle on things. It was called scuba. It was as butt ugly, just aggressively hostile to users. It was not fun to use at all. But I did one thing really well which is, like, you slice and dice in near real time dimensions that were very high cardinality. Then you could break down by, you know, one in a million users, and then breakdown by endpoint, you know. And it was just mind blowing to me that suddenly, like, these problems that had taken us days to figure out were, like, seconds, you know, not even minutes. Like, it wasn't even an engineering problem. It was a support problem. And that made a really big impression on me, because suddenly, I wasn't getting woken up anymore. And so when I left Facebook, it was like, well, I wasn't planning on starting a company, I really hate the whole, like, founder, industrial complex. But some people were like, offering me money and I realized that going back to living without this tool would, I would be so much less powerful as an engineer, and I didn't like that, so.

5:49 Miko Pawlikowski

That's a nice origin story, I gotta say, sounds like something from a comicbook.

5:54 Charity Majors

Rage.

5:56 Miko Pawlikowski

So Honeycomb then started BK, right? Before Kubernetes, I mean. Was it?

6:03 Charity Majors

No, it wasn't before Kubernetes. But it was before saying people were running Kubernetes.

6:10 Miko Pawlikowski

Gotcha, okay. Yeah. So do you think like, if you were to start it today, for example, if you know, the kind of tooling that everybody gets with Kubernetes, and entire ecosystem would be a different story, would you still?

6:21 Charity Majors

Oh, God, no. Kubernetes would make everything 10 times worse. 10 times weren't necessary, I should say the problem of cardinality is one, you know. And when I say high cardinality for, if anyone listening doesn't understand, it's like imagine you have a collection like 100 million users. And the highest possible cardinality would be like any unique ID, like your social security number, or random request ID. First name and last name, or lower than unique, but still high, very high cardinality. But gender is for low cardinality and species equals human would be very, very low cardinality. The information that is most useful for debugging is usually the high cardinality stuff. Because if you can pinpoint it to a single pod, or a single process, or a single built ID, then that tells you something interesting. But most tools out there are built for low cardinality stuff. And back in the days, we had just the app server and the database. Most problems could be debugged using just low cardinality tools, right? Because if all else failed, you would attach a debugger and just start stepping through the code. Now that we have microservices and everything, even the service name is a high cardinality dimension. Let alone, you know, everything's gotten smaller and more of them, which is very well ill-adapted to the last generation of tooling.

7:35 Miko Pawlikowski

So it made everything worse, but I guess it brought more business to you guys, right?

7:40 Charity Majors

Yeah, very much.

7:42 Miko Pawlikowski

I don't know if that's a secret, but are most of your plans now, kind of Kubernetes-based stuff, or is it kind of a mix?

7:50 Charity Majors

It's a mix of everything. I would say that most of our customers are doing microservices in some form or another.

7:56 Miko Pawlikowski

For good reasons, or just fashion reasons?

7:58 Charity Majors

Yeah, I plead the fifth.

8:03 Miko Pawlikowski

Fair enough. Okay. Smooth transition. Can you tell me one situation that you would like to share about some funny or interesting or haunting outage that you can describe? That you like, telling people about anything particular that springs to mind?

8:21 Charity Majors

Oh, gosh, out of all the outages that have come before.

8:25 Miko Pawlikowski

Your number one.

8:26 Charity Majors

Most of my worst stories for the past many years involve MongoDB because we were growing up with it at the same time as it was becoming a real database.

8:36 Miko Pawlikowski

You mean, before it went webscale.

8:39 Charity Majors

Yeah, yes. Before it went webscale, exactly. But I think that one of my favorite outages would have to be the last time in my life that I ever had to call a cab at 2am and go down to the club and flip the power switch at 2am. That was a glorious moment. And I will never repeat it, because now we have AWS to do this for us. And the world is a better place. I was talking to some kid the other day who is complaining about the cloud and AWS: "Oh, it's so complicated and all this stuff. Why should anybody use it?". And I was just like: "Have you ever driven to the club in the middle of the night? No? Then you don't get to have an opinion".

9:19 Miko Pawlikowski

The rite of passage. You should do it at least once to appreciate it.

9:23 Charity Majors

Exactly! We walked uphill, you know, in this in the snow barefoot both ways to get to our power.

9:31 Miko Pawlikowski

Yeah, speaking of which, of AWS, I watched Shelby stack on Graviton tune and sounded like you made AWS ran out of hardware. Is that they give you a badge for that? Did you get like another trophy that you can put on your desk?

9:46 Charity Majors

I wish, I wish.

9:48 Miko Pawlikowski

Yeah. That sounds like an achievement. I guess that gets you extra points. But I quite like the transparency that gets point in my book too. One thing that I also noticed, there was a different type of pattern, is that everything was named after a species of a dog. Everything for dogs.

10:05 Charity Majors

Yeah, Retriever. No, I don't, my co-founder does.

10:07 Miko Pawlikowski

Okay, so that's the story. Everything shall be named Chihuahua. Is that how it works?

10:14 Charity Majors

No, it's like the database is named Retriever and the front end is named Poodle. And you know, the proxy things named Basenji. There's just a lot of dog lovers at the office. I'm dog agnostic. They're a little meaty for my tastes, but they're fine. The other reason we have dunk games is because the company was originally named Hound, or Bloodyhound. Then we quickly shortened it to Hound and then we got a cease and desist from FCI. So that's how we became named Honeycomb.

10:46 Miko Pawlikowski

That's, that's great. We were actually thinking about naming the episode something with puppies, but then, you know, you didn't mention any puppies on your website. So I wasn't sure if it was like an official thing. I guess that explains why you have not mentioned any dogs on the website. Okay. One other thing I wrote down is the meantime to WTF. Did you come up with that? Or is that someone else's? Because I'm totally stealing that.

11:10 Charity Majors

I don't remember. I say shit all the time. And people quote me, and I'm like, I said that?

11:16 Miko Pawlikowski

Okay, that's fair. What about observability? I think you define it on your website, but can you give me your take? Because it's kind of like one of those things that I really liked, the word, but then it kind of becomes like a buzzword now.

11:29 Charity Majors

Yeah, it's really unfortunate, I will take some blame for the fact that it is a buzzword. Nobody was using the word when we got started. And I was looking for a way to differentiate what we were doing from monitoring, right? Because I've got mad respect for monitoring, you know, everybody should do some monitoring. But it wasn't what we were doing, right? When I googled the definition of observability, which comes from mechanical engineering, it's like the mathematical dual of controllability is observability. And it has to do with, can you understand what's happening inside the system? Any state that it's gotten itself into, just by asking questions from the outside. And that's when I had light bulbs going off, like: "Oh, my God, this is exactly what we're trying to do". The last generation of tools, it was built on metrics. It's all about dashboards, which are just like these artifacts of past failures. Every time tried to bug is littered with dashboards, right? And then you've got people that are just like, skinny dashboards are trying to jump to conclusions, then go look for evidence that they're right. But like with observability, it's about unknown unknowns, right? If monitoring is about known unknowns, observability is about unknown unknowns. And so it's about, the toolset that you have allow you to interrogate that state. So you would start with no prior knowledge. And without shipping, any custom code to handle that state, understand the state. And if you accept that definition, then there is a bunch of technical prerequisites that you need. You need to be able to handle high cardinality. You need to be able to handle high dimensionality. Any sort of indexing is, like, out to write any sort of like schemas are out because they all involve predicting what kind of data you're going to see and what kind of questions you're going to need to ask. I feel like, you know, we started talking about these problems and the solutions, and suddenly, everybody was like: "That sounds great. We do that too". And it's like: "No, you don't. There's a technical definition here. I would love it if you didn't need the technical definition, but you do". And so I feel like there's been a lot of froth in the environment over the past couple years. Because everybody's just like: "Yeah, it's just sort of a synonym for telemetry, right?". It's like: "No, that wasn't really the point of it, there's a technical definition". And it behooves us to be clear on this point, because there are lots of things that, you know, monitoring and metrics tools are great at. There are different things that observability tools are great at. And the best practices that you need are very different, like, for example, a classic monitoring best practices, you shouldn't have to look at the dashboards all day. The system should inform you when something's wrong, so that you will look at it. Whereas observability is much more about being in a constant conversation with your code. And looking at the graphs every time you ship new code and comparing it to doing what I wanted it to do. Like, through the lens of the instrumentation that you just wrote, is it doing what you expected? And is anything else look weird, right? That's a much more active and constant like engaging with the graphs than just the sort of like, it hasn't reached a threshold that sometimes you get paged, right?

14:20 Miko Pawlikowski

That makes a lot of sense. I don't know if you've heard but there's a new kid on the block now. There is the understandability that's the, all the rage now. You got to rebrand now? I think it's kind of like observability with the, you know, additional layer of actually understanding what you're observing.

14:40 Charity Majors

This is what's so frustrating, because if you have the right tools, and you look at your ***, you will understand it. And if you don't, you won't. And yeah, it's frustrating.

14:52 Miko Pawlikowski

I feel like we're kind of like circling in. I understand: you have Humio, you have flux, you have some dashboards, you made some perimeters dashboards. What is, like, the secret sauce now that turns this data into observability in the way that Honeycomb does it?

15:11 Charity Majors

First of all, Prometheus is a metrics based tool. It's a monitoring tool. The source of truth for observability, the data format is these arbitrarily wide structured data blobs. You gather your telemetry in one of those wide events per request per service. One way to think about this is when we blew up the monolith, we lost the ability to step through our code, because every time you have the network, you discard all that context. So by bundling up your telemetry this way, whenever truly wasters data blob request per service, you're actually shipping that context around with the request as it hops around. You can derive metrics from the arbitrarily wide changing blobs. You can derive logs from them, you can derive traces from them. You can't go the other direction, you can't take a Prometheus and get observability out of it. You can take the observability tool and get Prometheus data out of it. But it doesn't go in the other direction. Try to jam on these old tools, you can't get observability out of them. You have to start with the right data format.

16:10 Miko Pawlikowski

That makes sense. Question: I might be a little bit biased, but how do you see, like, Chaos Engineering, in fitting into all of that? Is that something that helps you with the observability?

16:22 Charity Majors

Chaos Engineering is being, like, there's been a massive shift in the landscape in the past. You know, it used to be that we're sinking all of our energy and all of our developer cycles into pre-production hardening. And all of these staging environments and all of this, blah, blah, blah. And I think that we reached the diminishing returns, to say the least. And I think over the past five years, you've seen a lot of little startups like us, and LaunchDarkly and Gremlin. And so the focus is, you know, shining light on pardoning your production systems themselves, and on instrumenting, and tooling, and just like feeding visibility, and understanding the understandability, right, of your production systems. And I think Chaos Engineering is part of that. I think that, you know, if you're just throwing Chaos Engineering at your systems, and you don't have observability, then then I would argue that you just have chaos. You should be able to see what you're doing before you go and start adding more of it. But I think that all of these trends are good. I think that they're all necessary. I think that we have really ignored production for too long when it comes to where we spend our tooling and our and our time.

17:21 Miko Pawlikowski

And I think part of the fact that we stopped ignoring is also because you know, the SRE and all this kind of ecosystem. And the cool factor kind of also made it possible to say: "Oh, yeah, I work on basic operations, and it's fine. And I have a cool company around them". Tell me, what do you think would be like the main technologies or methodologies? What would you look at in tech for the next few years? You know, what's the next step? Now, logically, after observability.

17:54 Charity Majors

You say that as though anyone's gotten there yet. There are very few companies who have achieved and kind of observability out there. And part of that is because of all of the muddying of the waters. Part of that is because it's hard to change. It's hard to learn new things. And I don't honestly think that you can ask someone to learn a new tool, change the way they do things, unless what you're offering them is an order of magnitude better than what they have. And I think that it's really just in the last couple of years that we can finally say that, yeah, actually, you know, this tooling has gotten mature enough that it's pretty drop in. It's not in the bleeding edge anymore. It's going to help more than it hurts. You know, certainly to like an order of magnitude.

18:34 Miko Pawlikowski

That's definitely true. But it's always nice to have like a little thing to look out for, put on your radar. There must be something that you can name here. Is it EBPF?

18:45 Charity Majors

I don't know dude, I'm an Ops. You know how much we hate new things. I don't even like to upgrade apps in my phone.

18:54 Miko Pawlikowski

Okay, so a little bit more of a personal question. If you had, like a kid come to you and say: "Okay, hey, what should I do if I wanted to be kind of like you when I grow up?" What would you tell them?

19:07 Charity Majors

Oh Jesus, don't!

19:08 Miko Pawlikowski

You seem to be having blast with a lot of different weird systems and defining observability. That sounds like a lot of fun, doesn't it?

19:16 Charity Majors

Honestly, starting companies is the most painful, awful thing I've ever done and I don't recommend it. It's gotten better. But the first four years were grim. Is it fun? I guess. It's fun in retrospect. I think that I mostly experience fun, retro-actively when I like, yeah, that was worth doing, you know. But I tend to lean into pain, I guess, a lot.

19:39 Miko Pawlikowski

All right, one last question, kind of to give our audience a little bit to take home. What would you say is like a single highest return on investment thing or activity that you did for your career? It could be an object or a course or whatever it is.

19:57 Charity Majors

The single biggest thing that I did for my career was learning to speak in public. I was not born with the ability to speak and think at the same time. And, like, I I gave my first talk, it was like a 10 minute talk at reinvent in 2015. And it was humiliating, like I couldn't, I was so terrified I printed out every word that I wanted to say, because I couldn't. I knew I couldn't remember anything if there are people staring at me. And it was so humiliating that I went home and I was like, that can never happen again. And so I submitted talks to like every conference that I could, and I started giving talks like one or two a month for the next three or four years. I got a prescription from my doctor for anti-anxiety medication. This blood pressure, it blocks adrenaline receptors in your brain so that you don't shake from the nervousness because I would just, I would have nightmares for months before each talk. So it was intensely awful. But about a year and a half after giving talks every, every couple weeks, I started forgetting to bring my prescription with me. And that's when I was like, huh, I've cured my fear of talking in public. I just don't see how I could have done anything like doing this podcast or pitching for money or any of that stuff. It just wouldn't have been possible if I hadn't overcome my fear of speaking in public.

21:14 Miko Pawlikowski

Wow, I can totally relate. I remember my first time. Jesus Christ, my hands were so sweaty. Fortunately, not many people came to my talk, so that helped. So, when you mentioned prescription, I thought you were going to say that the doctor prescribed you public speaking to overcome your fear. I was like, oh, wow! That's unorthodox.

21:37 Charity Majors

Propranolol is what it's called. If any, and kids out there want to go ask your doctor for some drugs. It'll help you get over fear public speaking. It's called propranolol.

21:45 Miko Pawlikowski

Okay, I guess we should add at the end. This is not medical advice. Seek professional help, if you need any of that. I've heard quite a few people actually say that the increase in just stamina for speaking in uncertain situations has really had a massive effect on the trajectory. So yeah, I think that's pretty good advice. That's really cool. Okay, this was really lovely. I've really enjoyed talking to you. I think I've learned quite a bit. Would you like to tell our viewers now, who might have heard about Honeycomb for the first time, how to go and get started, where to go and what to do?

22:23 Charity Majors

We're on the internet - honeycomb.io. We have a really cool blog where we talk a lot not just Honeycomb stuff, but how to instrument your code, how to think about observability, how to transform from the old up and down like monitoring sort of way to a more modern, you know, way of thinking about your systems. Also Honeycomb io on Twitter, or you can find me mipsy tipsy on Twitter, and my blog is charity.wtf.

22:49 Miko Pawlikowski

Awesome. Don't forget about Shelby's talk. Make sure you check that if you want to learn a lot of different names for different dogs. I definitely didn't know all of them. All right. Thank you so much. This has been a blast, we'll see you next time.

23:01 Charity Majors

Thank you for having me.