Transcript
            
            
              This transcript was autogenerated. To make changes, submit a PR.
            
            
            
            
              I'd like to start out my talk by dropping a little bit of a truth
            
            
            
              bomb on you, and that's that. If there's one thing I've
            
            
            
              learned in my career in tech, it's that we get to choose
            
            
            
              a lot. Now, we may not like
            
            
            
              what we're choosing things horror
            
            
            
              shows from my past that I've had to choose between, and we may not like
            
            
            
              why we have to choose any of these things at all.
            
            
            
              And we definitely don't like how often we get to choose.
            
            
            
              Don't bother reading the details of this slide. It's an eye exam that basically is
            
            
            
              saying, shit changes all the time. In fact,
            
            
            
              one of the things that I laughed about the most at the start of the
            
            
            
              pandemic was all the media outlets saying, oh, it's the new
            
            
            
              normal. And I'm thinking, oh, you mean when everything we know
            
            
            
              gets thrown out the window and we have to find a completely different
            
            
            
              way to do every aspect of our jobs? Yeah,
            
            
            
              in tech, we have a word for that. It's called, oh, my gosh, look,
            
            
            
              it's Tuesday. The sheer volume of
            
            
            
              opportunities to choose creates a number of challenges,
            
            
            
              but maybe none as much as the paralyzing fear
            
            
            
              that we might choose wrong, which is the
            
            
            
              point of my talk today. Thank you for joining me. As I look back across
            
            
            
              a lifetime of poor technical choices
            
            
            
              or another title for this is when you stare into the face of
            
            
            
              failure and success, smiles back.
            
            
            
              I don't know. It just happens. Thank you for joining
            
            
            
              me. My name is Leonidato. I'm a middle aged white dude giving a conference
            
            
            
              talk. I selected this picture to give you confidence that I know what
            
            
            
              the hell I'm talking about. Nothing else on that slide really matters,
            
            
            
              except that you can find me, as at Liana Dotto, on almost all
            
            
            
              social media. More importantly, I want
            
            
            
              you to understand that this is what I call an oyster talk. Now, I got
            
            
            
              that idea from the parenting educator, Barbara Colaroso.
            
            
            
              And the point is that I have about half an hour to
            
            
            
              talk to you here, and I cannot teach you anything in that amount of
            
            
            
              time. In fact, in that amount of time, about the only thing I can do
            
            
            
              is irritate you. Now, when an oyster becomes
            
            
            
              irritated, it does one of two things. It either creates a pearl
            
            
            
              or it dies. Now, it is my sincere
            
            
            
              hope that I do not irritate you to death today. However, I did think
            
            
            
              it was fair to let you know that those are some risks.
            
            
            
              Instead, what I hope is I give you just the grain of an idea
            
            
            
              that you will then think about and layer over with your own experiences
            
            
            
              until what you have is something that is polished and unique
            
            
            
              and valuable to you. Now,
            
            
            
              after 35 years in tech, I've learned something, and I want to share it with
            
            
            
              you all right up front. And that's that. There is no such
            
            
            
              thing as a poor technical choice. I know because I've
            
            
            
              made some doozies. And today I'm just going to share a few
            
            
            
              of the residents living in my technical house of horrors.
            
            
            
              Okay, I take that back. There are some common decisions which
            
            
            
              are clearly bad. For example, not having backups, that's bad.
            
            
            
              Or thinking that replicas or mirrors or raid are backups,
            
            
            
              that's also a bad technical choice. Moving fast and breaking
            
            
            
              things in test, sure. In prod,
            
            
            
              not so good. You do have
            
            
            
              a test environment, right? Like we all should have a
            
            
            
              test environment. We should all have an environment that we test things in that is
            
            
            
              not actually production. Okay, you get my point. Bad technical
            
            
            
              choice. Trying to do everything all by yourself. Bad technical
            
            
            
              choice. Not a personal choice. A bad technical choice. Using oracle
            
            
            
              for literally anything at any time can be considered a
            
            
            
              bad technical choice. Starting a land war in Asia.
            
            
            
              Okay, fine, that was a princess bride quote, but I still had to include it
            
            
            
              there. All right, so let's start. In 1984,
            
            
            
              case Western Reserve University, IBM, and at and T wanted to build the
            
            
            
              world's first free online community. Now, just for context,
            
            
            
              case Western is a university not known for its technical program.
            
            
            
              IBM is the stodgiest computer company in existence.
            
            
            
              And at T is the zombified remains of a phone provider that
            
            
            
              got more or less legislated out of existence. What could possibly go
            
            
            
              wrong with this plan? And unbelievably, they succeeded.
            
            
            
              They created a free public community computer system
            
            
            
              which bore a strong resemblance to the BBS software I downloaded
            
            
            
              onto my dad's compuware pc and ran when he
            
            
            
              wasn't home, until he got the bill after the first month, the telephone
            
            
            
              bill after the first month, and shut that all down. Now, I spent a
            
            
            
              lot of time on freenet for a few reasons. First of all,
            
            
            
              it was free. Second of all, you could reliably download uu
            
            
            
              encoded ASCII porn on it. And third, there was a one
            
            
            
              limit, time limit on that BBS system unless you became
            
            
            
              a moderator in one of the SiGs or special interest
            
            
            
              groups. So I became an admin on the dungeons and Dragons Sig
            
            
            
              because, of course I did. And I also became an admin on
            
            
            
              the word processing Sig and a few others along the way.
            
            
            
              Now, you'd have to be a child of the understand exactly how
            
            
            
              much time you could sync into a dial up service. But you would
            
            
            
              argue it's a complete waste of time, right? So freenet
            
            
            
              was basically a glorified message board. And what you did on it was
            
            
            
              that you posted messages on different areas and you
            
            
            
              sent comes to the maybe 20 other people in the world who had
            
            
            
              an email address in 1986. And to do that, you needed
            
            
            
              to select an editor to actually write that stuff. Now, you had a choice.
            
            
            
              You could use a homegrown abomination called Fred or
            
            
            
              Emacs or Vi entirely at random.
            
            
            
              I chose Vi as my default editor. I was 16
            
            
            
              and I didn't know any better. And I learned Vi like it was
            
            
            
              the most natural thing in the world. So,
            
            
            
              fast forward to one of my first jobs. I was working desktop support in
            
            
            
              an engineering company, and the Unix admin came down. He was looking
            
            
            
              for one of us to train into supporting one of the new
            
            
            
              systems. And so he sat us down. He was one of those typical,
            
            
            
              you know, grumpy big beard and everything. And he talked
            
            
            
              to us and he said, okay, well, I'm going to have to pick one of
            
            
            
              you boneheads to do this. And even before I can teach you the system,
            
            
            
              I have to teach you this weird thing that nobody ever gets, and it
            
            
            
              takes forever for you people to learn. It's called Vi.
            
            
            
              You know that scene in Jurassic park?
            
            
            
              Yeah. The point is that without those hours I spent on
            
            
            
              freenet, I wouldn't have gotten the chance to move up.
            
            
            
              By the way, the system that I learned was bind version
            
            
            
              eight, the first production version of what would eventually
            
            
            
              be called DNS. There was no part of this talk where I say that learning
            
            
            
              DNS was or is a good choice, just to be clear.
            
            
            
              So next story, I got my start in tech as
            
            
            
              a classroom instructor teaching adults how to use things like Wordstar
            
            
            
              Lotus one, two, three. Word perfect. And my boss thought it was important that
            
            
            
              I have some certification since I was coming from outside
            
            
            
              the tech sphere. And so I spent 80 hours taking my word
            
            
            
              perfect certified resource test, not studying for the test.
            
            
            
              It took me 80 hours to take the test.
            
            
            
              Back in my day, we had to certify 10 miles uphill
            
            
            
              both ways in the snow, and we liked it anyway.
            
            
            
              Where's word perfect now? I mean, it's kind of
            
            
            
              part of comes, but not really, but bought by Novell anyway, it was a
            
            
            
              complete waste of time. Right? By the way, you can see the certification right there.
            
            
            
              Yeah, complete waste of time, right?
            
            
            
              Well, Wordperfect had gone the revolutionary route of
            
            
            
              hiding formatting codes before that, if you wanted to format something, you actually put
            
            
            
              the formatting codes on the screen with everything else, but you would
            
            
            
              see instead a representation of bold or underline or whatever.
            
            
            
              Now, sometimes you had to fix overlapping codes, and for
            
            
            
              that there was a feature called reveal codes. It looked like that.
            
            
            
              So you can see down below in the second bottom half of my screen here
            
            
            
              where there's underline and where those bold starts and stops and
            
            
            
              all that stuff. Which is why a few years later,
            
            
            
              it took me about 15 minutes to wrap my head around how HTML
            
            
            
              worked. And that helped me in a lot of ways that I'm going to talk
            
            
            
              about in a little bit. But let me keep going.
            
            
            
              So, like you, my browser has about
            
            
            
              8000 open tabs. But before the Internet,
            
            
            
              the way that you would keep up to date on stuff is voraciously reading every
            
            
            
              tech magazine. I would tear out articles that I wanted to keep.
            
            
            
              I'd tape the pages into scrapbooks
            
            
            
              and create my own IT technology murder board and everything.
            
            
            
              And maybe worst of all, I really liked what I was learning
            
            
            
              and so I wanted to share it with everybody I knew. And so I would
            
            
            
              try to engage my coworkers in conversations. Did you see how Borland
            
            
            
              is taking on the limb standard? I was really a
            
            
            
              lot of fun at parties, let me tell you. I was so desperate
            
            
            
              to share the information that the Unix
            
            
            
              admin, the one who I knew Vi, and so he took a liking to me.
            
            
            
              Well, he knew I knew Vi and he knew I learned HTML pretty
            
            
            
              quickly. So he set up a web server for me and so I could
            
            
            
              type the snippets of the magazines into web pages and
            
            
            
              share them. Now, literally nobody ever read those pages,
            
            
            
              but I learned how to create web pages really
            
            
            
              quickly. And when I was out of work a few years later, I had
            
            
            
              everything I need to start building websites for people on the side and
            
            
            
              do it faster than most people could. Now, I'll explain more about that in a
            
            
            
              minute also. But knowing HTML literally helped me put food on
            
            
            
              the table. But there's more.
            
            
            
              Typing HTML week after week got tedious, so I asked the
            
            
            
              Unix admin how to automate it. He taught me literally enough
            
            
            
              perl to be dangerous. I wrote ridiculously complicated
            
            
            
              scripts that would take the raw text input out of a text file
            
            
            
              and then smash it together into web pages and figure out where the line breaks
            
            
            
              were and add the paragraph marks and everything and figure out what the next page
            
            
            
              name was supposed to be and then update the index HTML file.
            
            
            
              I was very, very proud of my achievement.
            
            
            
              But like I said, nobody ever read those pages. So what
            
            
            
              value did any of that have? So my leap
            
            
            
              into the world of monitoring started with a tool called Tivoli.
            
            
            
              Now later on, I would include tools, everything from
            
            
            
              openview to Nagios, to SolarWinds, to new
            
            
            
              relic to Shinkan to Zabix and Grafana.
            
            
            
              But I have to admit, Tivoli was my first tool
            
            
            
              for its time. Tivoli was a comprehensive multi million dollar
            
            
            
              solution. It had modules for system inventory and software distribution
            
            
            
              and metrics collection and event correlation.
            
            
            
              Tivoli was also basically a gang of perl scripts and
            
            
            
              a trench coat and that lobbed monitoring data
            
            
            
              into a Corba backend database. Nobody with an ounce of common
            
            
            
              sense wanted anything to do with this freak show of a tool. But of course,
            
            
            
              I heard the word perl and I was all in. And just like
            
            
            
              when I was 16, learning Vi, I was too new to everything to understand
            
            
            
              how truly awful it was. It all seemed perfectly normal.
            
            
            
              But everything I learned set me up for the next 25 years
            
            
            
              of my career. Now, Tivoli was good
            
            
            
              at a lot of things. Windows was not
            
            
            
              one of them. So we had to find this weird third
            
            
            
              party tool called SMS, which you see on the screen.
            
            
            
              But it's not actually the Microsoft SMS it was. Before that,
            
            
            
              it was just SMS systems,
            
            
            
              server monitoring system. And integrating the data
            
            
            
              wasn't just hard tools at that time not
            
            
            
              only didn't easily integrate
            
            
            
              or combine themselves with other tools,
            
            
            
              they actively resisted the
            
            
            
              effort of integration. It took months to get this
            
            
            
              all to fit together. And in all honesty, it was a bad tool and
            
            
            
              we probably should have built it ourselves at the time. But we learned a
            
            
            
              lot about integration and data types and transforms.
            
            
            
              And believe it or not, I use everything that I learned back at that time,
            
            
            
              even today as I'm working with newer and more robust
            
            
            
              tools. And yes, that SMS was bought by Microsoft
            
            
            
              and turned to Microsoft SMS, which became SCCM,
            
            
            
              which then became SCOM, which then became ECM, which then became whatever
            
            
            
              the heck Microsoft is calling it this week. And again, knowing the
            
            
            
              underlying foundations of how tools are built definitely helps.
            
            
            
              Tivoli had a GUI for doing things like grouping
            
            
            
              systems together for software distribution. And naturally you could create groups,
            
            
            
              and you could create groups of groups, and then you could create a group
            
            
            
              of groups of. Actually you couldn't do that because
            
            
            
              every time I did that, the entire database would crash and corrupt
            
            
            
              and I would have to either rebuild it from scratch or restore it from a
            
            
            
              backup. And I really wanted to know why. And so I did some digging.
            
            
            
              Remember that Corba backend that I talked about. It was one
            
            
            
              of the first object oriented databases in history,
            
            
            
              but it could only handle three levels of containership, of object
            
            
            
              containership. After that, it would just die. Now,
            
            
            
              Corba wasn't my choice. It was Tivoli's choice when
            
            
            
              they. But the tool. But over the three separate weekends
            
            
            
              that I spent rebuilding the Tivoli database, I learned a lot about the impact
            
            
            
              of downstream consequences and also the importance of
            
            
            
              understanding the architecture that was being implemented and knowing
            
            
            
              the ramifications of that. Because sometimes a software tool's pedigree
            
            
            
              really does matter. Okay. The work that
            
            
            
              I did on Tivoli got me noticed at the company I was at, which was
            
            
            
              Nestle, and they offered me the chance of a lifetime. They were
            
            
            
              going to move me and my family to the world headquarters
            
            
            
              and work on a really big project, global monitoring.
            
            
            
              250,000 systems in 5000 locations.
            
            
            
              So they packed up my family and we landed in Switzerland
            
            
            
              on September 1, 2000, and 110
            
            
            
              days before 911. It was also six months,
            
            
            
              almost to the day before the.com bubble burst,
            
            
            
              at which point everyone in the project was sent home in the middle
            
            
            
              of a recession. I was out of any
            
            
            
              kind of steady work for 18 months.
            
            
            
              I spent a lot of time feeling that that choice that I'd
            
            
            
              made to accept the opportunity to move to a new country
            
            
            
              and everything, I really spent time feeling that that choice had
            
            
            
              ruined everything.
            
            
            
              And over the next few months, I learned a little
            
            
            
              bit about humility. I learned a lot about myself
            
            
            
              and mental health and my value as a person
            
            
            
              and not just the source of a paycheck. I learned who my friends
            
            
            
              were and who I could rely on. And I learned
            
            
            
              that times can get very, very tight
            
            
            
              and still contain moments of joy and
            
            
            
              how important it was to recognize those moments and
            
            
            
              relish them and just be in
            
            
            
              those moments, regardless of what's happening around you.
            
            
            
              Okay. The next story actually wasn't me,
            
            
            
              but I observed it very close up and I wanted to share
            
            
            
              it with you. It's kind of poor technical choice. Voyeurism or rubber
            
            
            
              necking, as the slide says. All right. I saw two
            
            
            
              different sev one events. Events that brought
            
            
            
              the entire system crashing down. Two events, one week apart.
            
            
            
              In both cases, it was because of a configuration
            
            
            
              change that happened in production. One person
            
            
            
              was immediately walked out the door. The other person was given a stern talking
            
            
            
              to. So what was the difference? The first person made
            
            
            
              the change, saw that everything had crashed, immediately flagged
            
            
            
              it, let people know, owned up to it. They couldn't
            
            
            
              fix it. They didn't have the permissions or the skill, but they
            
            
            
              offered to be there through the entire process. They offered to bring coffee,
            
            
            
              to bring pizza, to watch, to learn,
            
            
            
              to make sure that they understood the impact of
            
            
            
              what they had done so that they would never do it again and that they
            
            
            
              could actually help when they saw other problems that were similar happen
            
            
            
              like it. The second person saw what
            
            
            
              happened and tried to bury it. There's always
            
            
            
              a log file. It's always in the logs. Now,
            
            
            
              those lesson here should be pretty obvious. Do not try to hide your mistakes.
            
            
            
              There isn't or shouldn't be the expectation of perfection,
            
            
            
              but there is, or at least there should be an expectation
            
            
            
              of decency. A few years ago I was tweaking an alert and
            
            
            
              I accidentally caught 772
            
            
            
              tickets twice in 15 minutes.
            
            
            
              And the worst part was, I didn't even know that I had done it
            
            
            
              until the entire help desk came up to my cubicle and invited
            
            
            
              me down to their area to help them manually close
            
            
            
              the 1544 tickets one
            
            
            
              by one. Yes, there was a mass close function. They wouldn't let
            
            
            
              me use it. Now, with those benefit of time
            
            
            
              and experience, maybe some alcohol, I can reflect
            
            
            
              on this formative event and wonder whose fault was it? Was it
            
            
            
              the tool that allowed such a thing to happen? Or was it the operator?
            
            
            
              There's probably enough blame to go around, but in a larger sense it
            
            
            
              was an immature philosophy about what an alert actually is.
            
            
            
              Now, all of that is the topic of a completely different talk. But for
            
            
            
              today, the lesson is on the screen right there.
            
            
            
              That screen was created by the vendor after the event
            
            
            
              I just described, because I told them about it and said, why did you let
            
            
            
              this happen? I wasn't the first person to experience it,
            
            
            
              but when I became a Devrel advocate for that vendor
            
            
            
              a couple of years later, it was something that I could articulate very
            
            
            
              passionately and convincingly and get them to make a
            
            
            
              meaningful change until they resolved it and gave a
            
            
            
              screen that said, oh my gosh, you're about to do something bad. Maybe think
            
            
            
              about it. So that's the point, is that the experience
            
            
            
              I had fed into my ability to
            
            
            
              make meaningful change. As I said at the beginning,
            
            
            
              the sheer number of choices and possible negative
            
            
            
              outcomes can be paralyzing. It is a Whitman sampler
            
            
            
              of possible crap. In fact, you get this sort of choose
            
            
            
              your own adventure kind of list of things. For example,
            
            
            
              do you do kubernetes or do you choose not to do kubernetes? Do you
            
            
            
              implement an integration to chat jeopardy. That is how you say it.
            
            
            
              Or do you not do it? Do you use blockchain or not use blockchain
            
            
            
              or this framework or that language or this platform?
            
            
            
              Or do you put your system on AWS or Azure or
            
            
            
              GCP or all three of them? Do you use oracle for
            
            
            
              literally anything ever? No, you don't.
            
            
            
              Do you use on prem versus colo versus cloud versus
            
            
            
              a hybrid approach? Do you use monitoring with this solution or
            
            
            
              that solution, or all the solutions combined,
            
            
            
              or do you build it yourself? All of these choices
            
            
            
              hold potential success or disaster,
            
            
            
              and there's almost no way of knowing Seth
            
            
            
              Godin, one of my personal heroes, addressed those inherent fear many
            
            
            
              of us have. He called it technical fomo,
            
            
            
              the fear of missing out, the fear of not getting into
            
            
            
              or onto the next thing. And it causes us to hesitate
            
            
            
              to get involved in something now. But what he said
            
            
            
              is so very relevant. This thing, those thing we have
            
            
            
              now, is worth working with because it offers so
            
            
            
              many opportunities compared with merely waiting for the next thing.
            
            
            
              It's always a good choice to work with
            
            
            
              what you have now and not worry that you'll be
            
            
            
              too engaged to pick up the next thing, which may be better,
            
            
            
              but I know I said that my point was there were
            
            
            
              no poor technical choices. But can
            
            
            
              that be it? I mean, that seems really trite, really banal, really predictable.
            
            
            
              Maybe the lesson is that the real treasure was the cables that
            
            
            
              we collected along the way. Or maybe
            
            
            
              it's something else. One last set of lessons.
            
            
            
              I learned the word loquacious very early. Loquacious means
            
            
            
              tending to talk a great deal. I learned it in fourth grade when
            
            
            
              it appeared on my report card. Because my teacher
            
            
            
              used it to describe me, I'd end up seeing that word a lot. Also overly
            
            
            
              communicative and oversharing and diarrhea of
            
            
            
              the mouth. I also wrote a lot
            
            
            
              all through school.
            
            
            
              Now, my notes were comprehensive, but not exactly
            
            
            
              on topic. If you wanted to know whose chair made that weird farting
            
            
            
              noise at 1115 in social studies class, I was your guy.
            
            
            
              My notes may not have had the facts or details, but the 1812 war,
            
            
            
              but in their own way they were comprehensive.
            
            
            
              Also, from the time I was ten years old until I graduated
            
            
            
              from college, 100% of my time was spent pursuing a
            
            
            
              career in theater. In fact, my senior year I
            
            
            
              had two academic classes and seven classes that
            
            
            
              were choir, drama, band, and so on.
            
            
            
              All that time spent, and I'm not doing any of that today.
            
            
            
              So you could argue that all three of these things that you see were poor
            
            
            
              uses of my time. But on the other
            
            
            
              hand, I'm a developer relations advocate
            
            
            
              or evangelist or whatever you want to call it.
            
            
            
              And how do you do that job? Well, you do a lot of talking to
            
            
            
              customers. You participate by typing in
            
            
            
              online forums. You create videos like this one you
            
            
            
              present at conference talks, you are with people on podcasts
            
            
            
              and so on, along with everything you see here. Remember that
            
            
            
              my entry into technology was to provide training to learn
            
            
            
              how to relate to a group and present detailed information in an interesting
            
            
            
              and even entertaining way, and sometimes to take a
            
            
            
              technical pie in the face just to keep everyone enjoying themselves.
            
            
            
              All that writing early on served me pretty well in the blogging circuit
            
            
            
              and theater definitely helps me be comfortable in situations
            
            
            
              like this where I'm standing up in front of an audience of people and sharing
            
            
            
              ideas with you. Today there really aren't any
            
            
            
              poor technical choices, and that's because it's
            
            
            
              not about the tech. In fact, it was never about the tech.
            
            
            
              It will always be about you.
            
            
            
              I've left this slide blank on purpose to
            
            
            
              leave room for what you bring to the story. You're probably already thinking about
            
            
            
              your experiences and how they relate to the ones I've been sharing. You, or are
            
            
            
              similar or different in many different ways.
            
            
            
              We pick up technical puzzle pieces throughout our
            
            
            
              journey in our career, and often it seems like the things that we pick
            
            
            
              up have nowhere to go, but the truth is that they will
            
            
            
              all fit somewhere eventually. In fact,
            
            
            
              I felt so strongly that everyone's journey is valuable and worth
            
            
            
              talking about that I made this talk open source. If you go
            
            
            
              onto my GitHub repo, which you'll see there, you can download it and
            
            
            
              submit it as a CFP to your own conferences, share your own amazing
            
            
            
              details, and run with it. There is never going to be
            
            
            
              a part of your skills, your experience,
            
            
            
              your life that won't apply at comes point.
            
            
            
              The trick the real lesson of this talk is that we have to commit
            
            
            
              ourselves to showing up every day ready to
            
            
            
              apply both who we are and what we know. And if
            
            
            
              we do that, there really will be no such
            
            
            
              thing as a poor technical choice.
            
            
            
              As I mentioned at the start of this talk, I can't teach you
            
            
            
              much, but I might irritate you. So if you're feeling irritated,
            
            
            
              it's time to start mulling over that grain of an idea and creating
            
            
            
              a pearl of your very own. Meanwhile,
            
            
            
              online line in the chat of those
            
            
            
              conference I am ready for your questions.
            
            
            
              Thank you very much.