I updated the DNS haiku, y'all:
everyone forgot
about the certificates
they changed and things broke
#certificates #devops #softwareengineering
Remote
drmorr
@drmorr@hachyderm.io
mastodon 4.7.3Optimizer. Photographer. In disbelief that computers actually work. Accidentally writing a role-playing game. (he/him)
303 Followers
185 Following
41 Posts
Joined January 26, 2023
GitHub:
Company:
man I'm real tired of literally everything not working right now
Replying to
@mattblaze@federate.social the other difference is that nobody talks on the phone anymore. I went all day as a T-Mobile customer and had no idea they were having any sort of outage.
In case you missed it, this is my SRECon recap, featuring all of my favorite talks!
Copy fail has really brought the armchair security experts out of the woodwork
This is a post by Anastasjia Uspenski about how the "myth of the 10x engineer" is actively harmful (the post is itself based on a talk by @mipsytipsy@hachyderm.io at the Craft Conference in Budapest):
https://shiftmag.dev/10x-engineers-charity-majors-5755/?ref=cloudnative.now
"If there is a single non-negotiable truth in Charity’s worldview, it’s this: shipping is how engineers learn. Not architecture diagrams. Not months of planning. Not 400-comment design docs. Shipping."
Replying to
@Migueldeicaza@mastodon.social lol it got half of the things incorrect, despite me not taking any particularly extreme measures to protect anything.
Replying to
I stayed up late to (finally) finish stranger things season 5. "it'll be just after midnight," i told myself. "not that much later than I normally stay up, it'll be fine."
I did not factor in the extra thirty minutes I would need to stop bawling my eyes out 😭😭😭
I believe.
I guess claude has a dependency on github.
Replying to
"No one ever made a decision because of a number. They need a story" Daniel Kahneman.
#srecon
Replying to
@b0rk@social.jvns.ca (I'm really sorry, I was making a 12yo joke 💩 , I'll get out of your mentions now)
Gonna liveblog @Rejekts@hachyderm.io over here! Follow along if you want.
Replying to
We're discussing how this applies to the Therac-25 story.
The simple story is that "it's about race conditions", but the real story is a lot more complicated.
One example: the machine would frequently error and the errors were harmless, and in this case the error meant "the patient's dose is too high" but the operators didn't have that info.
#srecon
Replying to
Another story, told by Edward Tufte (of Visual Display of Quantitative Information fame) said this was about "poor information presentation and bad visualization".
Not surprising that an info-vis guy said it was an info-vis problem 🤣
#srecon
Replying to
"Stories" stick in your head, which is helpful for getting information into the heads of people who need it.
#srecon
Replying to
When something negative, surprising, or unexpected (like an incident) happened, the human response is "How could this bad thing have happened????????"
Stories are important because they are a tool that humans have developed to make sense of senseless events.
#srecon
Replying to
- The story needs to be anomalous -- something in the story needs to disrupt your mental model of the world. There needs to be a disconnect between your belief and reality. Example: "This should never happen!" which means "I never expected that to happen!"
Replying to
How to get better at storytelling? Lots of practice! If you're involved in incidents you'll have lots of practice, and you can tell them to people at SRECon :D
#srecon
Replying to
"So what, Lorin, you told us all this stuff, what am I supposed to do with it?"
One take-away: when you do your incident writeups, tell it as a story.
#srecon
Replying to
The incident prevention grand challenge is "How do you get the right information into the heads of the right people at the exact moment when they need it?"
#srecon
Replying to
(Is this a good time to plug ACRL's one public postmortem? 🤣 🤣 🤣 🤣 🤣)
https://blog.appliedcomputing.io/p/postmortem-intermittent-failure-in
#srecon
Replying to
A third story told by Diane Vaughan said that challenger was a story about normalization of deviance.
If you tuned a noisy alert, that is normalization of deviance: if the alert is firing, but the system is healthy, we make the alert less noisy! We do this all the time.
#srecon
Replying to
The details of a story depend a lot on the perspective of the storyteller.
Example: the Challenger disaster. Feynman wrote an appendix to the "official" synopsis of the accident. He said this was a story about "management underestimating risks"
#srecon
Replying to
When it comes to incidents, there are different kinds or styles of stories you can tell:
- "The horror story": we failed over and the problem followed us to the new region! 😱
- "The morality tale": the engineer ignored the failing test and the bad code made it to production 😡
Replying to
"Only a fool learns from his own mistakes, a wise man learns from the mistakes of others" -- unknown
We've also talked a lot about vicarious learning, e.g., learning from watching or listening to someone else.
Example: Shoulder-surfing while someone is looking up dashboards.
#srecon
Replying to
- The story needs to be immutable (this is kindof a weird term to use): important details are preserved as the story gets passed on.
Replying to
@adrian@discuss.systems yea, will do it again next week I think. Or maybe I'll try shepherder's bread again.
Replying to
@adrian@discuss.systems yesterday I made challah, it was going to be really good, and then I popped it in the oven, didn't set a timer, and promptly forgot about it for the next hour and a half 😭😭😭😭
Replying to
We're comparing "nursing" to "SRE" -- for example, a patient is deteriorating but it isn't showing up in their vitals yet. In engineering, this is like the system is in bad shape but no alerts have fired yet.
#srecon
Replying to
Claim: good stories have two properties to be "useful" for some nebulous definition of "useful"
#srecon
