My 10 Favorite Pieces of Data (and Management) Content of 2022
Just like in 2021, I’ve sent over 30 newsletters in 2022, which together include hundreds of links to things that interested me at different points of the year. This is a quick summary of the things that stand out in my mind as favorites, now that the year is almost over.
To be clear, ‘favorite’ doesn’t mean that I agreed with everything in any post or video: many of these have aspects that I don’t understand, or strongly disagree with. But they have stuck with me and influenced my thinking. I hope that by highlighting them here, I’ll be able to find them more easily in 2030 when I’m thinking about what influenced me over the coming decade.
I also feel fortunate to be able to take this opportunity to reflect on the last year and what I’ve learned. I find that I’m able to identify patterns only when I take a step back. Last year, I was surprised at how much of my newsletter was devoted to management topics.
This year, when I look at my favorites, I notice a faint weighting towards questions of discovery, as opposed to management or data infrastructure concerns. How do we find things we’re looking for in a sea of information? How do we detect anomalies, or estimate missing values?
These posts are in chronological order (of when I sent them), not ranked, and some were new to me in 2022, but were written in previous years. Without further ado:
False Discovery in A/B Testing (Ron Berman, NYC Data #257).
I found this paper through one of the authors’ tweets. The paper analyzes over 2,700 online A/B tests that were run on Optimizely, and set some benchmarks for the likelihood that a tested feature would have no or minimal effect (about 70%), and the likelihood of a false positive given that class rating (it’s a lot higher than 5%, not-intuitively). I don’t know how well Optimizely tests would generalize to any particular team’s set of changes, but I’ve used the rule-of-thumb that about 2/3rds of changes don’t have positive impacts for a bit, so this definitely passes the smell test for me.
Imputing Missing Values for Categorical Attributes, at Scale (AI@Compass, NYC Data #258)
Very technical piece about a problem I’ve run across a bunch, and one of the few pieces I’ve seen with practical guidance about improving data quality when you have lots of missing or inconsistently named values. I’d only attempt this in very high-potential-value situations, since the amount of work to set this up seems pretty intensive (and the storage requirements are massive) but I can imagine situations where this would be very valuable.
The Decider App (Nobl.io, NYC Data #260)
This came up at work, and touched on something that I feel like I’ve struggled with as a manager: when should I reach consensus with my team, when should I dictate what we’re doing, and when should we put off making a decision at all. Before reading this I didn’t have any framework to think about which approach made sense in which context, other than some level of frustration when we couldn’t come to a decision, or concern about the impact of overruling (very capable) members of my team.
I’ve consulted this framework a few times over the last year, just as a gut check when I feel unsure of myself. Like all frameworks it isn’t infallible, but it’s given me a good starting point to think about how I make decisions as a manager, and how we do as a team.
You Must Try, and then You Must Ask (Matt Ringel, NYC Data #263)
Once or twice a year, I read something that has been online for many years and immediately say to myself ‘Oh man, where has this post been hiding?!’ Written in 2013, this post is a perfect distillation of how to work on a complex problem and how to effectively ask for help.
At work, we’re in the middle of reviews, and one question that comes up a lot is ‘Autonomy’. When we talk about it in a career growth context, usually we think of more autonomy as a good thing, but we don’t want people getting stuck on problems that have already been solved. This is a great piece for setting expectations around what productive autonomy looks like, especially for technical people relatively early in their careers
Hundreds of AI tools have been built to catch covid. None of them helped. (Will Douglas Heaven NYC Data #266)
We will (rightly) remember 2022 as a pivotal year in the productization of Artificial Intelligence. This summer’s explosion in the availability and power of novel image generation tools and the bombshell of ChatGPT (less than 30 days before when I’m writing this!) really feel like some sort of new chapter has been opened with this technology.
Maybe it’s my grumpy or skeptical nature, but I think it’s still worthwhile to recognize that there are also truly disappointing shortcomings in how we’re able to use data to improve our world. Covid certainly exposed these: never mind AI, basic forecasting models were, famously, all over the place. The glass half-full view might be that even in a world with phenomenal data products coming out seemingly every week, there’s still room to leave your mark.
PLUTO in 5 acts (Amanda Doyle, NYC Data #272)
I have a soft-spot for the PLUTO dataset (which covers all NYC tax lots); I used it when I taught Data Visualization at CUNY years ago. The author, who works at NYC’s Department of City Planning, takes us through the whole modernization & publishing pipeline.
I appreciate seeing how the sausage is made for these big, important datasets. The codebase (which is on GitHub) has almost a thousand commits, to give a sense of the complexity and work that went into this.
Predicting consumers’ choices in the age of the internet, AI, and almost perfect tracking (David Gal, NYC Data #277)
I saw this on Andrew Gelman’s blog, and it was the item on this list that had the biggest impact on how I thought about my work this year.
The paper itself is a bit on the dry side, but the central takeaway is worth repeating: “[P]rediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante.”
As one of my colleagues says: data often can tell us the ‘what’, but not the ‘why’. This piece inspired me to spend less time looking at the internal data that we’re able to generate further down the funnel, and spend a little more time thinking about the competitive landscape and the data that could help my employer understand these decision points.
Features are not just for Data Scientists (Josh Berry, NYC Data #280)
An entertaining story about ‘Data Democratization’ and what it actually feels like in practice (not all good!). My experience working with Data Scientists and Data Analysts has been that they are often very conflicted about other people depending on ‘their’ data. Yes, data professionals want to have an impact on organizations, but they don’t have a great sense of the failure modes of their data, and sometimes get nervous when they think that someone will actually use their data to make a decision (George Box quote, etc). The author’s conversion to seeing the value of making his data available broadly, even if it will slow down his team, is something that resonates deeply with me.
Building Airbnb Categories with ML and Human-in-the-Loop (Airbnb, NYC Data #286)
My favorite data feature of the year: Airbnb classified their inventory to give would-be vacationers a chance to see listings by ‘category’ instead of by destination, helping them discover new locations to visit and helping Airbnb spread demand across a broader supply of units.
The piece was also great, talking about all the work that went into categorization. And it’s only the first of 3: I’m looking forward to the next two!
Data Systems Tend Towards Production (Ian Macomber, NYC Data #287)
Similar to the post ‘Features are not just for Data Scientists’ that I previously mentioned, this is a case study in what happens to data systems over time, as different consumers inside the organization discover them. The author does a great job of identifying trends and giving clear suggestions about how to get the most value out of these systems.
The final warning in this piece, that ‘At some point, your data products will break the production application’, and the recommendation that goes with it, are great things to keep in mind, and to help overcome the fear of ‘messing things up’.
And of course, I can’t finish up without some honorable mentions:
Why skyscrapers are so short (Brian Potter, NYC Data #260)
What’s the most New York thing that’s ever happened to you? (Dan Saltzstein, NYC Data #261)
Bayesian Rock Climbing Rankings (Ethan Rosenthal, NYC Data #268)
What if every dashboard self destructed (Randy Au, NYC Data #281)
Measuring Search Relevance, Part 2: nDCG Deep Dive (Audrey Lorberfeld, NYC Data #282)
Some notes on the Stable Diffusion safety filter (Vicki Boykis, NYC Data #286)
Thanks so much for reading, and for being a newsletter subscriber (if you are one: if not, I’m amazed you made it this far and you can subscribe here). Happy Holidays, and best wishes to you and yours in 2023!