Kendra Vant on data & AI in 2024 and beyond
"We have to take LLMs and all the other useful tools we’ve come up with over the last 40 years and apply them to usefully and cost effectively solve specific important problems."
A bonus post for the holiday season! Merry Christmas to all those who are celebrating today (or tomorrow if that’s the side of the date line you fall on). It’s quite a special Christmas for me as I have my parents (over from New Zealand!) and all three children at the farm. Two of mine are off overseas next year so it could be the most densely ‘family’ Christmas for a while!
To my long time readers, thank you for your support and suggestions throughout the year. It has been a pleasure writing and learning with you.
A big welcome to all my new readers, it’s fantastic to have you here and I look forward to your thoughts, comments and topics suggestions throughout 2025 and beyond.
Yes, I’m answering my own questions below. Yes that is quite meta.
Kendra, 2024 has been another busy year for data & AI. What’s one development / milestone / news story that really caught your eye?
As long time readers will remember, I was really quite taken with LearnLM from Google DeepMind. I’ve loved the depth and intensity of focus the team is putting into the problem of how you might actually turn a large language model into a useful educational support for students and teachers around the world. There is a lot of noise in the EduTech sector - both from supporters and naysayers of Generative AI but I’ve found many of the available solutions inch deep. A standard chatbot is not a teacher. Obvious when you remember that teachers teach us to think and to figure things out - they don’t tell us the answers. I’ll link to the full blog below for those who missed it and would like to have a deeper dive, but what I loved to bits was the focus on evaluation - how will we quantitatively tell if we’re building a ‘better’ AI teacher? - and the warts and all nature of how they shared what worked and what didn’t.
You’ve been working in and around data & AI for a while now. Many things have changed! But tell us about something that was true when you started out in this space and is still important today.
You still have to actually understand the problem you are trying to solve. It is never as simple as you first think it is and you are going to get it wrong many times on the way to right. This is actually the main reason I have hired a fair few PhD qualified scientists over the years - not because they have an advanced degree in algorithm development (I don’t, mine is in experimental physics) but because completing a doctoral degree in an experimental field is really hard and you don’t usually get out the other side unless you get good at breaking down the hairy amorphous problem in front of you into something you can explain inside and out.
The next five years are going to be really interesting. We have to take large language models, diffusion models, rerankers, BM25 and all the other really useful tools we’ve come up with over the last 40 years and apply them to usefully and cost effectively solving specific and important problems. It should be fun! It won’t be easy.
It’s been a heady couple of years with 2024 almost as frothy as 2023. What's one common misconception about AI that you wish would go away?
That AI will ‘break loose’ and start working ‘for itself’ against humanity. It’s not the most common misconception but I do think it is one of the most dangerous because it distracts us, and when I say us, I mostly mean policy makers, from the much more likely-to-occur risk:
What all of us fallible humans might wander into doing with AI if we don’t think very, very carefully about how sharp a pair of scissors we may just have started running with.
To be clear, I’m not jumping on the ‘regulate the peta flops!’ bandwagon. I’m saying that we need to look at this from a whole-of-economy perspective and make some considered choices about what quality of work looks like for all kinds of humans.
The festive season is almost upon us, so many readers will have a bit of extra time to read / learn / reflect. Who do you follow to stay up to date with what’s changing in the world of data & AI?
Other folks have already mentioned many of the people I follow so I’ll only put in additions and limit myself to a few.
Still enjoying reading AI Snake Oil by Arvind Narayanan and Sayash Kapoor, their latest article on scaling is enjoyable and well reasoned as ever.
I also really respect the research output of Emily Bender. I’ve recommended Bender’s two papers written with Chirag Shah - Situating Search and Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web? before. Take some time these holidays and read them in full.
OK if you still want the summary version, I wrote about my key takeaways from both papers in March.
Leaning into your dystopian side for a moment, what’s your biggest fear for/with/from AI in 2025?
That we don’t figure out an equitable economic model for this hot mess of brilliance, greed, energy consumption and labour saving tech.
It seems entirely possible that we stumble into a future where we have companies more powerful than countries, poorly paid humans stitching together the bits of actual jobs that machines can’t be made to do yet and where the well off consult human doctors while the rest of the population asks an AI that simultaneously harvests their biological data to build better diagnostics. Hang on, isn’t there a movie about this stuff?
You did say dystopian side!
And now channeling your inner optimist, what’s one thing you hope to see for/with/from AI in 2025?
That we figure out how to make Generative AI less environmentally costly
I’m glad to already see smaller, and hence more resource efficient, models becoming popular. I hope this trend accelerates in 2025 so we don’t have to choose between using LLMs and reducing GHGs. I’m optimistic because it is standard practice to throw everything you can think of at trying to get a model to work and then subsequently, as you understand it better, being able to prune back a lot of the weight and complexity.
And remember, if you build software for a living, you can help out by following the carbon minimisation principles set out by the Green Software Foundation. (Go on, you know you want to know what embodied carbon actually is …)
And one more neat thing
Finally, I was super interested to see Google DeepMind come out with their FACTS Grounding benchmark a few days ago: “Evaluating the ability of LLMs to generate responses that are not only factually accurate with respect to given inputs, but also sufficiently detailed to provide satisfactory answers to user queries.”
Is the timing of release relative to the new Gemini Flash models and their appearance at the top of the leaderboard a wee bit suspicious? Well yes, but it’s such a good idea that it still makes me happy.
You know where you can follow me. Instead a shout out to the amazing Gemma Carr Photography where I just had my professional photos and headshots refreshed.