🌿  back

21.17 YRS

long-termist thoughtware

This is mostly a summary of a fascinating conversation I’ve had with Alex from 80,000 Hours, a non-profit organization offering one-on-one advice for people aiming to use their career as a means of improving humanity’s long-term future. If your interests and skills are close to mine, you might particularly benefit from the ideas below on how to help move the needle on global priorities. If not, I wholeheartedly recommend devouring the excellent content on their website on how to maximize your positive impact on the world career-wise.

I’m taking thoughtware here to mean the totality of tools which enhance human cognition, especially using AI – tools which help us perceive large amounts of information, navigate complex problem spaces, debug and refine our belief systems, etc.

Making progress on global challenges appears to require a wide breadth of knowledge, careful reasoning, and accurate models of complex systems. Global governance, AI safety, and climate change, as a quick pick of especially pressing global issues, all require remarkable levels of intellectual work. While specializing on one of those issues is probably the go-to way of improving the long-term future, there’s a different class of approaches you can take – capacity building. If we have more resources at our disposal for addressing long-term issues and opportunities (e.g. funding, expertise, infrastructure, etc.), then this should help on many fronts at once.

Thoughtware is all about creating artificial systems aiming to improve individual cognition, scaling all the way to collective intelligence. It’s all about capacity building on a cognitive level, which would plausibly help downstream in a myriad of ways, simultaneously. Researchers across fields need to navigate vast bodies of literature, policy makers across the board need to navigate multi-stakeholder problem spaces, and advocates need to grapple with belief systems not evolved to understand existential risk. There appears to be a large potential upside in improving our cognition, especially by focusing on general cognitive functions required in a vast array of disciplines.

There’s another reason why capacity building on this cognitive level might be effective. One of the things thoughtware might help knowledge workers become better at is… building better thoughtware. It’s plausible that by using augmented perception, memory, reasoning, etc., thoughtware engineers would be able to build tools which augment cognition even further, by using insights from cognitive science literature, underexplored regions of the problem space of thoughtware engineering, and beliefs about such tools rigorously evaluated against ideological unit tests. It’s one of those juicy recurrences which pop up in AI safety: iterated distillation, iterated amplification more broadly, recursive reward modeling, the very idea of an intelligence explosion, etc. In the best case scenario, thoughtware development feeds on itself exponentially, while outputting practical artifacts to most other knowledge workers.

Speaking of AI safety, this is probably the most widespread focus of people (1) with experience in AI, and (2) with long-termist interests, and for good reason. Is it better to focus on pure AI safety rather than thoughtware engineering if you’re in a similar position to me? Probably, though it depends on many other factors, and reaching out for advice on this would be a smart move! That said, I realized that it’s not an either-or decision. There are, in fact, many important overlaps between AI safety and thoughtware engineering. For one, they’re both interested in models of the world internalized by ML models. The superhistory perspective frames ML models as experts with thousands of years of experience which we could learn from, given the right tooling, while the microscope AI approach to AI safety argues that latent representations internalized by ML models could be AGI-level helpful without us even having to worry about misaligned agents going rogue, given the right transparency tools. They’re two sides of the same coin! Actually, Chris Olah and friends who advocate for microscope AI have been a huge inspiration for me when I started working on semantica a bit over a year ago, and it now feels like going full-circle with this long-termist involvement. Chris also has had a fascinating career path, getting to OpenAI and Google Brain without even an undergraduate degree, thanks in part to a lot of independent deliberate work.

The overlap between augmenting humans with AI and aligning AI to humans becomes increasingly obvious as you look at Ought, an organization which both develops assistive AI tools for improving human reasoning and does research on AI alignment. Paul Christiano’s latest report titled “Eliciting Latent Knowledge” comes from the new Alignment Research Center, and addresses the task of distilling human-readable information from ML models head on. It feels like AI safety people are interested in rendering ML models human-readable mostly for debugging and aligning them, while people dreaming of AI-enabled tools for thought are interested in similar techniques, but mainly for using those models and learning from them. Transparency, interpretability, and explainability tools will likely become central in thoughtware once we move beyond simply using off-the-shelf pretrained models. We’ll want a deeper integration between knowledge workers and their AI assistants – prepending transformer attention to human attention instead of running semantic search, using artificial languages evolved in hidden layers instead of passing English in and out of black boxes, interweaving semantic space with physical space beyond the antiquated desktop metaphor. Neuromancer will probably look tame, and hopefully outdated in favor of some more optimistic solarpunk.

Let’s rewind to more concrete actions available today. Going into the consulting program, I had three vague ideas of different career paths I could follow to pursue those ambitions: traditional academic research, more applied research in an industry lab, and independent research. I was quite uncertain about how much each of these paths would help improve human cognition, and still am. However, it became rather clear that traditional metrics like citations wouldn’t be the best proxies for success, especially not if Goodharting them. Industry contexts also provide financial proxies, which again feel prone to Goodharting, leading away from a hardcore computer revolution. Fortunately, pursuing a PhD appears to help on all three paths, providing useful research experience regardless of context, so I’ll be able to safely delay long-term commitments to a specific path for about 8-10 years, hopefully knowing better by then.

However, unlike the way formal education up to and including a PhD appears to provide transferable career capital across all those options, Alex helped me identify a systematic blindspot in my plans: engineering skills. Now more than ever, it seems that the border between research and engineering roles becomes increasingly ill-defined, especially around ML, meaning that such skills are highly valuable if not crucial for producing meaningful work in this space, be it in thoughtware or AI safety. To tackle this, I’ll probably focus on deliberately upskilling in ML engineering during my upcoming gap year, besides freelance work in industry. I might learn about parallel computing while training a language model on a poorman’s compute cluster. Or I might learn about data engineering while trying to look things up in some unwieldy Common Crawl dump. Or perhaps learn about container orchestration while serving a model with high availability. Or simply try to get some PRs merged in PyTorch or JAX. Or perhaps re-implement a minimal PyTorch from scratch. Andreas from Ought also put together a rough curriculum for prospective candidates, whose deployment/production content seem very useful here. There appear to be so many accessible ways in which to do this, which make it seem like determination is the main requirement going in. Also, just look at this little guy, it looks so fun!

One thing which Alex was particularly supportive about was my plan to experiment with the three career paths during my gap year. The unsurprising reasoning was that small-scale projects would help me get a better sense of the fit. For the industry research path, I’ll probably collaborate with a couple start-ups and labs I’ve been in touch with over the past months through project-based freelance work. For the academic research path, I’ll try landing a more legit research assistant role for the duration of a research project. If that doesn’t work out, I’ll probably try publishing a paper with some of the independent and fascinating internet strangers working in the field who I had the opportunity to meet. In terms of independent research, it’ll likely be business as usual with funky open source projects and write-ups. Is one year enough for all this? I have no idea, but I’m eager to find out.

It’d be a missed opportunity not to mention that if you’re looking for help with a (1-8 months) research project related to the above during the 2022-2023 academic year, be it in an industry or academic setting, please don’t hesitate to contact me. I’d love to help! Also, I’m open to temporary relocation virtually anywhere.

To wrap up, I hope those ideas help you at least ponder ways in which you could make use of your career as a means of bringing about lasting positive impact. The logical next step is, again, reading some 80,000 Hours materials – I can’t recommend them enough! In 60 minutes of one-on-one coaching, I felt I indirectly doubled the value of my network, found important opportunities for improvement, gained more big-picture clarity, and overall simply had a really pleasant experience.

I feel compelled to end this specific piece by thanking the generous people supporting me on GitHub Sponsors, who help tremendously in providing a safety net for tackling those somewhat risky plans with delayed ROI. You guys rock! Special thanks to Andreas from Ought in particular.