I plan on using my bachelor-to-master gap year (2022-2023) as a pilot project in self-guided learning. It will involve a year-long curriculum focused on upskilling in ML engineering and AI safety, complete with modules and tracks. For friends and family, this page should reply to the question of what the heck I’m doing with my life this year. For future employers, it should lend support to the idea that I enjoy learning new things. For everybody else, it might motivate you to consider running similar projects yourself, because it’s unexpectedly fun, especially the planning!
The curriculum is structured in tracks: engineering, theory, research, and hobby. Each track consists of a sequence of specific modules. Tracks provide an intermediate level in organization, halfway between individual modules and the complete curriculum. Tracks don’t necessarily have to switch modules at the same time, as periods and semesters often to in formal education – it’s difficult to perfectly predict how long those things take. However, in order not to get overwhelmed and chaotic, each track follows one module at a time (except for the hobby one). I have no idea how long each track will take or whether I’ll finish it before the gap year ends, yet I’m content with whatever it is I’ll manage. Currently, each module below lists a brief learning objective, but I’ll create individual pages for them as the year progresses.
|engineering||cloud computing||Set up a personal cloud based on a Kubernetes cluster weaved together in a mesh VPN. Deploy various third-party services on it (e.g. Nextcloud).|
|distributed analytics||Set up a loose replica of Pinecone, a distributed vector database which enables both vector search and traditional metadata queries. It should be able to handle multiple collections (e.g. papers, books, notes, etc.) and make use of tools like Spark.|
|distributed inference||Set up a loose replica of the Hugging Face Inference API on the previously-deployed cluster. Implement GPU acceleration, model pinning, caching, batching, and logging.|
|distributed training||Fine-tune the homebrew GPT-2 or BERT (see language models under theory) using different types of parallelism. Should make use of ephemeral vast.ai nodes attached to the previously-deployed cluster, perhaps via Hivemind.|
|data-driven documents||Implement three interactive articles using D3.js. Should help convey an intuitive explanation of abstract concepts in ML or AI safety.|
|theory||kaleidoscope mathematics||Consists in a refresher on the fundemantals of linear algebra, multivariable calculus, probability theory, and dynamical systems. However, the main intended outcome is a timeless flashcard deck to help drill down important theorems from multiple angles, building fluency with the objects.|
|computational graphs||Implement a minimal version of PyTorch from scratch. Should cover tensors, automatic differentiation, and full networks. Next, complete this exercise set in JAX.|
|language models||Implement BERT and GPT-2 from scratch, either using homebrew PyTorch or JAX. Load the weights of the original models and use them for inference. Implement a few decoding strategies for GPT-2. Inspired by this curriculum.|
|deep reinforcement learning||Watch the RL Lecture Series offered by DeepMind and UCL, especially the second half. Read through the Spinning Up in Deep RL guide by OpenAI, before tackling the exercise sets and implementing the suggested algorithms from scratch (e.g. DQN, PPO).|
|AI safety||Read through the resources linked in the alignment curriculum put together as part of the AGI Safety Fundamentals course offered by EA Cambridge. Possibly also this and this reading lists. Should result in at least three distillations, possibly cross-posted on the alignment forum.|
|research||N/A||Conduct research into synergies between human and artificial thought, resulting in interactive demos and write-ups. Will probably inform just-in-time learning goals beyond the fundamentals above.|
|hobby||drawing||Follow this superb course on the topic. Should result in ten pieces of concept art depicting futurist interfaces and, naturally, cute robots.|
|chinese||Follow the Refold methodology for language acquisition and related Mandarin resources. Should result in an HSK 4 certificate.|
|driving||What it says on the tin. Should result in not running anyone over, I guess.|
Soon after I started considering this, I realized that curriculum design is not as straight-forward as it might seem. This felt especially true after spending all my life in formal education, where important parts of the curriculum are predefined – presumably to help you focus on what matters™. Just like using managed cloud services led me to have no idea how container orchestration works, outsourcing curriculum design to a university led me to be overwhelmed when tackling the task myself. Outside the comfort of external scheduling, sandbox assignments, and delegated evaluation, you take full responsibility for your training. It’s full of learning with and from others, yet it’s you who decides why and what you’re learning.
Now, of course you can probably spend a decade on most of these topics and still have a lot of things to learn about them. What I’m going for here is a Pareto-heavy strategy in which I’m taking advantage of the fact that when you’re a newbie in a field, you learn new things at the highest possible rate. I don’t expect to become an expert in any of these topics in a year, but I expect to become significantly better at all of them than I was at the beginning.
I selected modules based roughly on a mix of the following heuristics. Most modules have a combination of them as motivation, but I listed some representative examples:
- What foundational concepts would I find important to nail? (e.g. automatic differentiation in computational graphs)
- What skills would I find useful to apply in my work? (e.g. putting together explorable explanations in data-driven documents)
- What artifacts would I find useful to have? (e.g. a self-hosted Google Scholar replica in distributed analytics? Yes, please!)
- If I’d be recruiting for an ML engineer or research engineer role, what skills would I find most valuable in prospective candidates? (e.g. multi-node multi-GPU training in distributed training)
- If I were to take on a meaningful research project as a one-man-research-group (bad idea, but useful as heuristic), what skills would I be missing? (e.g. awareness of the research landscape in trends in language modeling)
Besides wanting to upskill in ML engineering and AI safety, another reason for taking up all this was that I’ll likely consider some form of homeschooling a few eons into the future. From what I’ve heard, you can never be too prepared for it. This ties nicely with the fact that many tools I’m experimenting with are designed to help users learn effectively. What better incentive for building those than deschooling society?
Besides, I’d like to make a name for myself based on what I can actually create, rather than based on a brand affiliation. I feel that relying on a certain degree or title might make me more comfortable and complacent than I’d like to be. Going forward, I think I’d be more content knowing that I can live up to my expectations without invoking a recognized name to fill in the gaps.
Why not just try getting involved in an AI safety research group looking for ML engineers, learn things that way? First, there’s a chicken-egg problem. I currently lack familiarity with AI safety paradigms and decent ML engineering skills, especially when it comes to handling large models or datasets. This makes it hard to get into those settings, which makes this approach less feasible. A hyperfocused curriculum seems to me a more reliable and directed way of acquainting myself with those topics, before getting involved more heavily with those orgs later on. Second, I feel there’s stronger specialization as you move to more established research groups. At this stage, I want to get a broad sense of the whole range of tasks involved in research engineering in this space. Third, I really want to hone those self-guided learning skills and take full responsability for my learning process.
Thanks to the people who spared a bit of their time to offer feedback on those plans, including Alex Lawsen, Andy Jones, Luke Stebbing, and pseudonymous folks on the EleutherAI, ML Collective, and vast.ai servers. I’m also extremely grateful for the people supporting my work on GitHub Sponsors, especially Andreas Stuhlmüller. Finally, the constant support from my SO has been instrumental in getting this whole thing going.