🌿  back

21.25 YRS

representational resources

This one’s also adapted from an essay assignment, though I think it fits quite nicely with the themes I cover on the blog, riffing on many ideas related to the conceptarium. Also, I enabled a comment section at the bottom of the page on reflections and thoughtware write-ups via this funky tool! The intention was mainly to create a friendlier space for sharing constructive feedback, hot takes, and related resources. You can edit, subscribe to, or react to particular comment threads as you would with particular issues on GitHub. I’ll make sure to answer questions you might have about articles and projects in there, so feel free to try it out.

Evidence suggests that complex working memory capacity (CWMC) is one of the most reliable predictors of problem solving abilities. Students who find it difficult to process multiple fragments of information at the same time in an experimental setting consistently face difficulties in solving problems in real-life academic settings. Unfortunately, training programs for increasing an individual’s CWMC remain the subject of much controversy, with scarce evidence available for backing the bold claims of effectiveness around commercial apps. Given the shortcomings of the “memory as a muscle to be trained” metaphor when tested against real-world evidence, a remaining avenue for increasing the effective CWMC of an individual consists in making use of external memory aids. Digital tools specifically designed to extend one’s CWMC through active use, rather than through prior training, appear to provide promising opportunities for improving problem solving skills on a large scale.

For one, it has been argued that conscious awareness, as closely-yet-non-trivially related to working memory, relies on a process of representational resource allocation. Information which is consciously processed by the individual is then assumed to be represented in a neural substrate. Crucially, the representational resources available for active processing are deemed to be limited by physiological and metabolical constraints, prompting the individual to engage in allocating such resources strategically. In this context, even crude physical devices can help by offloading representational tasks – a physical sketchpad can trivially represent information across time as written symbols, extending the visual sketchpad described in early models of working memory into the external realm. This presumably frees important amounts of representational resources from the individual’s working memory, by only requiring source memory for sketched items – knowledge about what is located where, for subsequent access if required.

What’s more, the interaction between the native representational medium and the artificial one might become more seamless through the use of user models. If a digital sketchpad had access to long-term memories held by the individual, it could attempt to surface and visually represent them directly, without the person having to interrupt their thought process in order to pass the fragments of information through and offload them manually. Naturally, an accurate user model would result in better outcomes. For instance, if the user model included ACT-R-like activation estimates for different fragments of knowledge (i.e. similar to commercial apps like SlimStampen, SuperMemo, Anki), together with AI models to understand the user’s context and intent, the external memory aid could attempt to “echo” the individual’s working memory contents from the very beginning of the human-computer interaction, providing a fertile starting ground for engaging in problem solving.

However, a pure “mirroring” of the user’s working memory might not be immediately helpful for a boost in representational resources. The most salient items surfaced on the digital sketchpad might roughly coincide with the actual contents of the individual’s working memory, failing to make use of the additional resources due to redundancy. Deduplication on the digital aid’s side might happen by not surfacing the very items which the user is particularly likely to remember by themselves (i.e. based on activation). This way, the external representational medium would automatically fill up with the most relevant items which are likely to reside immediately outside of the individual’s working memory, due to resource constraints. The external representational medium would essentially provide an artificial prosthetic around the fringes of organic memory, adapting in such a way as to minimize redundancy while maximizing relevance.

However, critics of the alleged effects of CWMC on problem solving abilities point at the underestimated importance of being able to populate working memory with the right information given the context. This perspective argues that increasing the signal-to-noise ratio for the available representational resources is as important as the actual amount of resources – the representational “accounting” strategy might be critical. Following the opportunities offered by the previous interplay between organic and artificial representational media, it is natural to now consider whether external systems could help people act rationally in their representational resource allocation. Can AI models find items of higher relevance to the current problem solving episode than the actual person in which the information resides organically?

Advocates of rational analysis, the school of thought which supports the idea that memory is evolutionary well-adapted to its environment, might argue against this possibility. Native heuristics for representational resource allocation – favoring information of frequent, long-lasting, and personal relevance – might already provide a close-to-rational accounting policy. However, even if this were the case, it would at most be a bounded-rational policy, implementing the most effective strategies available for maximizing signal-to-noise ratio in recall given the again limited constraints of human physiology and metabolism. If the strategist who coordinated the resource allocation had extensive computational resources at their disposal, their policies might be even more effective than native human heuristics.

What’s more, the current discussion leaves out the possibility of attaching whole new bodies of long-term memories to a person as prosthetic experience. In many cases, the most relevant fragments of knowledge for solving a problem might happen to reside in other people’s long-term memories. Artificial systems capable of navigating such an overwhelming amount of information might further increase the signal-to-noise ratio of the contents of one’s working memory. This is arguably already happening through large-scale Internet search engines, yet paradigms for interfacing with this body of knowledge in more cognitively ergonomic ways could take it even further.

To sum up, literature hints at both theoretical and experimental reasons to be hopeful about extending cognitive function through external memory aids. Progress towards those ambitions would no doubt require significant breakthroughs in cognitive modeling, AI, HCI, and a host of other fields, yet the potential upside makes the endeavor appear worthwhile.