Enforcing local structure over latent activations and forcing their components to conform to a cookbook of quantized vectors might result in an inconvenient alignment tax on the performance of ML models, despite making internal representations more human-friendly. Fortunately, the few techniques available todayfor improving the interpretability of ML models using conventional transparency tools (e.g. SoLU) don't seem to incur significant capability penalties, which is encouraging.
Still, it might be the case that one "sentence" expressed in the interlingua takes one person a lot of time to parse out and understand. Local structure constraints might help break down the representations into decently decoupled chunks before people reconvening to piece the puzzle. It's unclear how much this could scale, though. Imagine tasking a thousand people with reading one page of a thousand-page book, before reconvening. This runs into HCH-like conceptual limitations.
Conversely, instead of horizontally splitting out the interlingua sentence, one might imagine a hierarchical variation which contains a top-level representation before zooming in on meanings. In the hexagonal toy example from [[how-might-an-interlingua-look-like]], one might imagine having a top-level lattice, where each symbol can be zoomed into, resulting in a nested lattice. Flows of information during learning, and hence the resulting local structure, would match the links contained in the hierarchy. That said, the devil might lost be in low-level details of combinatorial size.