The term 'sensory substitution' refers to the use of a sensory modality to supply environmental information normally gathered by another sense (Auvray and Myin 2009; Auvray and Farina, 2017). Sensory substitution devices (SSDs) thus provide through one sensory modality (the substituting modality) access to features of the world that are generally experienced through another sensory modality (the substituted modality). Since the early '60s, scientists have striven to create sensory substitution and augmentation devices capable of compensating for, replacing, enhancing or creating a sense by exploiting another sense. This research has produced some fascinating experimental results, which have furthered our understanding of the human brain and of the neural mechanisms behind visual restoration.
Fiona Macpherson brings together some of the world's leading researchers in the field to explore: (i) the idea that SSDs can offer a window into the study of brain organisation, (ii) the prospects and possibilities that these technologies offer for rehabilitation as well as (iii) their unfulfilled promises and limitations, and (iv) the nature of the perceptual experience associated with their use. The book contains a lengthy, extremely helpful introduction written by Macpherson and sixteen essays by philosophers, psychologists, and neuroscientists who look at the nature of SSDs, how they work, the sort of mental states they cause in people using them, and their potential for practical use (p.xix). The collection's main thesis is that research on SSDs is still in its infancy (p.xx) and research on it is immensely attractive because it can 'potentially transform for the better the lives of those who have lost a sense' (p.2).
I'll briefly summarise the book's contents and then consider an unexplored link in the book: the relation between SSDs and the Extended Mind Thesis (EMT). I investigate this relation and argue that the active coupling with an SSD can be taken as a paradigmatic example of EMT, so as to provide support for the idea that the cognitive processes that make up our minds can reach beyond the traditionally conceived boundaries of individual organisms to include as proper parts aspects of the organism's physical, technological and socio-cultural environment (Kiverstein, Farina, and Clark 2013).
Macpherson's introduction has five major goals: i) to describe sensory substitution and augmentation devices; ii) to discuss the kinds of perceptual experiences that the coupling with these devices give rise to; iii) to examine which account of perception is best supported by SSDs; iv) to analyse limits and potentialities of these devices; and v) to review some practical uses of these technologies. Macpherson meets these goals and convincingly makes the case that SSD perception (in trained users at least) belongs neither to the substituted nor to the substituting modality but rather is something unique, 'somewhat like normal human vision and somewhat like normal human touch' (p.21). This is a claim I myself have endorsed (Farina, 2013). Two minor points of criticism: a) in the discussion of the modality involved in SSD perception, she could have mentioned that according to some researchers (e.g. Nanay 2017), SSD usage is a special form of mental imagery triggered by corresponding sensory stimulation in a different sense modality ('multimodal mental imagery'); b) in the discussion of current limitations of SSDs, I was surprised not to see reference to the work of Jack Loomis (Loomis et al., 2010, for example).
Laurent Renier provides a comprehensive review of recent progress on SSDs and argues that these devices give proficient users experiences that are in the substituted modality. Renier also shows that SSDs are demonstrably suitable for many tasks (including spatial localisation, distance estimation, and shape recognition) and that while many technological hurdles remain before these devices can be adopted in daily life on a large scale, they have offered very valuable contributions to the understanding of human perception, brain plasticity, and sensory interactions.
Barry C. Smith briefly but critically responds to Renier, arguing that in SSD perception the substituting modality takes on some of the functions of the substituted modality, therefore calling into question the idea that SSDs give their users an experience that has the phenomenal character of an experience in the substituted modality' (p.65).
Jennifer Corns critically focuses on Ophelia Deroy and Malika Auvray's (2012) rejection of the perceptual assumption, which implies adopting a model of SSD perception 'that represents the vertical integration of novel cognitive skills onto pre-existing sensory pathways' (the reading model in Deroy and Auvray's view). She argues that such a model (despite having provided a wealth of precious insights) can still be perfected.
Paul Noordhof, drawing from relationalist theories of phenomenal properties, offers an original account of SSD perception, where (following proper acclimatization to the device) properties proprietary to a sense modality can be presented in another modality.
Sarah Hillenbrand, Dina Raveh and Arnir Amedi discuss how SSDs can be used as a window to study the organisation of the human brain. The researchers also investigate the nature of brain plasticity and cross-modal connections in sensory substitution and look at the prospects of these devices for rehabilitation. One of the central ideas underlying this chapter is that the human brain is a mosaic of meta-modal operators. This idea is based on Alvaro Pascual-Leone and Roy Hamilton's (2001) thesis that we should understand the sensory cortex as 'meta-modal'. On a meta-modal view the brain is constituted of operators that execute a given function or computation regardless of sensory input modality. This view thus suggests that the sensory cortex exhibits functional specialisation irrespective of the nature of the inputs that are projecting to this area. The chapter provides an excellent review of neuroscientific findings supporting this idea. However, and this is only a minor point of criticism, it could have linked and contrasted the presentation of these findings with a more traditional idea of sensory processing, one that, for example, goes back to Johannes Müller and his law of specific nerve energies (Müller 1843/2003). Müller believed that sensory differences are determined by the specific anatomical regions in the brain to which the nerve fibres lead. So, on Müller's view, visual perception is the result of nerve fibres that connect the retina to the visual areas of the brain. The discovery of cross-modal plasticity and the idea of a meta-modal brain strongly undermine Müller's traditional account of sensory processing. A discussion of this point would have grounded the chapter historically and improved its dialectic substantially.
Derek Brown critically responds to Hillenbrand et al., arguing that SSDs may facilitate behavioural transference of skills across sense modalities and that 'these skills are therefore a-modally represented' (p.38).
Maurice Ptito, Katherine Iversen, Malika Auvray, Ophelia Deroy, and Ron Kupers provide a comprehensive review of current SSDs, helpfully discuss why cross-modal plasticity matters in SSD perception, and highlight a functionalist bias affecting behavioural research on SSDs. Together with chapter 6, this essay provides crucial insights for understanding the architecture of the human brain and how certain areas can be recruited via cortical plasticity. The chapter's central thesis is that through the coupling with an SSD a blind user may get a taste of vision (a claim made by the authors elsewhere too: Ptito, Chebat, and Kupers 2008). I'm sceptical of the authors' claim, for a number of reasons. SSDs -- at present -- do not provide their users with colours; neither do they endow them with the capacity smoothly to track objects, or the ability fully to perceive depth. Through the coupling with an SSD, an experienced user can therefore at best acquire only a few aspects of normal vision but 'the image-like quality of vision still seems far away' (O'Regan 2011, pp.142-143). In addition, and more importantly perhaps, evidence that our brains are meta-modal engines suggests that SSDs may unmask a latent potential in the sensory cortex to process stimuli regardless of sensory modality. This in turn suggests that proficient SSD users may acquire, through the coupling with the device, a brand-new phenomenal way of experiencing the world, which in part resembles vision and in part resembles touch (see Kiverstein, Farina, and Clark 2015; Farina 2013).
Michael J. Proulx, David J. Brown, and Achille Pasqualotto focus on what spatial navigation can tell us about SSD perception. They argue that insights from spatial navigation can help us compare different sensory substitution systems, assess their practical usefulness, and even offer new important intuitions for theories of cognition.
J. Kevin O'Regan defends a sensory motor account of SSD perception, according to which the nature of the perception obtained through the 'active' coupling with these devices undergoes a change from the substituting to the substituted modality. However, the reason for this experiential change, it is argued, is not cortical plasticity itself (as often claimed) but rather the change in modes of interaction between the agent and the environment that it allows. I sympathise with O'Regan's sensory motor account of SSD perception and think it can be used to support the idea that SSDs count as case studies for EMT (more on this below).
Robert Briscoe critically responds to O'Regan by showing that even if we accord active usage a decisive role in SSD perception, we still have good reasons to resist a sensorimotor story and we may just interpret distal attribution as a solution to a causal inference problem faced by the subject's perceptual systems.
Thomas D. Wright and Jamie Ward provide an original taxonomy of sensory tools (on two scales and four categories) and in light of this taxonomy go on to offer a new definition of sensory substitution.
Jonathan Cohen distinguishes between two types of SSDs: those that provide raw information to their users and those that provide higher-level emergent features. He argues that more devices belonging to the latter category should be built and lays out a programmatic vision of how the design of these devices should proceed.
Drawing from empirical findings in psychology (on latent inhibition and attentional weighting), Kevin Connolly lays out a novel way for SSDs researchers to avoid prior exposure and more meaningfully answer the philosophical question regarding the nature of SSD perception.
Charles Spence helpfully highlights fundamental limitations of SSDs that have prevented their adoption amongst individuals suffering from sensory loss. These limitations include the impossibility to fully substitute for vision given the imbalance in neural cortical resources given to processing information in various senses; but also lie in the inability of current SSDs to deliver hedonic attributes to their users. He also notices severe technological shortcomings and aesthetic concerns about the appearance of users wearing these devices as factors affecting the uptake of these sensory tools in society. The existence of all these limitations prompts Spence to depict a very negative picture of the prospects of SSDs research and of its potential for people affected by sensory loss. This is an excellent chapter that does a great job in pointing out the current limitations affecting research on SSDs but also its unfulfilled promises. I agree with much of the criticism raised by Spence but am somehow less pessimistic. Specifically, I believe that many of those shortcomings (especially those inherent to technological developments and design) will be overcome in the future. Those shortcomings that, according to Spence, are insurmountable (e.g. impossibility to fully substitute for vision), seem, on the other hand, to confirm my thesis that SSD perception in experienced users belong to a new sensory modality.
David Suarez, Diana Acosta-Navas, Umut Baysan, and Kevin Connolly analyse the lack of positive emotional experiences that SSDs users often report and suggest a practical way to overcome this problem.
Jérôme Dokic, like Suarez and colleagues, also looks at the emotional and hedonic experiences that characterise human perception and notices that these are typically constituted by noetic feelings of familiarity and presence. Unlike Spence, however, he develops a more optimistic view, according to which such feelings can be enjoyed by proficient SSDs users.
Next I will briefly discuss a promising link not explored in the volume: the relation between SSDs and EMT. I claim that active SSD usage provides an empirical confirmation for EMT.
As we have seen above, EMT (Clark and Chalmers 1998a) paints the mind (or better, the physical machinery that realises some of our cognitive processes and mental states) as, under humanly attainable conditions, extending beyond the bounds of skin and skull (Kiverstein, Farina, and Clark 2013, p.1; Clark 2008). EMT is usually depicted as flowing naturally from functionalist views concerning the 'multiple realizability' of cognitive processes (Wheeler 2010). One strand of argument for EMT therefore invokes the so-called "parity principle". However, a second strand of research on EMT emphasises considerations of "complementarity". Complementarity approaches (Clark 1998b; Sutton 2010) investigate the many different ways in which diverse components (neural and extra-neural) of a cognitive system intermingle and function together in forging a novel unit of cognitive analysis and complex cognitive behaviours that wouldn't otherwise be experienced by the user's brain on its own. In what follows, I argue that active usage leads to such behaviours and that it can therefore confirm EMT.
Notably, there are two cases of SSD perception. Cases of passive use (Bach-y-Rita's chair) where the user does not move or experience the world with the devices, and cases of active usage (TDU, the vOICe etc) where the user actively uses the device to explore the surrounding environment (O'Regan, above). In the case of active usage, two mappings are being performed when the perceiver actively uses the device. The first is the mapping that is generated by the device itself. This mapping is shared by both active and inactive users. The device at this stage physically maps the environment onto either auditory or tactile inputs. As a result of this mapping activity, the user can either get something that vaguely resembles a tactile stimulation or an auditory one. Upon receiving this crucial information from the device, the perceiver's brain finds in the auditory/tactile stream specific sensorimotor contingencies (pattern of dependencies that hold between the perceiver and its movements (O'Regan and Noe 2001) that characterise perception. This is where the second mapping begins to work. The sensorimotor contingencies that become available to the proficient user inform the perceiver about things that are out there, in the external environment. The brain now starts processing auditory or tactile inputs and gradually transforms these inputs so that they can be understood as being about stuff out there. The device is therefore transforming environmental inputs into outputs that the brain re-cycles and re-elaborates in conjunction with the SSD. Crucially, the outputs the brain is producing are device-involving. These outputs in fact contribute to create a new space of biotechnological synthesis between the user and the world and therefore allow for incorporation of the device (Kiverstein and Farina 2012). This partnership enables cognitive transformations that gradually allow the device to become fully transparent and be part of the machinery that instantiate cognition, leading the user to experience new phenomenal occurrences.
I believe this is important and needed because it offers an interesting way to further and expand the book's scope and contents towards new domains and directions in the philosophy of cognitive science.
The volume is lucidly written. It does not shy away from conceptual complexity -- quite the opposite; it displays analytical rigour and eloquence of argument. This is a book to be discovered, one whose margins should not remain blank but rather be filled with many doodlings and scribblings. Impressive both in scope and depth, it is vital reading for those (especially philosophers, psychologists, and neuroscientists) who want to know more about sensory substitution and augmentation devices, their philosophical and neurobiological underpinnings, and about the interdisciplinary study of perception and action.
The attention to empirical details as well as to theoretical (mostly philosophical) issues make this collection a complete and authoritative contribution that will certainly have a long life as a definitive account of the subject.
ACKNOWLEDGEMENTS
I would like to express my appreciation to the British Academy for the Humanities and the Social Sciences, to the Russian Foundation of Science (Project No. 18‒18‒00222), and to King's College London for financing my research. A special thanks goes to David Papineau and Anastasia Gutting for stimulating feedback on earlier drafts of this manuscript.
REFERENCES
Auvray, M., & Farina, M. (2017). "Patrolling the boundaries of synaesthesia: a critical appraisal of transient and artificially-acquired forms of synaesthetic experiences". In Deroy O. (Ed.). 'Sensory Blending: On Synaesthesia and Related Phenomena', (pp.248-274). Oxford, UK: Oxford University Press.
Auvray, M., & Myin, E. (2009). Perception with compensatory devices: From sensory substitution to sensorimotor extension. Cognitive Science, 33(6), 1036-1058.
Clark, A. (2008). Supersizing the Mind: Embodiment, Action, and Cognitive Extension. Oxford, UK: Oxford University Press.
Clark, A., & Chalmers, D. (1998a). The Extended Mind. Analysis, 58(1), 7-19.
Clark, A. (1998b). Being there: Putting Brain, Body, and World Together Again. Cambridge, MA: MIT press.
Deroy, O., & Auvray, M. (2012). Reading the world through the skin and ears: A new perspective on sensory substitution. Frontiers in psychology, 3, 457.
Kiverstein, J., Farina, M., & Clark, A. (2015). "Substituting the Senses". In M. Matthen (Ed.). The Oxford Handbook of the Philosophy of Perception, (pp.659-678), Oxford, UK: Oxford University Press.
Kiverstein, J., Farina, M., & Clark, A. (2013). The Extended Mind Thesis. Oxford, UK: Oxford University Press.
Kiverstein, J. & Farina, M. (2012), "Do Sensory Substitution Devices Extend the Conscious Mind?" in F. Paglieri (Ed.). Consciousness in interaction: the role of the natural and social context in shaping consciousness. (pp.19-40). Amsterdam: John Benjamins.
Farina, M. (2013). Neither Touch nor Vision: Sensory Substitution as Artificial Synaesthesia?. Biology & Philosophy, 28(4), 639-655.
Loomis, J. M., Wiener, W. R., Welsh, R. L., & Blasch, B. B. (2010). Sensory substitution for orientation and mobility: what progress are we making? Sidebar 1.1 (pp. 7-10) to Chapter 1, Perceiving to move and moving to perceive: control of locomotion by students with vision loss by David A. Guth, John J. Rieser, and Daniel H. Ashmead (pp. 3-44). Foundations of Orientation and Mobility (History and Theory), 1.
Müller, J. (1843/2003). Müller's Elements of Physiology (trans. W. Baly, 4 vols). Bristol: Thoemmes Press.
Nanay, B. (2017). Sensory substitution and multimodal mental imagery. Perception, 46(9), 1014-1026.
O'Regan, J.K. (2011). Why Red Doesn't Sound Like a Bell: Explaining the Feel of Consciousness. Oxford, UK: University Press, Oxford.
O'Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences, 24(5), 939-973.
Pascual-Leone, A., & Hamilton, R. (2001). The metamodal organization of the brain. In Progress in `brain research (Vol. 134, pp. 427-445). Elsevier.
Ptito, M., Chebat, D. R., & Kupers, R. (2008). The blind get a taste of vision. In Human Haptic Perception: Basics and Applications (pp. 481-489). Basel: Birkhäuser.
Sutton, John. (2010), "Exograms and Interdisciplinarity: history, the extended mind, and
the civilizing process', in The Extended Mind, Richard Menary (ed), (pp. 189-225). Cambridge, MA: MIT Press.
Wheeler, Michael. (2010)"In Defence of Extended Functionalism." In The Extended Mind.
in Richard Menary (ed.), (pp.245-270). Cambridge, MA: MIT Press.