Well well well, do we have some very exciting news to share with you today! The More Than AI Sandbox projects have been announced. MyWorld, in collaboration with Watershed, has awarded six teams with £45000, to experiment creatively with AI.
...and we are beyond excited to announce that we are one of the 6 teams who have been selected for the sandbox!
Each of these participating teams have been commissioned to create new innovative prototypes that push creative boundaries and utilise technology to creatively and critically engage in ways of working with artificial intelligence and diverse networks of intelligence. Through this exciting 3 month-long residency, pyka will be working out of the Pervasive Media Studio at Watershed in Bristol to prototype a new system that will sit at the centre of The Expression Orchestra; a brand new AI-powered adaptive audio engine called Conductor.
ϟϟϟϟϟ r e c o r d s c c c c r r r r a a a a a a t c h ϟϟϟϟϟ ...but wait, there’s a long old journey that we’ve been on to get here. The Expression Orchestra has been an extended development focus for us at pyka_production for several years now, so what exactly has happened to get us to this point? ...why does The Expression Orchestra exist? ...and why are we about to explore AI as part of EO's upcoming round of product development?
It's safe to say that it’s time for a recap.
Flashback to the summer of 2018 and the shiny happy pyka_people were getting ready to deliver an exciting and ambitious R&D project funded by The Arts Council of Wales that would ultimately conclude with the conception of The Expression Orchestra - our suite of alternative electronic music instruments designed to broaden access to empowering musical performance.
However, this was never the intention for this project. This project would actually start off with us seeking to modify and adapt some of our existing screen-based creative sound apps (e.g. pyka_loop, pyka_lens) to work with assistive technology platforms such as the Eyegaze - an eye-tracking computer interface designed to support individuals from multiple locked-in diagnoses.
Only... we didn’t end up doing that. It was through this initial R&D that we discovered that modifying preconceived software and tools to support assistive technologies is somewhat problematic... and not just on a technical level. Sure, there’s a mountain to summit on that front alone, but more important to us was the ethical and philosophical quandaries it presented: 'retrofitting' meant that accessibility was never a part of the design equation in the first place. Now, we’re not saying this should never be done. There are plenty of people doing lots of meaningful things in this arena. However, the purpose of Project Everyone (with its name very much wearing its heart on its sleeve) was to ensure that more people would have access to expressive interactive sound and music opportunities, and yet in this instance retrofitting our creative tools with accessibility features would actually put us at odds with those aspirations. How so? Well, during early consultation and explorations with the PMLD staff and learners at Ty Gwyn throughout the first phase of Project Everyone, we discovered that only a small number of individuals would go on to be introduced to training interventions for assistive eye-tracking technologies like the Eyegaze and, even then, only a small percentage of those individuals would acquire the proficiency to utilise the kind of feature-rich software and tools that we were seeking to bring to the platform. In a school with around 130 learners, we’re talking about fewer people than hot meals you’ve eaten today. Extend that to the wider world beyond Ty Gwyn and all of a sudden Project Everyone was at risk of becoming Project... Some? It was at the precipice of each of these revelations that we decided to radically course correct. Instead, we decided to set out on the creation of a brand new digital tool for expressive sound and music opportunities, entirely from the ground up.
This moment in time would mark the beginning of our access-first design approach at pyka. Through this new perspective shift, any new design concepts would be generated in response to specific access needs from the very beginning. Very much inspired by the revised 'my-hands-are-up-lets-own-this' design revelations of Don Norman himself, we were moving from a human-centred design approach to a humanity-centred design approach.
“What you’re doing here [with The Expression Orchestra] is really revolutionary.” Jamie Brotherton, Deputy Head, Ty Gwyn Federation Special School.
To identify access considerations for this new sound-tool-in-waiting we initiated the next phase of Project Everyone, which saw our pyka_workshop facilitators engaging the learners and support staff of Ty Gwyn, Riverbank Primary, and Woodlands Comprehensive in a myriad of creative explorations. Before we began to even think about what our new digital sound tool could be, the focus of these sessions was to get to the root of how empowering expression is enabled for students in these settings and the role that different objects, materials, spaces, and interactions can play in this process.
Many weeks went by, many artforms were explored, and many incredible sessions with learners between the ages of 3-16 took place. Observations from these sessions then went on to inform a new creative engagement framework called Passing the Power.
Passing the Power is a conceptual guidebook containing a variety of engagement approaches to support teachers and facilitators in planning and scaffolding meaningful creative engagements with learners of all ages and abilities. You can read more about the 6 approaches that form Passing the Power via the free-to-use engagement framework document we developed as part of this R&D.
Using this engagement framework as a foundation, we were then able to establish some crucial design rules for our new creative sound tool that would guide us on our access-first design aspirations moving forward:
1). Reducing The Channels
While delivering creative engagement workshops at Ty Gwyn we discovered that too much choice was always detrimental to authentic and focused creative expression. To alleviate this we would establish spaces of exploration that reduced the channels. When reducing channels for creative interactions it is important to
acknowledge the sensory abilities of the learner(s) in order to appropriately
adapt the sensory focus of the activity. Reducing channels in this context
can act as an extension of sensory regulation for the learner, leading to
increased engagement and more authentic expression. Constraints aren’t
the boundaries of creativity, but the foundation of it. Through explorations with staff and learners in these settings, we found that the best way to enable this was to offer up simple, clean, and clear points of interaction. This was often realised through the use of a raw material, a simple object, or a simple person-to-person interaction. So from here, we realised that objects that present a single, or prominent, affordance served this need best, and this would be a defining feature of the new digital sound tools that we were about to begin building.
2). Access & Richness Through Polyphony
The main focus of our conversations with countless support staff, technicians, and learners at Ty Gwyn, was on figuring out how individuals with differing needs could benefit from the new creative sound tool we were about to build. Naturally, conversations like this start to push concepts towards a one-size-fits-all approach, with any new concept at risk of overstretching both with regards to the number of features it offers up, as well as how many problems it is trying to resolve – but we knew that this would never be the way to go. Conversely, creating a digital tool that serves a need really well, may be of little to no value to someone else with different needs.
At first, this felt like a significant impasse, however coming to terms with that latter point was the singular reason we came to realise that this 'new tool' that we would eventually design and build should actually become a suite of tools – more specifically – an orchestra.
In doing this, very much borrowing on the paradigm that every instrument in a typical orchestra has its place and has its role, so too would the instruments within The Expression Orchestra. Through this, different and even conflicting needs no longer become a problem, but instead a celebrated element of the experience.
“If you can make a suite of digital instruments where complicated calibration procedures are not required then we would absolutely include this as part of our everyday classroom activities at the school.” Lydia Jarvis, Teacher at Ty Gwyn Federation Special School.
...and from those two design ideologies, The Expression Orchestra was born! Or at least the foundations of it were. From this we presented all of our creative observations, findings, and technical prototypes to wider public audience during a week-long takeover of Chapter Arts Centre in Cardiff.
Some of the learners and their families joined us at Chapter Arts as they got to experience the 3 new prototype instruments together as a small interactive ensemble for the very first time. Additionally, through engagement from the wider general public, we could see that these instruments were also appealing to a wider audience.
Which is much more than we could've hoped for at this stage in the project. We feel that this is a much more authentic, wholesome, and inclusive way to think about accessibility when building these kinds of products and experiences. For us it's not about creating tools that are for "us or them", it's actually about "all of us together".
"I know you created these instruments alongside people with disabilities, but I actually really enjoy playing them too!” Non-disabled member of the general public using FOCUS, and Eyegaze trial concept instrument, at Chapter Arts Centre.
Cut to 2023 and, thanks to the wonderful Welsh music charity Anthem and their Atsain fund, we were able to recruit two new junior facilitators, return to Ty Gwyn Special School to pick up where we left off, and develop 4 new 'pretotype' instruments that were, this time much more reflective of the philosophies that define the Expression Orchestra - introducing THROW, ROLL, PRESS, and SPIN as the first official instrument concepts within the suite:
…and all of that brings us to the right here and now; the spring of 2024. We now find ourselves equipped with a robust model for a suite of instruments, some very promising mid-stage prototype instruments, and lots of scope for new prototype instrument concepts as we move towards the full spectrum of instruments that The Expression Orchestra will ultimately offer.
So what are our hopes for our time on the More than AI Sandbox residency?
Well, we believe that AI has a powerful, nay, vital role to play within The Expression Orchestra, and the instruments still have many shortcomings. For example, we're still dealing with 'unheard interactions' whereby an individual's interaction with one of the instruments falls outside an expected range of stimulus that the integrated sensors expect to see. Whenever this happens we are failing to keep our promise that The Expression Orchestra instruments "are always listening and always responding". So we're very interested in how AI could help us acknowledge these moments and ensure that a rewarding degree of depth of expression is still possible for individuals who may have a softer touch, or a different way of moving. We will be setting out to develop our own machine-learning models to ensure these considerations are being addressed as we progress with the development of The Expression Orchestra and introduce Conductor as the AI-powered adaptive audio engine that will sit at the centre of the experience. Beyond the technical aspects of the project, we're also passionate about furthering the ethical and philosophical considerations behind the ongoing development of The Expression Orchestra. We will continue to develop a praxis that will guide and support the activities for this production, asking uncomfortable and sometimes unanswerable questions to ensure that our practices and outputs are considered, inclusive, and responsive to diverse needs and perspectives. And lastly, there's a public relations element we want to address through this project as well. As artist-designers, we're WELL aware of the bad rep that AI is gaining within a non-trivial proportion of the communities across the arts and cultural sector. It's evident that torches have been lit and pitchforks are poised. Whilst some of this anger is valid, we're fearful that this has fuelled a somewhat allergic reaction to AI within the wider arts community, which means that, as swathes of creative people block their ears and turn their back on its presence in our industry, the opportunity for exploration and discovery is greatly diminished. We firmly believe other non-generative versions of AI can play a valuable, non-extractive, non-reductive role within the wider cultural sector, and the creation of Conductor will champion this in every regard.
Buuuuuuuuut... we're only just getting started! Watch this space as we dive headfirst into this thrilling summer R&D with support for the wonderful team at The Pervasive Media Studio.
“Artificial intelligence is becoming ever more part of our daily lives (whether we notice it or not). So, now is the time to ask how we want it to be used, to what ends, and who do we think should be leading that work? When I look at this brilliant group of projects and people I think - YES! These are the kinds of experiments that excite me about the future.” Jo Landsdowne, Executive Producer at Pervasive Media Studio
So... as you can see, a lot has happened on this journey so far, but it goes without saying that there is A LOT more to come.
...what do you think about all that, ChatGPT?
Nawwwh thanks Chat! We really think so too.
More Than AI Sandbox is part of the MyWorld IDEAS programme, funded by UK Research and Innovation’s Strength in Places Fund to establish the West of England as the global centre of creative technology excellence.