BRIANNA BLASER: Stefik, I'm turning it over to you.
ANDREAS STEFIK: Sounds good.
OK, so I have the pleasure of this morning,
as people are still kind of logging on a bit,
of introducing Mr. Neil Brown.
Neil lives in the UK, and has been
doing some really fantastic--
I would actually argue, seminal work
on understanding at scale how people code.
A lot of the work that they've done
is on Black Box, which some of you may know.
But they've also done some really amazing work
on these crazy little tools called Stride, and Greenfoot,
and all sorts of things that he probably
knows better than I do anyway.
So I find Neil's work pretty inspiring, broadly,
and I'm really glad that he was willing to do this talk for us.
So at that point, I will be quiet
and, I will allow Neil to do the talking.
So thank you again for joining us, Neil.
OK.
NEIL BROWN: Thank you very much.
OK, so I just need a moment to share my screen.
OK, so hopefully you're looking at my slides.
BRIANNA BLASER: It looks beautiful.
NEIL BROWN: Someone give me a thumbs up.
OK.
All right.
OK, so my name is Neil Brown.
I'm from King's College London in the UK,
and I'm going to be talking to you about, essentially,
block based programming, how at the moment
it's kind of inaccessible and what we could do about it,
and some of the work that we've been doing towards that.
So I'm just going to start with some acknowledgments.
So a lot of the background research for this talk
was done by Vaibhav Patel.
The design of the tools was done jointly
with Michael Kolling and Pierre Weill-Tessier.
Some of those might be on the call at some stage.
And some of the development work was
done by a few undergraduate students of ours.
And I just wanted to acknowledge and Andy
Stefik, who we just pat each other on the back all day long.
He's been very helpful and encouraging for us
to take on this work.
So what I'm going to be talking about today is accessibility,
but of course accessibility is huge.
There's all kinds of different parts of accessibility.
So I'm going to focus in on vision impaired users who
require a screen reader.
And most of the talk is just going
to be about, essentially, screen readers
and how we get them working with blocks.
So for those of you who were unaware,
and I wasn't really when I started out on this,
there's a variety of different levels and types
of vision impairment.
So to give an example I'm familiar with.
My neighbor has age-related macular degeneration.
So he's legally classified as blind,
but if I take him to the supermarket,
he can push a trolley round without hitting
anyone or anything.
So he has that level of vision, but he
can't read any of the signs.
He can't read any of the packaging.
So there's all kinds of different vision impairments
that have different effects for different people.
So I'm going to particularly focus
on the sort of actual rarer edge of the spectrum, which
is people with no vision, just because that's sort
of the most difficult problem.
And if we can make progress on that area, then generally,
we'll have benefits for people with other levels of vision
impairment.
So let's talk about the current state
of things and accessibility.
So I've got various demo videos that I'm
going to show you involving screen readers.
So I set out--
I thought, OK, well, I'll start by making one
to show you what text programming is
like with a screen reader.
And already that turned out to be a lot more
difficult than I thought.
So I tried IntelliJ, which is the IDEA I usually use.
And it was silent.
I couldn't get it to say anything
with the screen reader.
And I think it's something to do with the Java access
bridge, but technical problems.
So I thought, OK, I'll try Visual Studio Code.
It just said blank on every single line,
like actually read out blank, blank, blank,
as I went up and down.
So I went OK.
I can't use that.
I tried Visual Studio, sort of, full Visual Studio.
It gave several errors initially once I had NVDA set up,
but eventually an update fixed it.
So this just kind of gives you a flavor
of what it's like to try and actually get
screen readers working.
Like I'm a programmer.
I have full vision, and I'm sort of running into all
these kind of problems.
You could imagine how difficult it
would be for somebody who's maybe new to programming
and not sort of very technical to actually even just get
these things going.
But I did manage to make you a video,
so I'm just going to show you what
it sounds like as you move the cursor around within a text
based editor.
[VIDEO PLAYBACK]
SCREEN READER: Slash slash program uidof 689.
Begin data scope.
Blank.
Slash slash attach shader shader 687.
NV exec GL GL attach shader program uid 689.
shader uid 687.
Space space space N, V. Line E, X, E, C, line, G, L,
left paren, G, L.
[END VIDEO PLAYBACK]
NEIL BROWN: OK, so that's the end of that bit.
Can I just get a quick thumbs up from anyone who-- just check
you can hear the video, OK?
Because, OK.
Just want to check my sound was shared properly.
So that's kind of what a screen reader sounds like.
I actually gave a demo of this talk.
And one of the audience said like they
find it very hard just to even understand the screen
reader because it's a very robotic voice.
It doesn't really have natural sort of intonation or anything.
And it can be quite difficult to understand.
But I'm going to make some observations
on how these screen readers work for text based programming.
So when you change line and your cursor moves to a new line,
it reads the whole line out, but there is a bit of a problem.
Some of the punctuation is just omitted
when it reads that whole line.
So I don't know if you noticed, but there
was a line with only a curly bracket,
and when you move there, the screen reader just
doesn't say anything.
So you don't actually know that the curly bracket is there
on that line.
So, obviously, that's very difficult,
if you also can't see that the curly brackets there.
And as you move the cursor around,
you get rid out the single character
that you've moved next to.
So it always reads the single character
that's ahead of your cursor.
But some of the names are a bit unexpected.
So what I would call an underscore, it called a line.
When it read out, it just said NV line.
And another thing that's a bit subtle at first,
but you kind of get used to it, is
that capitalization is indicated by a different pitch.
So when it actually said GL capitalized,
it used a slightly different pitch to GL lowercase.
So there's various sort of interesting features of screen
readers, but you can already see that it's a bit awkward even
for text based programming.
So now we're going to move on to talk
about block based programming.
So block based programming is very popular with kids,
for teaching programming to young children,
especially sort of under the ages of 12.
And the problem is that a lot of these systems
are simply inaccessible for those with impaired vision.
And that's actually kind of twofold.
One is the lack of screen reading, which is mainly what
I'm going to be talking about.
But also just the fact that it's primarily mouse-driven.
So if you have impaired vision, using a mouse
is generally more difficult than using a keyboard
because you can't always see clearly where the cursor is.
Or if you have no vision, then you
can't see the cursor at all.
So I set out to make a video of what Scratch
is like with a screen reader.
So let me just make sure it's up.
And as I made the video, I was trying
to touch all sorts of keys to see what they did.
So this first bit of the video, this
is the response to me banging all kinds of keys.
[VIDEO PLAYBACK]
SCREEN READER: Space space.
[END PLAYBACK]
NEIL BROWN: So it at least recognize the space keep,
but nothing else happened.
So I said OK well I'll use the mouse to see what happens.
So I went and clicked in this field.
[VIDEO PLAYBACK]
SCREEN READER: Three.
Zero.
[END PLAYBACK]
NEIL BROWN: So you can see that when I typed,
it actually read it out, but otherwise it's just silent.
There's no way to actually read the content of any
of the blocks.
It didn't even read the content of the text field
when I focused it.
I expected to at least get 15, and then 15
as I moved the cursor around.
But it was just completely silent
until I managed to focus something and type something.
So the issue there is that it's not just fairly silent
with the blocks, but it's actually even more silent
than I expected.
I thought at least the text fields would sort of have
standard accessibility on it.
So you can see that this is essentially unusable if you're
using a screen reader.
You just can't get any sense of what code is there
or sort of even enter the code yourself.
So it's not all quite as bad as that.
There's been a few attempts to improve
some other block based systems.
So I'm going to look at, not all of them,
but just a couple that have been done for Blockly.
So Blockly is a Google originated system
for block based programming.
So one attempt I think was by the Google developers,
but it wasn't completely clear, and that looked like this.
So really this is quite interesting
because it doesn't look like blocks at all.
It's a sort of completely different display.
So what we've got here is sort of a nested list.
At the topmost level, it says if true,
do print "abc," which is the content of the whole code.
And then there's nested sublists that actually breaks that
apart, so you can read the separate components.
So the first item is if, which on the block
would be a sort of little label.
Then the second item is actually the sort
of manipulatable true part of the condition of the if.
Then you've got another label which is the do.
And then there's the body of the if, which here is printing abc,
and then that's also kind of decomposed into different bits.
So it's quite interesting because they've
taken this, actually, the sort of syntax tree
that was always underlying the blocks,
and turned it into a very text friendly representation.
So this works well with a screen reader actually.
You can move up and down the list.
You can navigate around.
It will read out all the text that you would expect.
It's just-- is this sort of blocks at this point?
Is it a kind of different structured editor,
but it's certainly an interesting attempt.
So I think I've covered those.
So yeah, the screen reader has good support for nested lists.
So it's quite a good representation.
And a different independent attempt at Blockly
accessibility was by a set of researchers
led by Mountapmbeme.
And that was just published only about a month ago or so.
So what I've done here is I've borrowed the demo video
that they submitted with their ACM paper,
so you can go and check this out if you
find their paper afterwards.
So I'll just let their video run for a minute.
[VIDEO PLAYBACK]
SCREEN READER: Meeting controls window.
Blockly interface.
Google Chrome.
Tree view.
Logic not selected 1 of 8 level 1.
Loops 2 of 8 level 1.
Repeat.
Blank.
Blockly interface document.
Edit mode entered now.
Top connect times connection.
Tree view.
Loops 2 of 8 level 1.
Math 3 of 8 level 1.
Number block.
Blockly interface document.
Edit mode entered now.
Edit mode exited.
5 unselected.
Blockly interface document.
Repeat 5 times.
Edit mode entered now.
Top connection.
Times connection.
Do connection.
Tree view.
Block.
Loops 2 of 8 level 1.
Math 3 of 8 level 1.
Text 4 of 8 level 1.
Disabled text block.
Disabled prompt for text with message text block.
Print.
Block.
Blockly interface document.
Edit mode entered now.
Top connection.
Text connection.
Tree view.
Logic not.
Loops 2 of 8 level 1.
Math 3 of 8 level 1.
Text 4 of 8 level 1.
Text block.
Blockly interface document.
Edit mode entered now.
Edit mode--
[END PLAYBACK]
NEIL BROWN: OK, so at this stage I'll
pause it because we've probably seen enough.
Sorry.
There we go again.
So a few observations about that.
I mean, obviously we're better than the sort of silence
that we saw with Scratch, but it still
seems that it sort of reveals the underlying awkwardness
of some of the blocks aspects.
So there seems to be a lot of excess speech,
and it was sort of quite laborious to navigate the block
palette in order to find the blocks that you wanted,
and then actually insert them into the structure.
So I mean, that's because Blockly already
had this catalog of blocks type system or palette,
it's sometimes called.
So that's not so well suited for a keyboard.
It's quite good with a mouse.
And if you can immediately see all the blocks,
but with keyboard navigation, and not necessarily seeing them
then it's a lot more difficult. And also
I think you need to understand the block
structure quite well in order to manipulate it effectively.
So although it does reveal to the screen reader
and to the keyboard manipulation all the possibilities,
you have to navigate top connect,
to do connect, to bottom connect,
to navigate around those and actually
be able to go through the structure.
I feel like if you knew Blockly well already,
you could sort of do well with this.
But if you don't know it, and you're just revealing it
through the screen reader, it might be quite difficult.
Also, the block canvas can make things
maybe a little bit awkward, I think,
to then navigate around all the different sort of groups
of blocks that you might have in a canvas
and how to maybe place those.
But it seems all the possible solutions
for accessible blocks, they all have one thing in common,
and that's the idea of having a single point of attention-- so
cursor or focus.
I particularly mean the keyboard cursor here.
So we're used to this concept, of course,
already whenever we're doing or editing
a Word document or something.
The cursor gives a focal point where we can then insert text.
We can delete text.
We can select it.
But also, as we've seen with the text based programming,
it gives a focal point for the screen reading as well.
So it seems like all the block solutions
I've seen involve some notion of a cursor or some kind
of persistent focus that moves around the different block
structure.
So it isn't usually there in a sort of purely kind of mouse
driven block based system.
So it seems like the cursor gives us screen reading,
and it tends to give us keyboard manipulation.
And so that's what we really need for accessibility.
So I'm going to talk a little bit about some of our ideas now
and they do involve the notion of a cursor.
So we've been working for a few years on something
that we call frame based editing, which
is a lot like blocks, but it's just kind of different enough
that we decided to give it a different name
to distinguish it.
So we have this cursor.
There's a blue cursor there that sits between the frames.
That can be moved up and down.
It allows insertion using the keyboard of different frames.
It also allows selection, as we'll see in a minute.
But unlike blocks, expressions are edited solely
using the keyboard.
So if, in a block based system, you want to do something like 1
plus 2 times 3, you'd need to drag in a plus block.
You'd need to drag in a times block.
You'd need to then fill in all the different fields separately
using the keyboard.
In our system, you'll just type 1 plus 2 times
3, and type it in with the keyboard.
So that's one vital difference between the two.
So I've pre-recorded a demo, so I'll just play that for you
now.
[VIDEO PLAYBACK]
Frame based editing has the same basic idea as blocks.
The code is structured as a series of snapable items.
So although they're formatted to look like they flow together,
the increment hunger and increment age method
calls here are actually frames that can be selected
and dragged around and dropped.
The key difference between frames and blocks
is the idea of the frame cursor.
So the blue bar here is essentially a cursor
that sits between frames.
Using my keyboard, I can move it up and down
with the arrow keys.
And I can also then use it to insert a new frame
if I want to.
So the shortcuts are shown on the right hand side here.
If I press I, then I get an if frame.
So just as with blocks, this is one cohesive frame.
I can, again, drag it around if I want to.
But the difference to blocks is that when
I come to enter the condition, I don't then
continue to drag in a little smaller blocks or smaller
frames in order to fill the condition in.
I just type it using the keyboard,
much as I would in a text based editor.
So I can write this method call here canBreed,
and then if I press down on my keyboard,
I can go inside the if, and I can choose to add more frames.
So if I press the shortcut space here
that inserts a method, which again, I can just type here.
So I can then tell it to print out a message like so.
So as you can see, some of the syntax
here is automatically inserted.
So when I added the open quote, I automatically
got the closed quote.
There's no way that I can delete this closed quote now
without deleting the entire string.
So the string is, therefore, always sort of either complete
or not present.
Same with the brackets here.
When I added the open brackets, the closed bracket
was automatically inserted.
And I can't delete this closed bracket
without deleting the open brackets.
So they're always paired, and this
reduces the amount of possible syntax errors,
much like blocks does.
But the difference here is the entry
can be done with the keyboard, so it's
faster and easier than the blocks, where you
have to drag in blocks, rather.
And so with this frame cursor, I can move around.
So there's extra shortcuts.
If I hold down the Control key here when
I'm pressing up and down, I move up and down
just at the same level.
So rather than going into this if, I skip past it.
So that's useful for navigation.
I could also select multiple frames,
which is sometimes the pain in block based systems.
I can then choose to delete these,
if I wanted to, as a selection.
Or I can choose to grab them both
and move around and manipulate.
Although I use the mouse for that interaction,
the same is available with the keyboard.
For example, if I select these again, if I hit Control-X,
it cuts them.
If I hit Control-V, I pastes them.
[END PLAYBACK]
OK, so that's just a sort of overview of frame
based editing in general.
Not yet as accessibility capabilities,
but it's important to understand the general model
before we go on to talk about that in a little bit.
So some of the features there, so we
have a lot of the advantages of blocks,
but with a few differences.
There's no block canvas, where you
can get lots of little different bits of code
hanging around in Scratch.
So everything is just in a single sort
of cohesive structure like in text based programming.
There's no white space to deal with.
You don't type any white space.
It's all derived from the placement
of the frames what's inside what,
just as you have with blocks.
You don't get any of the structural syntax errors,
like unterminated scopes, you know, open curly bracket,
but no close because you're not dealing with those.
Also the errors are fairly localized,
so when you do get an error, you could usually narrow it down
to one particular sort of text field
and say, OK, the error is in this particular bit.
So they don't sort of span an entire file
when you have an terminated method or something like that.
But compared to blocks, we also get keyboard entry
and keyboard manipulation.
I was talking through there some of the shortcuts I was using.
You can see that you can insert frames.
You can manipulate frames.
You can delete frames all using the keyboard.
So you don't have to use the mouse for any interaction.
You can do it all with the keyboard if you want to,
and that's important if you're someone
that is unable to use a mouse.
So we can talk now about some of the higher level
issues of accessibility.
So from a different paper by Mountapmbeme from
earlier this year, they had a sort
of review of accessibility and blocks.
They talked about five aspects of programming to consider.
So there's navigating your way around the code,
being able to comprehend the code, editing the code,
skimming-- which is sort of like just sort of glancing up
and down, rather than sort of specific navigation--
and also debugging.
And so this is a nice classification,
and to some extent, I would have liked to start the talk here
and say, OK, here's the higher level
aspects of a really good, accessible design.
But it felt kind of like whipping out
some Maslow's hierarchy of needs pyramid of all these sort
of complicated things we could do,
when the problem really is that the patient is flatlining
and they need a defibrillator to get going.
Like the bar is so low at the moment that
talking about these higher level issues
feels like a bit of a luxury when
so many of the systems just kind of don't really
have any support for accessibility at all.
But nevertheless, we can talk about a couple
of these aspects.
I'll show a couple of existing examples
and show some of the things that we've been working on.
So one piece of existing work is Struct Jumper.
So Struct Jumper was a plug-in for Eclipse which takes
your code--
I think it was text-based code, but the idea
would apply just the same, I think, to blocks.
And then it would turn it into a sort of tree view,
showing the general structure of your code.
So it would show the method header as one item,
and then sort of any scopes inside.
So for loops, while loops, if statements would then
appear as nested items within the tree view.
So you can sort of skim the code.
It's almost a bit like code folding
that you sort of fold it up your code.
And then you can go down the tree view--
but it is a proper tree with sort of nesting and so on--
in order to find your way to the piece of code that you want
or just to get an impression of what the code is like overall.
And then you could return to the main editor.
So it's a sort of extra navigation pane.
Another idea is this CodeMirror blocks.
So this is a plug-in for an editor that already existed
for text called CodeMirror.
And what it would do is allow you to take some of the text
and turn it into blocks, so you get
a sort of block like display that you can
navigate as if it were a tree.
So it's a bit like some of the hybrid editors
that you get with things like Pencil Code,
where you can switch back and forth between text and blocks,
but they had a particular focus on making sure
that it was accessible, and that it could all
be navigated with the keyboard.
So some of our ideas for how we might improve frame
based editing for accessibility.
So one thing is the idea of--
I don't know the proper term I think here,
but let's call it speech hierarchy,
like typographic hierarchy.
So for screen readers, you often want the important text first.
And so the reason for that is let's
say you're going down a list of items,
if you get a lot of boilerplate text before each one,
then you have to listen to the boilerplate
before you can decide if this is the item you want.
Whereas if you have the unique, distinguishing text right
upfront and said first, then you can
say, oh, that's not the one I want
and skip down the list much faster.
So you really want to order it so that the most important
thing is spoken first.
The problem is in most programming syntax
like you'd use in text based programming,
you get a whole load of bits at the front
that you don't necessarily want.
So in Java, for example, most of the fields
usually have their access modifier at the front.
So the first thing is private.
So I mean the classic public static void
main, the public static void are not that interesting.
The key part is the main, because that's actually
the name of the method that distinguishes it.
So you'd like to hear the main first.
And with frames we actually have the ability
to change what's being read first,
so we don't just have to follow the order.
So here the frame cursor in our model of accessibility
reads out the frame that comes after it.
So I'll just play what that sounds like.
[VIDEO PLAYBACK]
SCREEN READER: field age of type int with private access button.
Field foodlevel level of type int
with protected access button.
[END PLAYBACK]
NEIL BROWN: So maybe we left the screen on a bit faster there,
but what it was saying was like field foodLevel of type int
with protected access.
So we've got the name right up front.
Actually, it says field first.
I've been wondering as I made this talk whether we should
scrap that, whether we should instead
say foodLevel field of type int with protected access.
There's also a question for sort of users
who use this a lot whether you actually
need the sort of stop words in there about of type
and with protected access.
If you've been going for a little bit,
you can pretty guess that private protected public
is an access level.
So there's a few design decisions
that you could tweak either way, but the principle there
was that we can actually put the field name at the front, which
is the most important bit, even though in terms of text order,
it's actually at the end of the line.
Our syntax derives from Java.
The idea is that people can use our Stride editor here,
and then sort of transition into Java afterwards
once they're familiar with it.
So another issue we can consider is that of context.
So knowing where you are within the code is useful,
and I don't just mean line 362--
I mean, sort of within the actual structure
proper of the code.
And you also want to know--
so the text based programming system,
it reads out the line you're on.
So you at least know the full line,
but you don't necessarily know what's around you.
So what we've been experimenting with is having more information
read out so that with each frame that you're next to,
it not only tells you the information about the frame,
as I showed you in the previous video,
but also it gives you information on the context.
So here's a demonstration of what that sounds like.
[VIDEO PLAYBACK]
SCREEN READER: Method incrementHunger
with private access and void return type of documentation.
Make this fox more hungry.
This will result in the fox's death button
your're before a method in the class box.
Assign foodLevel with foodLevel minus 1 button
before an assignment frame in the method incrementHunger
in the class fox.
If frame with condition food level
is less than or equal to zero button.
You're before an if block in the method incrementHunger
in the class fox.
Method call set left bracket right bracket button.
You're before a method call in the [INAUDIBLE] frame
with condition foodLevel less equals 0
in the method incrementHunger in the class fox.
[END PLAYBACK]
NEIL BROWN: OK, so there's a few things to unpack.
I'm going to come back to it occasionally
says button later on.
So there's a few design decisions
we might sort of polish up here.
It gives you context every single time,
even though in this case, we had got repeating context.
You maybe don't need it every single time.
If you've heard that bit of context recently,
we could maybe iron out that we don't have to read that out.
But in general, what it was doing
was it's telling you the frame your next to,
and then if you wait there to hear more context,
then it will slowly kind of read backwards up the tree
to tell you, you're in this if, you're in this method,
and you're in this class.
And if users want to, they can just skip down
and that will sort of cancel the reading,
and it will read out the next item.
So it doesn't interfere because it's later on, as we
talked about this hierarchy.
Because it's later on, it doesn't really
interfere with their understanding it just
gives them more information if they want to.
We can maybe tweak quite how much they want,
but the idea seems reasonable that you
get this reverse hierarchical context of where
you are from the most distinguishing low level, up
to the least distinguishing high level.
So another thing we could maybe improve on is navigation.
And so we had this view already in Stride,
which we call Birdseye view, which is essentially
a bit like code folding.
And so just actually in preparing this talk,
I've made that properly accessible.
So I'll show you what that looks like and sounds like.
So when you hit a shortcut, it folds up
as it will in the second.
[VIDEO PLAYBACK]
SCREEN READER: [INAUDIBLE] with parameter list
less animal greater new foxes with public access and void
return type button.
Method incrementAge with private access and void
return type button.
Method, method, method giveBirth with parameter list less animal
greater new foxes with private access and void return
type of button.
[END PLAYBACK]
NEIL BROWN: So it kind of gives you a quicker way
to move up and down the code to see what methods
are within that class without having
sort of interfering bits.
So because it's folded away the documentation,
it no longer bothers reading out the documentation.
And simply there, I was just using the down cursor key
to go between the methods.
So it makes it kind of an easy way to navigate around the code
and find out what's there, and then you can press another key,
and it folds back out again.
And so in general we've kind of tried to improve the movement
and give you different options with the keyboard.
So we've seen a lot of these, but up and down moves
the frame cursor up and down.
If you press right, it actually enters the text fields
in the frame and lets you manipulate the expressions
in there.
If you press left, it does the reverse.
So it comes back out, and then continually pressing left moves
back up the tree.
So if you think of it as moving left on the indent levels,
it's slowly taking you further and further back up
your ancestors.
And we also have this control up and down
that moves at the current level.
So we feel like this gives a good variety of options
for moving around the code.
But the nice thing about having frames rather than text
is it's quite constrained in a good way.
You don't sort of end up at a destination you don't really
want to be in, like the middle of a white space indents
and wondering where you are.
You're always sort of at a useful location with the frame
cursor, because you're always beneath--
so just between or above a particular frame.
So there's various challenges of working with screen readers,
and I thought it might be good to highlight some of these,
having worked on it for a bit.
They can be slightly frustrating pieces of software.
So cross-platform difficulties.
So Java and JavaFX, which is the GUI toolkit we use,
offers a unified accessibility API that, in theory,
works the same everywhere.
Of course, it doesn't.
There are differences between operating systems
as to what happens.
But actually there are probably even bigger differences
between different pieces of screen reader software.
So the one I used for these demonstrations
was NVDA, which is a popular free piece of software.
There's also a built-in one on Mac.
There's a built-in one on Windows.
There's some paid for screen readers.
And they all will read out something slightly different
depending on the context that's there, trying to be helpful.
And of course that can make designing and testing
this kind of thing awkward, because sometimes you
change something on the Windows and it works wonderfully
with NVDA, and then try it with a Mac built-in screen reader,
and all of a sudden you've completely
broken screen reader sort of accessibility.
We definitely had that a couple of times
when we tried to be too clever.
Punctuation is an issue.
So punctuation in natural language
is generally used for clarification or emphasis.
It's not necessarily a crucial part of things,
whereas of course in program code, it's critical.
It indicates structure.
We had that line with a curly bracket
on at the beginning that wasn't read out by the screen reader.
But actually whitespace is not any better.
If you've got whitespace sensitive languages
like Python, it can be difficult with a screen reader
to know if you've got the right number of spaces
so that it lines up with the line above.
And it's not solved by moving to frame based editing either.
We still have brackets.
We have plus.
We have minus.
And so those aren't always read out quite
as you would think you would.
So one solution is to do something with like Quorum.
So Quorum is designed to reduce the amount of punctuation
and weird curly brackets and stuff
to use more text based syntax.
But again, like ours it's still got pluses and brackets
and your expression and things.
You can't escape it entirely.
I was reminded of this quote that I gave in a separate talk
last week, which from Kahneman said,
"I knew all I needed to know, but I did not
think it through."
And that's kind of summed up screen readers for me
because they actually have all the necessary information.
It's not a failure of our system or a Visual Studio
or whatever to not supply the punctuation,
the screen reader has the literal string.
It just decides not to read it out to you.
So it's a problem with a screen reader.
And I can see why they do that.
They're not built only for programming.
They're built for other people who
don't have to worry about punctuation,
but it seems generally quite difficult or actually
impossible with some of the built-in screen
readers on Windows and Mac to actually get
it to read out the necessary information for the screen
reader.
Another thing is that the screen reader is kind of declarative,
not imperative.
So in general, you're kind of presenting a sort of structure
to the screen reader, and it will decide what to read out
and when to read it out.
So in a way, it's a good principle.
The user is in control, and the only speech happens in response
to them performing an action, often moving the focus
from one component to another.
And they don't get speech interrupting them
at random times, which I think would be frustrating.
But at the same time, there's not much support
for interrogating other data.
So let's say you move into a text
field in the frame based editing,
there's an error there.
How do we indicate that?
There's not an obvious way to tell the screen reader there's
something else here the user might want to know
about this being an error.
Or if the user wants to ask more information.
They might want to know--
maybe refresh themselves as their position in the code,
or ask for some other information
about the item they're beneath.
And there's often not good ways to actually
do that-- both in terms of a standard way for the user
to ask for more information, but also to give it back,
because generally the screen reader only acts, like I say,
when you change focus on an item perhaps.
So you then need to maybe sort of
have a pop-up in response to the key that they've pressed,
but then that's awkward because they
have to dismiss the pop up.
And so it's kind of extra clicks to get where they want to go.
So that was something we found difficult in trying to design
more interesting interfaces.
We've sort of run up against this screen reader
model where it's like, no, it doesn't work like that.
And basically you sort of can't add the extra information
that you want, not easily.
And on that theme, so screen readers generally
have a sort of idea about common GUI widgets,
especially kind of 1990s or so.
So there's text fields.
There's buttons.
There's labels.
There's lists of texts.
There's tree views.
And when you want to do something
maybe a bit different, then you run into problems.
What is a frame cursor?
I said earlier I'd tell you why it
was saying button in the video.
That's because the frame cursor is actually
a button as far as the screen reader's concerned.
That doesn't really make a lot of sense,
but you have to make it something and button was
kind of the closest option.
Some screen readers will say button.
Some don't.
That's the difference between different screen reader
software.
But maybe you say, OK, well, that's
silly because in text you don't say that I've
got the text cursor focused.
I've just got the text area focused.
So we'll just say we've got the sort of frame area focused.
But what is that in the screen reader's mind?
It's got to be a standard GUI control.
Maybe it's a list or a tree, but then both of those
are generally expected to be a list or a tree of text items.
But we don't have a tree of text items.
We've got a very complicated tree
of frames, which inside it has sort of a list of text fields.
And so you just sort of run into this restriction
that it's got a kind of look to the screen reader
like a sort of Windows 3.1 GUI.
And how can we actually take what
we want to have, and manipulate it, and present it
to the screen reader to make sense.
So I mentioned this to Stefik at one point.
He said maybe try making it a tree view,
but I had a little go with that, but again,
we got into this problem that the screen reader is fetching
a very classic tree view and because ours
is more complex, at some point, it would just kind of
break the system.
So it is a bit of a challenge to take novel interfaces
and fit them into the screen readers worldview.
So there's definitely room for improvement.
So we actually had a meeting about accessibility
at some point where we presented all this to Michael,
who's the head of our group.
And he said, well, should we just abandon this and work
on a better screen reader?
And, OK, that's way outside our scope and a bigger job
than it should be, but we're not going to do that.
But he's not wrong in a sense.
Some of the problems that we're running into
are kind of caused by the screen reader.
That's why as I understand it, Stefik actually
ended up sort of mostly replacing the screen reader.
Is that right?
If you give me a nod?
No.
OK, not quite.
So he can talk more about that.
ANDREAS STEFIK: On a very early project, we did.
On a very early project that we did, but that had all
sorts of other complexity that I won't bore people with.
NEIL BROWN: OK.
So basically it's hard to get rid of them.
You've kind of got to live within them.
One possibility I saw NVDA allows add-ons.
I haven't looked hugely into that yet,
but maybe that's a route to have an add on that actually reads
out punctuation like a sort of programmer-friendly mode
perhaps for text and for blocks and frames.
So just a few final bits then to sum up.
So we're planning to continue developing
Stride, which is the editor I've been
showing for screen readers.
We want to add similar support to our new Strype system.
So Strype is another frame based editor,
but it lives in the web browser, which
is a whole new set of challenges,
I imagine, to make it accessible.
And we're interested to connect with potential users.
We're hoping in the new year to return
to do some more development on this,
and it would be good to actually have some users who
are expert in this area, who are vision impaired themselves
in order to sort of test out some of these ideas.
So let us know if you're interested.
If you want to try the work at its current state--
so pretty much everything you saw today
in Stride is there except for the folding view because I only
fixed that a few days ago when I realized
it wasn't very accessible.
But it's in our BlueJ and Greenfoot system.
So if you download BlueJ or Greenfoot,
when you make a new class, there's
a Java or Stride toggle.
If you do Java, you get text based editing.
If you say Stride, you get frame based editing.
Strype is available, but we haven't yet
worked on the accessibility for that.
So it's not yet properly accessible,
but that's available at Strype.org if you're
interested to try that out.
So Strype with a y.
So in summary, the current state of block accessibility
is pretty terrible.
There have been some attempts at solutions.
All of them seem to involve some notion of cursor or focus.
So I think it's fair to say that's a critical element
to provide accessibility for a screen reader.
Even once you implement the basics,
there's then the sort of high level challenges
around things like navigation, around skimming.
I think that's when it gets particularly
interesting from a designer's point of view.
But some aspects of screen readers provide a challenge.
You've got to fit into their worldview.
They're not reading out punctuation properly.
So although they're necessary, they're
also quite a hindrance in what you want to do.
So we had some ideas, and I've shown
you those about developing support
that's unique to frames, but actually the point I'd make
is although it's pretty terrible in blocks at the moment,
I think blocks or frames, if it's done correctly,
could actually potentially be more accessible than text
because you don't have this issue with kind
of having to navigate around a lot of whitespace and quite so
much punctuation.
You can do things like the sort of structural context a lot
more easily.
Just think about reordering which order things are read out
in, you can do in frames more easily than you can in text.
So I think, in theory, a proper sort of structured editor
like blocks and frames could be more accessible than text.
It's just that at the moment we're
sort of starting from nothing, but I
think the ceiling is actually potentially higher.
So thank you very much for listening,
and I'm happy to move to questions.
I guess, Stefik, you will say what's next.
ANDREAS STEFIK: Thank you, Neil.
That was really, really fantastic.
I'm super excited about all the stuff
that you guys have been doing just
because, like you, said the state of things
is so nonfunctional that you could throw a dart
and come up with something that's better than something
that doesn't read at all.
But having more scholars thinking
about more things in this space is a really big deal
because we need lots of ideas to get this right.
So I'm just saying, thank you.
But at this point, I will open it up
to questions from anybody.
You're welcome to either flip on your camera if you'd like to,
or raise your hand if you prefer,
or type something in the chat.
Whatever works for you is just fine.
If no one has a question, I have plenty,
but I don't want to capitalize either.
OK, well, fine.
Then I'll steal the first question, then.
So I had-- one of the things that we've
been thinking about on ours, too,
because we've been really inspired by the Strype work
that you guys have done, is there's this thing that you
mentioned where in a frame you use the typing for many
of the expressions, which is something that we will probably
end up doing as well.
But obviously, the actual block editors usually don't do that.
You drag and drop them or you press keys or something
hypothetically for even expressions like plus and minus
and stuff like that.
Do you think that typing will have--
what do you think the impact will be
on younger versus older people?
One of the considerations we've been thinking
is if you have a kid that's like six or seven,
is that a downside or a good thing?
Or what are you thinking about the various ages related
to blocks with that decision?
NEIL BROWN: Yeah, so I think we thought definitely
for someone who's more proficient, it's quicker.
Like 1 plus 2 is quicker than dragging in a plus.
Like it seems fairly obvious once
you're sure what you want to do.
I think something that's also a bit
underappreciated with the block dragging model
is that it forces you into a prefix mode.
So if I want to do type 1 plus 2, I'm 1 then plus then 2.
Whereas if I want to do the blocks, I've got to think
what's the operator?
So it's plus.
So it's more like a scheme type thing
where I'm doing plus 1, 2 in order to make it prefixed.
And at the risk of offending any scheme people on the call,
I think infix is better.
It just-- it's how we're taught in-- especially for younger
people, we teach them in math and we teach infix.
So to then say, oh now you've got to sort of enter a prefix,
like it just seems like a bit of extra cognitive load that's
going to sort of wear on them.
So I don't really see a huge advantage,
especially in like plus minus times of having
the block model to drag in.
It seems like an easy win to change it to text.
Something more like the method calls,
OK, we can have all kinds of debate about that
because if you want to drag in like a square root
or something, maybe that's easier
than typing the square root.
Maybe it's a different decision.
But I think at least for operators,
I'm confident that typing is just better.
ANDREAS STEFIK: Interesting.
Thank you.
Richard, go ahead.
RICHARD LADNER: Neil, thank you very much.
That was fantastic and really enjoyed that.
Getting back to Stefix's question a little bit more,
a lot of the block's programming environments
were designed for little children--
like Stefix said, six-year-olds.
And for blind children, at least in the United States,
they're introduced to the iPad fairly early.
And so they learn voiceover on the iPad
and are very proficient at a pretty young age.
And so that was one of the motivations
for creating Blocks For All with Lauren Milne,
so that there would be some alternative that's accessible.
So if you have any thoughts about what you describe
today would be fine for high school maybe,
but what about younger children?
NEIL BROWN: Yeah, so I think there's sort of there's
two issues.
One is the voice entry, which is something
we haven't really looked into.
I think is a very interesting area,
but we've mainly been focused on keyboard entry and screen
reader display, rather than alternative sort of methods
of entry.
So I think that's interesting, but it's just something
that we haven't looked at all either the way.
I think the other aspect about age.
I mean, obviously in our editor, you can see that.
We've got a very Java like syntax.
It's a Java semantics because our existing tools were all
Java, and so we were doing Stride to then reach Java.
But I think the decisions we've made for the frame-based
editor--
if we wanted to make a Quorum frame-based editor
or some other language that was even
simpler and suitable for kids, then I
think some of our design decisions
would transfer reasonably.
It's not wedded to the fact, that we've
got this Java semantics that's just
a particular implementation of our language.
But I think the idea is of frames
and frame manipulation would transfer to other sort
of simpler languages.
ANDREAS STEFIK: We'll find out soon
because we literally copied that part of our stuff
for Quorum blocks.
That was not an accident.
So it's your fault when it fails, Neil.
I just want to point that out.
So [LAUGHS].
No, I think it's a good idea.
Sorry.
Richard, did you have another question?
RICHARD LADNER: No.
I've got to figure out how to lower my hand, though.
ANDREAS STEFIK: Oh, gotcha.
OK.
Jim Salmons asked, if we'll be sending out a recording.
And we will be sending out a recording.
We always do the closed captioning first,
just so that we can make sure it's accessible.
That does take a little while, but we will send out a video,
and we'll put it up on our website, which I imagine,
I just scooped Brianna by saying, but--
BRIANNA BLASER: No, I was going to say that I will email folks
when that is available as well.
And I'm going to [CLEARS THROAT] excuse
me, drop a link in the chat for an evaluation in a second.
So you can fill that out and I will follow up
with that over email as well.
NEIL BROWN: I'm also happy to share my slides,
although ironically, because I've got some demo
videos in there, they're not actually
necessarily the most accessible by themselves
compared to the text.
But that was just kind of the nature.
ANDREAS STEFIK: Yeah, one thing we always
talk about with accessibility is perfect is
the enemy of the good.
When you're talking about a state of the practice
where people can't get access at all,
sometimes you have to live with the fact
that something is not perfect.
So if your slides are not perfect,
we're not going to complain too much.
We actually really appreciate you doing work in this space,
really.
Other questions or thoughts for Neil here?
OK.
Well, then I just want to say thank you one more time.
I will probably reach out at some point to you, Neil,
as we get going and we can compare notes
once we have our--
we'll actually be ready with Quorum stuff
in a couple of weeks with some--
I mean, it's not quite ready yet.
But we should compare notes just sort
of figure out what's working and what isn't and stuff like that.
And I will mention one last thing
that you can do on the metadata side before you go.
There is a way to tell the operating system you've
got a spelling error.
It's not an error, per se, but it
does cause it to have this little click,
and it works pretty well for errors.
So if you can figure out how to hack it in,
that does actually work.
NEIL BROWN: Yeah, unfortunately I
don't think the Java API has support for that.
ANDREAS STEFIK: It does not.
NEIL BROWN: And you know, you're always
at the mercy of these like APIs.
We've got the Java API, and then it's
talking to the screen reader API.
And actually the fault often lies with a screen reader
API in my mind.
But the Java API is sometimes missing things.
There's also this kind of idea of a help key
that seems to exist, and Java will
allow you to respond to it.
But we found that in some screen readers
it is not quite clear how to trigger that
or how to make it apparent that you could trigger that.
So again, it's tricky.
ANDREAS STEFIK: Oh, yes on those.
There's also a whole bunch of other metadata things
like those that get sent down to the screen reader
and the screen reader's all ignore them.
So just be aware that that-- it might not even be your fault.
It might be that it's actually there, but that NVDA is like,
meh.
I'm not going to respond to that.
So it's not that unusual, sadly.
So yeah.
NEIL BROWN: It's challenging.
Yeah.
RICHARD LADNER: Yeah, there's a comment
from Jim Salmons in the chat.
And maybe Jim could you introduce yourself?
[INAUDIBLE] It he wants to?
He's still muted.
ANDREAS STEFIK: I'll read it out.
He's a disabled developer working developer research
study and support program using GitHub
Copilot for disabled developers and disabled STEM students.
Then he has a link in there.
In the immediate term, he's working
on GitHub next developers at the Hey GitHub voice coding
assistive tech.
We have a need for 20 to 25 disabled developers
to volunteer for a feedback group
to assess the current state.
He can be reached at--
and then there's some links in there for going.
It's possible that we can send that on a mailing list
or something possibly?
BRIANNA BLASER: Yeah, I'll follow up Jim.
But it sounds really interesting work.
ANDREAS STEFIK: Yeah.
So.
In any case.
Thank you all.
And thank you especially to Neil for your time.
So we appreciate it a lot.
Thanks for coming today.
NEIL BROWN: OK.
Thanks very much.
Bye.
ANDREAS STEFIK: Bye-bye everyone.
BRIANNA BLASER: Bye.