The future interfaces

While computers were getting smaller, one thing was getting bigger - comptuer displays. And no one doubts that it would be helpful to have even larger and higher resolution displays. Switching between applications using either Alt-Tab or Expose is never going to be as effortless as turning the head or just looking around. And to provide more and more real estate, computer displays have been steadily increasing in size and resolution (today more than 75% use 1024x768 and higher resolution) since the day one. So, of course, we will have bigger and better displays in the future, but the growth won't be just incremental — with e-paper, OLEDs, portable projectors, etc. the difference with today is going to be huge. But with a huge display the mouse input simply doesn't cut it anymore. You can't find a cursor on a giant screen, it takes some real effort to drag it across it, etc. A commonly proposed solution is to let the user use the hand. That's what we see in that overhyped movie Minority Report.

A hard to use, stupid and pointless interface from the unimaginative future of Spielberg's Minority Report.

There even is a real-life system currently in development by Raytheon based on the same ideas. Of course, the proponents of this approach forget that it's tiring to hold the hand in front of you for prolonged periods of time. They also rarely explain how exactly people would control the computers using their hands. Using hands to point to things is not energy efficient at all and using gestures a lot would destroy the wrists quickly. Tom Cruise manages to fake controlling the computer, waving expressibly to Schubert's "Unfinished" Symphony, but that won't work in real life just as easily.

Another problem is that, despite having a team of futurists that included Jaron Lanier and Kevin Kelly, the creators of the film couldn't come up with even a half-decent use case. I simply don't understand why John Anderton needed a 3 meter screen to play back three poor quality video clips. I can do it on a 17" CRT display much better, thank you very much. :)

Of course, this is not to say that big displays are useless or that hands can't be used to control them. But before that can happen we probably need to change the dominant GUI paradigm.

If a big screen is used to display a single (at most a few) object, such as a large map or a video clip, we don't need hands. Projectors work just fine today, controlled from laptops with touchpads. We would need hands only if the "control density" of the computer output (that is the number of possible operations you can do per square metre) drastically increases. If you need to manipulate parts of the output (such as tap a house on a map for more information) or manipulate (move, arrange, sort, etc.) a large number of objects quickly and visually, then hand control starts to make sense.

A much better fictional depiction of this scenario is the VR control room in Zion in the Matrix Reloaded.

Storyboard from the Virtual Control sequence

The concept art created for the Matrix actually makes much more sense than that Minority Report nonsense. In the above image we see different types of information displayed efficiently, because:

• they occupy the whole available visual field of the operator
• the size of particular "sheets" corresponds to the amounts of information
• the 3d stacking is used to organise the "sheets"
• traditional "physical" controls compliment the 3D display
• the operator sits in a comfortable chair
• the hands can rest on something
• the image quality is much better than what John Anderton had in 2054

The actual shots from the film look even cooler and actually make a huge step forward in terms of usability. If there ever was an uber-sexy GUI, that was it.

Virtual Reality interface in Matrix Revolutions

The key change that we see is that the flat "sheets" are gone. Now the information is availble in separate "chunks", groups of individual elements that can be moved (using a finger to drag them) around a vertical 2D space. This eliminates useless filler and allows the user and the computer to position information much more efficiently. As a result, this removes the problem of stacking (all information is instantly visible) and frees the background for 3-dimensional models of real-world objects, such as hovercrafts, tunnels and cannons. Also, ergonomic split keyboards neatly compliment the on-screen controls.

To effectively use such interfaces, the computer GUI itself needs to be redone from scratch - we need to go even earlier than Alto, all the way to NLS and start from there again. :) The future GUI needs to combine content-based structure with visual structure. We already have content-based structure — lists, menus, etc. But the visual structure is almost absent. One example is the cluttered Windows desktop, where users arrange icons according to their preferences. Another example are toolbars in applications such as Photoshop (but it's tools, not the information, that is arranged). But the most interesting field that is currently emerging are the graph-based interfaces. There aren't many examples yet, but it looks awfully suitable for our future needs. The complexity of managing the growing amounts of information, the complex categorization schemes and the interrelations between different pieces of data.

Some existing examples of this are:

1) Liveplasma service for graphically navigating the world of movies and music; 2) FreeMind mindmapping application for organising information structurally, but navigating it visually too; 3) CmapTools for concept map modelling and collaborative online editing.

Interestingly, the interface in the Matrix can actually serve as an inspiration not only on how we can manage the "chunks of information", but also to how the display itself may work. Why use a 3-metre plasma or an OLED display when you can have a virtual one? And we won't need to plug-in into Matrix in order to get it. Augmented reality may be a partial answer to the challenge of creating large displays, where the controls such as in the Matrix example are overlayed on the real world. Of all technologies capable of creating virtual 3d objects that the user can interact with this one actually looks the most promising. Having glass surfaces, walls and other screens may not be necessary, when we can just hang the virtual objects in front of the user?

Sousveillance - left or right?

Stephan (blog), I read about sousveillance quite some time ago, but I could never fully understand it. I never doubted that it's ultimately a good technology and that it's already feasible to some extent, I just couldn't see (from all the articles I read) what is it ultimately useful for, beyond pissing off department store clerks and security officers. ;)

Since I didn't have much of an idea of what sousveillance is going to be used for, I couldn't really understand how it fit withing different parts of the political compass. After some deliberation with a cup of tea, though, I got a couple of ideas. This may not make much sense and may be ultimately wrong, but that's how I see it now. This applies mostly to "inverse surveillance" sousveillance.

First, in a liberatrian society (if such can exist) people definitely have it the easiest to justify sousveillance - the individual just has the right to do it and noone should be able to limit his freedom. Liberatrians won't have much use for it, though, since the libertarian government doesn't actually care one way or the other about anything its citizens may record.

People in socialist countries might be luckier. While the critics may remember the tightly controlled xerox machines and limits on free speech, it should be noted that only subversive speech was unwelcome (and of course, spying in secret cities, military bases, etc.). The Soviet government and the soviet society in general welcomed communications and feedback, from wall newspapers to letters to Pravda and the Central Committee. One can't tell that for certain, but I suspect that the Communist Party of the Soviet Union would not particularly mind e-mails (even MMSed) with complains and suggestions with photos and videos attached. :)

But still, I can see only a limited use for sousveillance in a socialist state. It looks to my eyes that sousveillance would benefit the most people living in oppressive, but democratic (theocracies and military dictatorships don't qualify) societies. That is, people living in ordinary Western capitalist democratic countries, such as the US or many Western European countries. In these countries there is plenty of material that is:

1. not widely known
2. should be filmed and revealed
3. to some extent covered by the government
4. accessible to the people
5. can lead to changes if made public

To look at the sousveillance in general (not just at inverse surveillance), we can consider the deeper differences between libertarians/right and socialists/left (not the only dichotomic division, but let's tackle this first). Very simply put, libertarians consider the individual most important, while socialists consider the society most important. It's obviously a crude simplification, but without the society there is little use for personal recordings, except for the functional (memory aid). The potential of sousveillance is utilized only when we can share the recordings with others, be it through glogging or something else. Obviously, the possibilities are largely untapped at this point, but there must be a lot of ways that the experiences, knowledge, etc. could be shared through this. This application doesn't look very political though, more social and philosophical. Clearly it leads to more openness, tighter social connections, better understanding, etc., just like open communications in general. Sousveillance doesn't so much empower the individual as it strengthens his connections with others. This is clearly something that, for example, Soviet thinkers and artists in 1920s and 1930s would be extatic about. :) So I guess one can make a case that sousveillance is very compatible with leftist ideas.

Soviet Union - Understanding the truth

The lies about Soviet Union are already entrenched in our collective mind. We don't remember what was happening in 1930s and mostly base our judgements on the lies spread later. That's why it's so important to read unbiased historic personal accounts to understand how deep and detestable these lies are.

Two excellent books that are best suited for this are Return From the USSR by Andre Gide (Amazon) and Moscow, 1937 by Lion Feuchtwanger (Amazon). Both these writers were not avid supporters of communism, both were honest and widely respected by their contemporaries, both were invited to Soviet Union and both had the ability to travel around the country and see everything without, not just the facade, without any pressure or control. Both include some criticism (Gide criticises more), but both admit that Soviet Union was a widely successul majestic country populated by truly happy people that loved their country, were happy to work to make it better and sincerely believed that socialism (communism) is the right way to go and saw everyday the most impressive results. They had the reason to be proud of living in the Soviet Union and people all over the world had the reason to look at Soviet Union for hope and inspiration.

It is a simple fact, Soviet Union was the best and the most forward looking country in the world. Sadly, it collapsed under the attacks from outside and from within, but it's still the best model society we ever had and the best way to build a truly free and happy society is to base it on the Soviet model.

For the people by the people - not for dictators, not for the Party, not for the capitalists or for the monarchs anointed by god, but for people. The goal was to build the just, prosperous, happy and free society and everyone shared that goal.

Realism of computer games

As of 2005 we are on the threshold of realism in games. It is finally possible to simulate certain aspects of reality in real time and with sufficient precision to declare it an accurate simulation overall.

For example, the Forza Motorsport racing simulation for Xbox is physically realistic. It is mostly on par with reality, even though it's not indistinguishable yet. To achieve this, programmers from Microsoft Game Studios take into account between 3000 and 10000 variables and simulate all aspects of driving, running the simulation at 240 ticks per second. For Race Against Reality Popular Science asked a veteran gamer and a professional race driver to extensively test drive both real cars and their virtual prototypes. The conclusion was that the game simulation is accurate.

Similar level of realism is available for the flight simulators, again from Microsoft. Some simulators are so realistic that pilots are allowed to log the virtual hours just like the real ones.

However, these simulations are not completely realistic yet. There are still things that can be improved though before we have perfect VR.

• Graphics aren't perfect yet. One of the bigger problems is lighting and shadowing. To make realistic materials technologies such as RealReflect need to be developed.
• Sound - there is still no good programmatic sound generation. It's all samples, mostly.
• Global physics - it's possible to simulate several objects (cars, planes) very accurately, but an all-encompassing simulation is still too complex for the tech we have.
• Simulation of acceleration, tactile contact and everything else related to physically "being there".
• AI to make the world come alive

The video-realistic graphics based on general-purpose stable rendering systems (i.e. no more custom-made rendering engines for every new project) will come around 2010-2015. The programmatic sound may be delivered somewhere between 2015 and 2025. Global physics may be done sufficiently well around 2015-2020. The realistic simulations of all senses may come somewhere between 2015 and 2025. Sufficiently good non-human and domain specific human AI (i.e. for an NPC that can realistically perform in a narrowly defined context) may come somewhere about 2015-2020. Good human-level AI (in the context of video games it's a companion that you can interact closely for many hours in a variety of situation, including free-form talking) is a more complex problem and will probably not be achieved until the 2030s.

Still, we have already entered the realm of virtual reality. In some aspects, although not in all, virtual environments are already as good as real ones.