Lev Manovich

What is New Media?: Eight Propositions

From “New Media from Borges to HTML,” commissioned for The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort, The MIT Press, 2002).

I now want to go through other possible concepts of new media and its histories (including a few proposed by the present author elsewhere). Here are eight answers; without a doubt, more can be invented if desired.

1. New media versus cyberculture.

To begin with, we may distinguish between new media and cyberculture. In my view they represent two distinct fields of research. I would define cyberculture as the study of various social phenomena associated with Internet and other new forms of network communication. Examples of what falls under cyberculture studies are online communities, online multi-player gaming, the issue of online identity, the sociology and the ethnography of email usage, cell phone usage in various communities; the issues of gender and ethnicity in Internet usage; and so on. Notice that the emphasis is on the social phenomena; cyberculture does not directly deal with new cultural objects enabled by network communication technologies. The study of these objects is the domain of new media. In addition, new media is concerned with cultural objects and paradigms enabled by all forms of computing and not just by networking. To summarize: cyberculture is focused on the social and on networking; new media is focused on the cultural and computing.

2. New Media as Computer Technology used as a Distribution Platform.

What are these new cultural objects? Given that digital computing is now used in most areas of cultural production, from publishing and advertising to filmmaking and architecture, how can we single out the area of culture that specifically owes its existence to computing? In my book The Language of New Media I begin the discussion of new media by invoking its definition which can be deduced from how the term is used in popular press: new media are the cultural objects which use digital computer technology for distribution and exhibition. Thus, Internet, Web sites, computer multimedia, computer games, CD-ROMs and DVD, Virtual Reality, and computer-generated special effects all fall under new media. Other cultural objects which use computing for production and storage but not for final distribution—television programs, feature films, magazines, books and other paper-based publications, etc. – are not new media.

The problems with this definition are three-fold. Firstly, it has to be revised every few years, as yet another part of culture comes to rely on computing technology for distribution (for instance, the shift from analog to digital television; the shift from film-based to digital projection of feature films in movie theatres; e-books, and so on) Secondly, we may suspect that eventually most forms of culture will use computer distribution, and therefore the term “new media” defined in this way will lose any specificity. Thirdly, this definition does not tell us anything about the possible effects of computer-based distribution on the aesthetics of what is being distributed. In other words, do Web sites, computer multimedia, computer games, CD-ROMs and Virtual Reality all have something in common because they are delivered to the user via a computer? Only if the answer is at least partial yes, it makes sense to think about new media as a useful theoretical category.

3. New Media as Digital Data Controlled by Software.

The Language of New Media is based on the assumption that, in fact, all cultural objects that rely on digital representation and computer-based delivery do share a number of common qualities. In the book I articulate a number of principles of new media: numerical representation, modularity, automation, variability, and transcoding. I do not assume that any computer-based cultural object will necessary be structured according to these principles today. Rather, these are tendencies of a culture undergoing computerization that gradually will manifest themselves more and more. For instance, the principle of variability states that a new media cultural object may exist in potentially infinite different states. Today the examples of variability are commercial Web sites programmed to customize Web pages for each user as she is accessing the particular site, or DJs’ remixes of already existing recordings; tomorrow the principle of variability may also structure a digital film which will similarly exist in multiple versions.

I deduce these principles, or tendencies, from the basic fact of digital representation of media. New media is reduced to digital data that can be manipulated by software as any other data. This allows automating many media operations, to generate multiple versions of the same object, etc. For instance, once an image is represented as a matrix of numbers, it can be manipulated or even generated automatically by running various algorithms, such as sharpen, blue, colorize, change contrast, etc.

More generally, extending what I proposed in my book, I could say that two basic ways in which computers model reality – through data structures and algorithms – can also be applied to media once it is represented digitally. In other words, given that new media is digital data controlled by particular “cultural” software, it make sense to think of any new media object in terms of particular data structures and/or particular algorithms it embodies. Here are the examples of data structures: an image can be thought of as a two-dimensional array (x, y), while a movie can be thought of as a three-dimensional array (x, y, t). Thinking about digital media in terms of algorithms, we discover that many of these algorithms can be applied to any media (such as copy, cut, paste, compress, find, match) while some still retain media specificity. For instance, one can easily search for a particular text string in a text but not for a particular object in an image. Conversely, one can composite a number of still or moving images together, but not different texts. These differences have to do with different semiotic logics of different media in our culture: for example, we are ready to read practically any image or a composite of images as being meaningful, while for a text string to be meaningful we require that it obeys the laws of grammar. On the other hand, language has a priori discrete structure (a sentence consists from words which consist from morphemes, and so on) that makes it very easily to automate various operations on it (such as search, match, replace, index), while digital representation of images does not by itself allow for automation of semantic operations.

4. New Media as the Mix Between Existing Cultural Conventions and the Conventions of Software.

As a particular type of media is turned into digital data controlled by software, we may expect that eventually it will fully obey the principles of modularity, variability, and automation. However, in practice these processes may take a long time and they do not proceed in a linear fashion – rather, we witness “uneven development.” For instance, today some media are already totally automated while in other cases this automation hardly exists – even though technologically it can be easily implemented.

Let us take as the example contemporary Hollywood film production. Logically we could have expected something like the following scenario. An individual viewer receives a customized version of the film that takes into account her/his previous viewing preferences, current preferences, and marketing profile. The film is completely assembled on the fly by AI software using pre-defined script schemas. The software also generates, again on the fly characters, dialog and sets (this makes product placement particularly easy) that are taken from a massive “assets” database.

The reality today is quite different. Software is used in some areas of film production but not in others. While some visuals may be created using computer animation, cinema still centers around the system of human stars whose salaries amount for a large percent of a film budget. Similarly, script writing (and countless re-writing) is also trusted to humans. In short, the computer is kept out of the key “creative” decisions, and is delegated to the position of a technician.

If we look at another type of contemporary media – computer games – we will discover that they follow the principle of automation much more thoroughly. Game characters are modeled in 3D; they move and speak under software control. Software also decides what happens next in the game, generating new characters, spaces and scenarios in response to users’ behavior. It is not hard to understand why automation in computer games is much more advanced than in cinema. Computer games are one of the few cultural forms “native “ to computers; they begun as singular computer programs (before turning into a complex multimedia productions which they are today) – rather than being an already established medium (such as cinema) which is now slowly undergoing computerization.

Given that the principles of modularity, automation, variability and transcoding are tendencies that slow and unevenly manifest themselves, is there a more precise way to describe new media, as it exists today? The Language of New Media analyzes the language of contemporary new media (or, to put this differently, “early new media”) as the mix (we can also use software metaphors of “morph” or “composite”) between two different sets of cultural forces, or cultural conventions: on the one hand, the conventions of already mature cultural forms (such as a page, a rectangular frame, a mobile point of view) and, on the other hand, the conventions of computer software and, in particular, of HCI, as they developed until now.

Let me illustrate this idea with two examples. In modern visual culture a representational image was something one gazed at, rather than interacted with. An image was also one continuous representational field, i.e. a single scene. In the 1980s GUI redefined an image as a figure-ground opposition between a non-interactive, passive ground (typically a desktop pattern) and active icons and hyperlinks (such as the icons of documents and applications appearing on the desktop). The treatment of representational images in new media represents a mix between these two very different conventions. An image retains its representational function while at the same time it is treated as a set of hot spots (“image-map”). This is the standard convention in interactive multimedia, computer games and Web pages. So while visually an image still appears as a single continuous field, in fact it is broken into a number of regions with hyperlinks connected to these regions, so clicking on a region opens a new page, or re-starts game narrative, etc.

This example illustrates how a HCI convention is “superimposed” (in this case, both metaphorically and literally, as a designer places hot spots over an existing image) over an older representational convention. Another way to think about this is to say that a technique normally used for control and data management is mixed with a technique of fictional representation and fictional narration. I will use another example to illustrate the opposite process: how a cultural convention normally used for fictional representation and narration is “superimposed” over software techniques of data management and presentation. The cultural convention in this example is the mobile camera model borrowed from cinema. In The Language of New Media I analyze how it became a generic interface used to access any type of data:

Originally developed as part of 3D computer graphics technology for such applications as computer-aided design, flight simulators and computer movie making, during the 1980's and 1990's the camera model became as much of an interface convention as scrollable windows or cut and paste operations. It became an accepted way for interacting with any data which is represented in three dimensions — which, in a computer culture, means literally anything and everything: the results of a physical simulation, an architectural site, design of a new molecule, statistical data, the structure of a computer network and so on. As computer culture is gradually spatializing all representations and experiences, they become subjected to the camera's particular grammar of data access. Zoom, tilt, pan and track: we now use these operations to interact with data spaces, models, objects and bodies.

To sum up: new media today can be understood as the mix between older cultural conventions for data representation, access and manipulation and newer conventions of data representation, access and manipulation. The “old” data are representations of visual reality and human experience, i.e., images, text-based and audio-visual narratives – what we normally understand by “culture.” The “new” data is numerical data.

As a result of this mix, we get such strange hybrids as clickable “image-maps,” navigable landscapes of financial data, QuickTime (which was defined as the format to represent any time-based data but which in practice is used exclusively for digital video), animated icons – a kind of micro-movies of computer culture – and so on.

As can be seen, this particular approach to new media assumes the existence of historically particular aesthetics that characterizes new media, or “early new media,” today. (We may also call it the “aesthetics of early information culture.”) This aesthetics results from the convergence of historically particular cultural forces: already existing cultural conventions and the conventions of HCI. Therefore, it could not have existed in the past and without changes it is unlikely to stay for a long time. But we can also define new media in the opposite way: as specific aesthetic features which keep re-appearing at an early stage of deployment of every new modern media and telecommunication technologies.

5. New Media as the Aesthetics that Accompanies the Early Stage of Every New Modern Media and Communication Technology.

Rather than reserving the term “new media” to refer to the cultural uses of current computer and computer-based network technologies, some authors have suggested that every modern media and telecommunication technology passes through its “new media stage.” In other words, at some point photography, telephone, cinema, television each were “new media.” This perspective redirects our research efforts: rather than trying to identity what is unique about digital computers functioning as media creation, media distribution and telecommunication devices, we may instead look for certain aesthetic techniques and ideological tropes which accompany every new modern media and telecommunication technology at the initial stage of its introduction and dissemination. Here are a few examples of such ideological tropes: new technology will allow for better democracy; it will give us a better access to the “real” (by offering “more immediacy” and/or the possibility “to represent what before could not be represented”); it will contribute to “the erosion of moral values”; it will destroy the “natural relationship between humans and the world” by “eliminating the distance” between the observer and the observed.

And here are two examples of aesthetic strategies that seem to often accompany the appearance of a new media and telecommunication technology. (Not surprisingly, these aesthetic strategies are directly related to ideological tropes I just mentioned). In the mid 1990s a number of filmmakers started to use inexpensive digital cameras (DV) to create films characterized by a documentary style (for instance, Timecode, Celebration, Mifune). Rather than treating live action as a raw material to be later re-arranged in post-production, these filmmakers place premier importance on the authenticity of the actors’ performances. The smallness of DV equipment allows a filmmaker to literally be inside the action as it unfolds. In addition to adopting a more intimate filmic approach, a filmmaker can keep shooting for a whole duration of a 60 or 120 minute DV tape as opposed to the standard ten-minute film roll. This gives the filmmaker and the actors more freedom to improvise around a theme, rather than being shackled to the tightly scripted short shots of traditional filmmaking. (In fact the length of Time Code exactly corresponds to the length of a standard DV tape.)

These aesthetic strategies for representing the “real” which at first may appear to be unique to digital revolution in cinema are in fact not unique. DV-style filmmaking has a predecessor in an international filmmaking movement that begun in the late 1950s and unfolded throughout the 1960s. Called “direct cinema,” “candid” cinema, “uncontrolled” cinema, “observational” cinema, or cinéma vérité (“cinema truth”), it also involved filmmakers using lighter and more mobile (in comparison to what was available before) equipment. Like today’s DV realists,” the 1960s “direct cinema” proponents avoided tight staging and scripting, preferring to let events unfold naturally. Both then and now, the filmmakers used new filmmaking technology to revolt against the existing cinema conventions that were perceived as being too artificial. Both then and now, the key word of this revolt was the same: “immediacy.”

My second example of similar aesthetic strategies re-appearing deals with the development of moving image technology throughout the nineteenth century, and the development of digital technologies to display moving images on a computer desktop during the 1990s. In the first part of the 1990s, as computer’s speed kept gradually increasing, the CD-ROM designers have been able to go from a slide show format to the superimposition of small moving elements over static backgrounds and finally to full-frame moving images. This evolution repeats the nineteenth century progression: from sequences of still images (magic lantern slides presentations) to moving characters over static backgrounds (for instance, in Reynaud's Praxinoscope Theater) to full motion (the Lumieres' cinematograph). Moreover, the introduction of QuickTime by Apple in 1991 can be compared to the introduction of the Kinetoscope in 1892: both were used to present short loops, both featured the images approximately two by three inches in size, both called for private viewing rather than collective exhibition. Culturally, the two technologies also functioned similarly: as the latest technological “marvel.” If in the early 1890s the public patronized Kinetoscope parlors where peep-hole machines presented them with the latest invention — tiny moving photographs arranged in short loops –exactly a hundred years later, computer users were equally fascinated with tiny QuickTime Movies that turned a computer in a film projector, however imperfect. Finally, the Lumieres' first film screenings of 1895 which shocked their audiences with huge moving images found their parallel in 1995 CD-ROM titles where the moving image finally fills the entire computer screen (for instance, in Johnny Mnemonic computer game, based on the film by the same title.) Thus, exactly a hundred years after cinema was officially "born," it was reinvented on a computer screen.

Interesting as they are, these two examples also illustrate the limitations of thinking about new media in terms of historically recurrent aesthetic strategies and ideological tropes. While ideological tropes indeed seem re-appearing rather regularly, many aesthetic strategies may only reappear two or three times. Moreover, some strategies and/or tropes can be already found in the first part of the nineteenth century while others only make their first appearance much more recently. In order for this approach to be truly useful it would be insufficient to simply name the strategies and tropes and to record the moments of their appearance; instead, we would have to develop a much more comprehensive analysis which would correlate the history of technology with social, political and economical histories of the modern period.

So far my definitions of new media have focused on technology; the next three definitions will consider new media as material re-articulation, or encoding, of purely cultural tendencies – in short, as ideas rather than technologies.

6. New Media as Faster Execution of Algorithms Previously Executed Manually or Through Other Technologies.

A modern digital computer is a programmable machine. This simply means that the same computer can execute different algorithms. An algorithm is a sequence of steps that need to be followed to accomplish a task. Digital computers allow us to execute most algorithms very quickly, however, in principle an algorithm, since it is just a sequence of simple steps, can be also executed by a human, although much more slowly. For instance, a human can sort files in a particular order, or count the number of words in a text, or cut a part of an image and paste it in a different place.

This realization gives us a new way to think about both digital computing, in general, and new media, in particular, as a massive speed-up of various manual techniques that all have already existed. Consider, for instance, the computer’s ability to represent objects in linear perspective and to animate such representations. When you move your character through the world in a first person shooter computer game (such as Quake), or when you move your viewpoint around a 3D architectural model, a computer re-calculates perspectival views for all the objects in the frame many times every second (in the case of current desktop hardware, frame rates of 80 frames of second are not uncommon). But we should remember that the algorithm itself was codified during the Renaissance in Italy, and that, before digital computers came along (that is, for about five hundred years) it was executed by human draftsmen. Similarly, behind many other new media techniques there is an algorithm that, before computing, was executed manually. (Of course since art has always involved some technology – even as simple as a stylus for making marks on stone – what I mean by “manually” is that a human had to systematically go through every step of an algorithm himself, even if he was assisted by some image making tools.) Consider, for instance, another very popular new media technique: making a composite from different photographs. Soon after photography was invented, such nineteenth century photographers as Henry Peach Robinson and Oscar G. Reijlander[LM1] were already creating smooth "combination prints" by putting together multiple photographs.

While this approach to thinking about new media takes us away from thinking about it purely in technological terms, it has a number of problems of its own. Substantially speeding up the execution of an algorithm by implementing this algorithm in software does not just leave things as they are. The basic point of dialectics is that a substantial change in quantity (i.e., in speed of execution in this case) leads to the emergence of qualitatively new phenomena. The example of automation of linear perspective is a case in point. Dramatically speeding up the execution of a perspectival algorithm makes possible previously non-existent representational technique: smooth movement through a perspectival space. In other words, we get not only quickly produced perspectival drawings but also computer-generated movies and interactive computer graphics.

The technological shifts in the history of “combination prints” also illustrate the cultural dialectics of transformation of quantity into quality. In the nineteenth century, painstakingly crafted “combination prints” represented an exception rather than the norm. In the twentieth century, new photographic technologies made possible photomontage that quickly became one of the basic representational techniques of modern visual culture. And finally the arrival of digital photography via software like Photoshop, scanners, and digital cameras in the late 1980s and 1990s not only made photomontage much more omnipresent than before but it also fundamentally altered its visual characteristics. In place of graphic and hard-edge compositions pioneered by Moholy-Nagy and Rodchenko we now have smooth multi-image composites which use transparency, blur, colorization and other easily available digital manipulations and which often incorporate typography that is subjected to exactly the same manipulations. (Thus in post-Photoshop visual culture the type becomes a subset of a photo-based image.) To see this dramatic change, it is enough to compare a typical music video from 1985 and a typical music video from 1995: within ten years, visual aesthetics of photomontage have undergone a fundamental change.

Finally, thinking about new media as speeding up of algorithms which previously were executed by hand foregrounds the use of computers for fast algorithm execution, but ignores its two other essential uses: real-time network communication and real-time control. The abilities to interact with or control remotely located data in real-time, to communicate with other human beings in real-time, and control various technologies (sensors, motors, other computers) in real time constitute the very foundation of our information society – phone communications, Internet, financial networking, industrial control, the use of micro-controllers in numerous modern machines and devices, and so on. They also make possible many forms of new media art and culture: interactive net art, interactive computer installations, interactive multimedia, computer games, real-time music synthesis.

While non-real time media generation and manipulation via digital computers can be thought of as speeding up of previously existing artistic techniques, real-time networking and control seem to constitute qualitatively new phenomena. When we use Photoshop to quickly combine photographs together, or when we compose a text using a Microsoft Word, we simply do much faster what before we were doing either completely manually or assisted by some technologies (such as a typewriter). However, in the cases when a computer interprets or synthesizes human speech in real time, monitors sensors and modifies programs based on their input in real-time, or controls other devices, again in real-time, this is something which simply could not be done before. So while it is important to remember that, on one level, a modern digital computer is just a faster calculator, we should not ignore its other identity: that of a cybernetic control device. To put this in different way, while new media theory should pay tributes to Alan Turing, it should not forget about its other conceptual father – Norbert Weiner.

7. New Media as the Encoding of Modernist Avant-Garde; New Media as Metamedia.

The approach to new media just discussed does not foreground any particular cultural period as the source of algorithms that are eventually encoded in computer software. In my article “Avant-garde as Software” I have proposed that, in fact, a particular historical period is more relevant to new media than any other – that of the 1920s (more precisely, the years between 1915 and 1928). During this period the avant-garde artists and designers have invented a whole new set of visual and spatial languages and communication techniques that we still use today. According to my hypothesis,

With new media, 1920s communication techniques acquire a new status. Thus new media does represent a new stage of the avant-garde. The techniques invented by the 1920s Left artists became embedded in the commands and interface metaphors of computer software. In short, the avant-garde vision became materialized in a computer. All the strategies developed to awaken audiences from a dream-existence of bourgeois society (constructivist design, New Typography, avant-garde cinematography and film editing, photo-montage, etc.) now define the basic routine of a post-industrial society: the interaction with a computer. For example, the avant-garde strategy of collage reemerged as a "cut and paste" command, the most basic operation one can perform on any computer data. In another example, the dynamic windows, pull-down menus, and HTML tables all allow a computer user to simultaneously work with practically unrestricted amount of information despite the limited surface of the computer screen. This strategy can be traced to Lissitzky's use of movable frames in his 1926 exhibition design for the International Art Exhibition in Dresden.

The encoding of the 1920s avant-garde techniques in software does not mean that new media simply qualitatively extend the techniques which already existed. Just as is the case with the phenomenon of real-time computation that I discussed above, tracing new media heritage in the 1920s avant-garde reveals a qualitative change as well. The modernist avant-garde was concerned with “filtering” visible reality in new ways. The artists are concerned with representing the outside world, with “seeing” it in as many different ways as possible. Of course some artists already begin to react to the emerging media environment by making collages and photo-montages consisting from newspaper clipping, existing photographs, pieces of posters, and so on; yet these practices of manipulating existing media were not yet central. But a number of decades later they have to the foreground of cultural production. To put this differently, after a century and a half of media culture, already existing media records (or “media assets,” to use the Hollywood term) become the new raw material for software-based cultural production and artistic practice. Many decades of analog media production resulted in a huge media archive and it is the contents of this archive – television programs, films, audio recordings, etc – which became the raw data to be processed, re-articulated, mined and re-packaged through digital software – rather than raw reality. In my article I formulate this as follows:

New Media indeed represents the new avant-garde, and its innovations are at least as radical as the formal innovations of the 1920s. But if we are to look for these innovations in the realm of forms, this traditional area of cultural evolution, we will not find them there. For the new avant-garde is radically different from the old:

1. The old media avant-garde of the 1920s came up with new forms, new ways to represent reality and new ways to see the world. The new media avant-garde is about new ways of accessing and manipulating information. Its techniques are hypermedia, databases, search engines, data mining, image processing, visualization, and simulation.

2. The new avant-garde is no longer concerned with seeing or representing the world in new ways but rather with accessing and using in new ways previously accumulated media. In this respect new media is post-media or meta-media, as it uses old media as its primary material.

My concept of “meta-media” is related to a more familiar notion of “post-modernism” – the recognition that by the 1980s the culture became more concerned with reworking already existing content, idioms and style rather than creating genially new ones. What I would like to stress (and what I think the original theorists of post-modernism in the 1980s have not stressed enough) is the key role played by the material factors in the shift towards post-modernist aesthetics: the accumulation of huge media assets and the arrival of new electronic and digital tools which made it very easy to access and re-work these assets. This is another example of quantity changing into quality in media history: the gradual accumulation of media records and the gradual automation of media management and manipulation techniques eventually recoded modernist aesthetics into a very different post-modern aesthetics.

8. New Media as Parallel Articulation of Similar Ideas in Post WWII Art and Modern Computing.

Along with the 1920s, we can think of other cultural periods that generated ideas and sensibilities particularly relevant to new media. In the 1980s a number of writers looked at the connections between Baroque and post-modern sensibilities; given the close links between post-modernism and new media I just briefly discussed, it would be logical if the parallels between Baroque and new media can also be established. It can be also argued that in many ways new media returns us to a pre-modernist cultural logic of the eighteenth century: consider for instance, the parallel between eighteenth century communities of readers who were also all writers and participants in Internet newsgroups and mailing lists who are also both readers and writers.

In the twentieth century, along with the 1920s, which for me represent the cultural peak of this century (because during this period more radically new aesthetic techniques were prototyped than in any other period of similar duration), the second culturall peak – 1960s – also seems to contain many of new media genes. A number of writers such as Söke Dinkla have argued that interactive computer art (1980s -) further develops ideas already contained in the new art of the 1960s (happenings, performances, installation): active participation of the audience, an artwork as a temporal process rather than as a fixed object, an artwork as an open system. This connection makes even more sense when we remember that some of the most influential figures in new media art (Jeffrey Shaw, Roy Ascott) started their art careers in the 1960s and only later moved to computing and networking technologies. For instance, in the end of the 1960s Jeffrey Shaw was working on inflatable structures for film projections and performances which were big enough to contain a small audience inside – something which he later came back to in many of his VR installations, and even more directly in EVE project.

There is another aesthetic project of the 1960s that also can be linked to new media not only conceptually but also historically, since the artists who pursued this project with computers (such as Manfred Mohr) knew of minimalist artists who during the same decade pursued the same project “manually” (most notably, Sol LeWitt). This project can be called “combinatorics.” It involves creating images and/or objects by systematically varying a single parameter or by systematically creating all possible combinations of a small number of elements. “Combinatorics” in computer art and minimalist art of the 1960s led to the creation of remarkably similar images and spatial structures; it illustrates well that the algorithms, this essential part of new media, do not depend on technology but can be executed by humans.

---