BA Computer Visualisation & Animation
Innovations in Computer Animation
Photo manipulation, montage and mosaics
as an integrated visual effects solution
1. Abstract
2. Introduction
3. Research
3.1. Photographic manipulation, montages and mosaics
3.1.1. The work of David Hockney
3.2. Photo mosaics
3.2.1. The work of Chuck Close
3.3. Goal reassessment
4. System Design
4.1. Initial considerations into the solution
4.2. Final solution proposal
4.3. Proposed usage/output
5. Implementation
5.1. The solution overview
5.1.1. The main Maya plugin - mosaicFunction.mll/.so
5.1.2. Averaging the reference image - averageFunction.cpp
5.1.3. Re-sizing a folder of images - resizeFunction.cpp
5.1.4. Querying a specified image - queryFunction.cpp
5.1.5. Using a depth map - depthFunction.cpp
5.2. The different styles
5.2.1. Single thumbnail image mosaic
5.2.1.1. Shader creation
5.2.1.2. Colour palette
5.2.1.3. Brush styles
5.2.1.4. Colour controlling height
5.2.1.5. Animation sequences
5.2.2. Multiple thumbnail image mosaic
5.3. GUI
6. Analysis
6.1. Conclusion
6.2. Next step
6.2.1. Further studies into aesthetic styles
1. Abstract
This report introduces the reader to the possibility of successful integration between existing image manipulation and montage techniques and the exciting new age of digital content and creation. Discussing the styles and methods available and documenting the results of several investigations into photo montages and mosaic effects within digital worlds.
2. Introduction
This report will document and discuss personal investigations into a photo manipulation/montage effect pipeline that integrates fully into Alias’ Maya. The report will cover the initial research conducted into the subject, looking into the photo montage and mosaic styles, covering the artistic background of the medium and the influence on this project. As well as research into the artistic aspects of the project, the report will document the careful analysis of any currently available solutions that could inspire the functionality of the toolset.
The report progresses to outline how this solution’s design has evolved and which possible techniques were considered and for which reasons they may have been rejected or accepted for its continuing development.
Subsequently documenting how the final toolset has been both designed and achieved, breaking down the various parts of the pipeline and describing how the composite parts were accomplished and integrated alongside the main backbone of the tool.
Finally the report will detail the analytical and conclusive stages of the project. Including consideration for the functions of the toolset that have been successfully implemented, those that require more attention together with any that are problematic as they stand. Concluding with the areas of the tool that will be developed for future applications, forging the tool into a more complete and powerful solution.
The appendices will include the user manual for the plugin with descriptions on how to integrate the toolset into Maya and any sample variables that will allow for creation of a successful result. The second appendix will document the code that was written for the solution and highlight and acknowledge any code that has been sourced from a third party and used to provide functionality to the plugin.
3. Research
The first subsection of research documented, looks into the basic history and technique of photo montage and its development as a standalone artistic medium and investigates one of its main protagonists.
The research then diverts from the artistic to the technical, looking at solutions that are currently on the market and ways that these could influence the design, implementation and result of the final tool.
3.1 Photographic manipulation, montages and mosaics
The techniques of manipulating images can surely be dated to the first time a photograph was printed; dark room development can only ever be an approximation of the image as it was meant to be seen through the viewfinder. Image manipulation has been running parallel with photography since the first decades of photography in the mid 1800’s. Russell Roberts, head of photography at the National Museum of Photography, Film and Television, explains that “examples of images which claimed to be accurate depictions of events but were in fact highly stage managed” (2004).
The style of photomontage as is widely recognised today, however, can be traced back to the world wars. Various innovative and exciting techniques have been developed over the years; photographers have been interacting with objects placed directly onto the photographic plates for years. Double exposures and darkroom masking techniques, were all techniques developed very early along the relatively short life line of photography.
The montage style was set to carry on exploring fresh boundaries with a new group of artists, the Dadaists, at the helm in the 1920’s. Whist looking for new and exciting mediums they embraced montage work; a work process that allowed them to reject the prevailing drift towards abstraction, yet gave them the opportunity to breakaway from the remaining, more formal, figurative movements.
Photomontage gradually faded back into darkness until artists began experimenting in the 1960’s with new media and imagery in the Pop Art movement. However, montage’s popularity continues today with its powerful and striking applications embracing the new innovations in digital video, sound and multimedia imagery.
3.1.1 The work of David Hockney
Having fallen out of love with the medium he once pioneered in the 1980’s. Hockney now believes photography is in need of a serious creative injection. Sighting its increasingly digital form, the ease of use this brings with it and common misapplication as its downfall.
In an interview with the Guardian, Hockney states that “modern photography is now so extensively and easily altered that it can no longer be seen to be true or factual”. (2004)
It is sad to read such disheartening comments from an artist whose work has enthused so many. However ostracised his recent lack of interest in the medium makes him; his contribution to art and photomontages were a real inspiration both in the formulative stages of the project and as it has developed during production.
The images above (fig 1a, 1b) show a Polaroid composite of David Graves, done by Hockney in 1982. (Fig 2) shows another Polaroid composite by Hockney entitled “Kasmin”. They demonstrate the style perfectly that originally inspired the project, and the interest into implementing a similar technique for digital media was born.
Looking into more of Hockney’s work it became apparent that his style would be a massive aesthetic inspiration to the project. The idea to try and implement these styles into a post production technique that could be used with either, CG environments or captured live action footage, was evolving quickly. With the aim to inject some originality into what Hockney has lost so much faith in, it provided the drive that a successful project needs.
The research into Hockney’s work and his use of fragmented and disjointed images to create beautifully powerful montages allowed the research to naturally progress to photo mosaics.
3.2 Photo mosaics
Hockney’s work was a strong influence over the initial ideas into the look and feel that the project intended to explore, however it felt important to make sure the brief was kept open at this stage. This next section of research documents the collective studies into image based mosaics.
The term photo mosaic is often associated with the classic images of Bob Marley or the Star Wars characters created entirely from smaller images that we see so often.
These images above are taken from their website http://www.photomosaic.com/ who claim to be the pioneers in mosaic technology and a browse of their site reveals some interesting images. Figure 6 above, an image of Jesus constructed from images of the Dead Sea Scrolls. The ideas are generally the same as many other examples of photo mosaic work available but these seem to be executed with a greater degree of competency and aesthetic judgement. That same site posts a suspicious warning…
“Photomosaic® is a registered trademark of Runaway Technology. The Photomosaics software is patented (US Patent No. 6,137,498) and the Photomosaic "look and feel" are protected by the patent, copyright, and other intellectual property laws of the United States and other major countries. We protect these rights vigilantly.”
3.2.1 The work of Chuck Close
The artist Chuck Close is a master printmaker, who has, over the last 30 years, created some exciting pieces using simple grid systems as a basis for his work. This simple structure provides the means for simply based images that are incredibly intricate and enchanting in there final form.
Working on this grid system allows Close to divide up the canvas into sections that he can then analyse and develop further adding more and more layers of colour and texture into these disjointed sections but when viewed as a whole the pieces work brilliantly, His work is even more impressive and inspirational when you consider that none of Close's images are digitally produced.
This artistic style of mosaic is so much
more interesting and powerful than the usual mass produced mosaic images and
posters that are often for sale. They provide much more aesthetic pleasure and
could work brilliantly in a dynamically animated piece. It is a must that the
project explores such artistic possibilities as well as being a competent
technical solution.
3.2.2 Goal reassessment
Originally this section was intended to document any existing solutions of photo mosaic software but as the project has been evolving it has become less and less of an inspiration and a decidedly marked down part of the final design considerations will be geared towards this end of the spectrum. Having considered the visual research compiled here it now seems that these photo mosaic generators are not creating any inspirational or groundbreaking content. It would certainly be detrimental to the ethos of this particular project and report were it to follow down these narrowing paths.
From this point it seems the future of this project is increasingly becoming dominated by striving to create some interesting content. Perhaps at the detriment of producing a fully functional tool however it is certainly important to me as the author that the project looks towards innovation rather than striving for perfection
4. System Design
With all the research collated and documented, this section of the report will document decisions and considerations that were interrogated before the design of the final system could be drawn up. Subsequent to these possible methods of execution will be the final outline for the tool, describing the chosen architecture and approach coupled with the finished tool’s intended application.
4.1 Initial considerations into the solution
The amount of possible routes that could have been taken at this stage was overwhelming and the daunting task of considering the various languages, architectures and software solutions began.
The key aims that were taken into account at this stage were:
- Image handling libraries
- Output
- System portability
- Personal knowledge
The main considerations at this stage were taking shape in the form of programming languages and either internally associated image libraries or external 3rd party image libraries.
One of the strongest of these proposals came in the form of an external 3rd party resource, the image library “imageMagick”. The functionality that “imageMagick” provides is in the form of a collection of tools and libraries that provide image handling and manipulation functions to the C and C++ programming languages.
Looking into what the function set could provide it was promising to see support for all the familiar image formats. It was also encouraging to read that ImageMagick could perform any of the image I/O operations required for the brief as well as being able to handle image manipulation effects such as resizing or rotating, together with comprehensive colour correction tools.
However comprehensive the function set was, it soon became too restrictive, and doubts were beginning to be cast about the ability of the libraries to be manipulated to serve the needs of the project. It would be perfect base for a more rigid image retouching or managing brief but for this project it felt to restrictive in its creative outputs.
The second programming language that presented itself was OpenGL. The OpenGL programming language is used extensively in the development of interactive 2D and 3D graphics applications. Having been introduced and exposed to this language over the last two years it had obvious personal knowledge advantages.
OpenGL claims on its website that…
“…Since its introduction in 1992, OpenGL has become the industry's most widely used and supported 2D and 3D graphics application programming interface (API)…”
This would be very good for ensuring that the solution was extremely portable and system independent. Meaning the tool could be ported to any client specification machine without the end user having to have any other software to ensure it would run.
The website goes on to state…
“OpenGL fosters innovation and speeds application development by incorporating a broad set of rendering, texture mapping, special effects, and other powerful visualization functions”
This was another extremely attractive sales pitch from OpenGL, with the ability to use geometry, texture mapping and powerful lighting and camera solutions; it could provide some very exciting results.
The powerful features and cross platform support that OpenGL had to offer began to make it the favoured option for the implementation of the final solution. However it was also the trigger to cause a slight shift in project goals.
Ideas began to formulate about using a powerful geometry based solution to integrate the planned image effects into a 3D geometry rich environment. Allowing interactive interaction between created 3D computer graphics and 2D photomontage, image creation and manipulation techniques
4.2 Final solution proposal
With the new ideas and challenges in place it became apparent that perhaps the Maya API would be the best way to implement the solution to the level of CG interaction that had been envisaged.
It was now the aim of the project and its solution to investigate and explore several different and exciting methods for creating mosaic or montage effects, now less orientated towards a final product. It would be interesting to see what styles and results can be achieved.
Having been introduced to Maya’s API in a series of optional lectures held by Rob Bateman, it was clear that the combination of the Maya API and Maya’s MEL scripting language was a very powerful resource for the graphics programmer. Offering everything that the latest brief demanded, this combination became the first choice for the solution.
This was the first real test into the API and the results were promising. The API would provide the image handling capabilities whilst a Mel solution would provide the access to Maya's powerful geometry tools. This combination would create a good basis for creating the dynamic outputs that the brief had scoped.
4.3 Proposed usage
In an ideal, final release version, the toolset will be best suited to working with rendered sequences from a 3D package making use of the possibility of RGB, alpha and depth channel support.
For the tool to be most successful it needs to be able to integrate into an existing scene as well as a new one, meaning scene management is as important as the other areas. It is important that the output from the tools should be able to as impressive with 2D images either animated or still.
5. Implementation
5.1 The solution overview
The implementation of the toolset is broken down in this section of the report into their component parts; firstly discussing the API plugin and its integral commands and the MEL procedures that call them. Then dividing the solution up into its separate styles and documenting them within their own respective headings. Finally this section of the report covers the GUI that wraps all of this functionality up.
The tool is based on the ability to sample and/or average the pixel values of a specified point or area in a given image. The way this plugin has been designed is based on a simple grid design. The image to be sampled and averaged - the reference image - is opened and held in memory using the MImage class in the API, this image is then split into an imaginary grid, divided by a specified sample area width and height, the size of which controls the level of detail in the final image. This averaged pixel information can then be used to control the selection of a shader in Maya that closely correlates with the sampled pixel data for the corresponding area on the reference images as is being reproduced in the mosaic.
This is the very basic overview of the functionality of the tool, the subsequent sections and subsections go on to describe in-depth the main component parts described in the previous paragraph.
5.1.1 The main Maya plugin - mosaicFunction.mll/.so
The backbone of the toolset is the Maya API plugin - “mosaicFunction.mll/.so”. This plugin can then be broken down into several respective commands that provide the separate areas of functionality for the plugin - averageFunction.cpp, resizeFunction.cpp, queryFunction.cpp and the depthFunction.cpp. Each of these functions is registered and deregistered within the plugin.
5.1.1.1 Averaging the reference image - averageFunction.cpp
The first functionality of the project to be developed was the ability to specify and open an image in the Maya API. Using the MImage class, as discussed in the previous section of the report, as the image manipulation handler, the function was able to open and handle with any image format supported by Maya. With a way of image I/O available it was feasible to begin writing the functionality to give the code access to the pixels comprising the opened image.
const unsigned char* pix = image.pixels() + (w * py * 4 + 4 * px);
MIntArray result;
result.append(pix[0]); // r
result.append(pix[1]); // g
result.append(pix[2]); // b
result.append(pix[3]); // a
The extract above is taken from one of the API lectures from Rob Bateman, and was the basis of the first personal exploration into the Maya API. Here the char pointer const unsigned char* pix provides access to the pixel data image.pixels() + (w*py*4 + 4*px); of the array (width * height * depth) at the given pixel (py and px). This was exactly what was required for the function’s needs, however it would necessitate several nested “for loops” to allow the function to read every pixel in a specified sample area and average the result, and for this for each sampled area of the reference image.
Figure 1 below demonstrates the premise behind the approach.
Complications soon began to appear when trying to sample images that were not divisible by the specified sample area. Referring back to figure X above, the red area demonstrates the difficulty. Here the image is of dimensions 512 x 512 and the sample area is specified to be 20 x 20. This was acceptable for the first 25 sample areas along the image’s width but on the 26th sample area, it would try to sample pixels outside the array and would not evaluate as expected. This proved difficult to trace at first because it seemed to be working for certain size images and of certain aspect ratios until it was more thoroughly tested with stricter image resolutions. The method employed to solve this problem was to use the modulus operator.
if ((width % sectionWidth) != 0 || (height % sectionHeight) != 0) {
newWidth = (width - (width % sectionWidth));
newHeight = (height - (height % sectionHeight));
}
Where “width” is the width of the open image, “sectionWidth” is the width of the sample area and “newWidth” is the width to resize the opened image by. If the width of the image is not divisible by the “sectionWidth” then it will resize the image to “newWidth” without the excess pixels. This does result in the outputted image being a maximum of 5 pixels squashed in either width or height but this is negligible enough to be ignored for this release of the tool.
Given access to the image’s pixel information, the averaged RGBA values for a given sampled area could be computed. Assigning the value of the separate colour channels to separate integer variables allowed mathematical operations to be completed on the values and then placed back into the array for returning to the Mel script that had called it.
5.1.1.2 Re-sizing a folder of images - (resizeFunction.cpp and relevant Mel code)
It was important to allow the user to quickly and easily create the effects they required. It became apparent that the magnitude of thumbnail images required for a photo mosaic effect would quickly escalate. It would soon be impractical or at least frustrating and time consuming for the user, to request that all of the thumbnail images be resized already.
It was also important that the original images to be resized were acceptable in any of the main image formats and of any size. The user should simply select a directory of images.
It is worth noting at this stage that this version of the command only supports resizing the images without maintaining the original aspect ratio. This is due to the fact that at this time the toolset as a whole will only be working with square aspect ratio thumbnails. This function employs interaction between Mel and the API with the Mel proc.
proc int resizeThumbnails(int $nThumbs, string $thumbOrgDir, string $thumbNameTextFile)
This procedure in the main Mel script controls the calling of the resizeFunction command in the plugin. Returning an integer value of the number of thumbnails it has resized, it takes the arguments - number of thumbnails to resize “$nThumbs”, the pathname of the directory of images to be resized “$thumbOrgDir” and the pathname to the text file to write the names of resized images “$thumbNameTextFile”.
The procedure calls getFileList -folder $thumbOrgDir; and assigns the result to the string array $thumbList[];, this array now holds the names of the original images to be resized.
$nThumbs = (size($thumbList));
$fileId=`fopen $thumbNameTextFile "w"`;
for (; $i != $nThumbs; $i++){
if ($thumbList[$i] != ("Thumbs.db"))
fprint $fileId ($thumbList[$i] + " ");
}
fclose $fileId;
This array was then written to a text file that could then be opened within the API rather than having to deal with passing all the data across.
Once the file was opened within the ifstream it was possible to open and resize each specified image and finally save it as an .iff file.
5.1.1.3 Querying a specified image - queryFunction.cpp
The ability of this function is never really explored at this stage of the implementation and is relatively uninteresting. In a more final release increased functionality could be added to allow a more comprehensive query function. However as it stands the function purely takes away any of the calls to the other functions that ask for information about the specified image, notably the dimensions of the image.
5.1.1.4 Using a depth map - depthFunction.cpp
The idea to process a depth map was included to investigate the ability to represent 3D objects purely from a representation of 2D geometry. A depth map of the image that is being sample as the reference image must be supplied.
The depth map function works in a similar way to the averaging function. However because of the finer detail in the gradation of the depth map image, it became apparent that the averaging of values would be to harsh for the detail.
Figure 12 above demonstrates the problem that was encountered when averaging the depth map image. Figure 13 is an observation of the system working. The solution was to; sample the depth map image per pixel at the bottom left corner of each sample area, and this meant the results could still be calculated by using the sample area solution to control the colour and shader selection.
This technique has been explored in a similar way using the colour of the averaged area to drive the height of the tile, and is documented in section 5.2.1.4
5.2 The different styles
Having discussed the elements of the project that provide the underlying functionality of the solution, this section and its subsections serve to document the various aesthetic styles implemented in the project
5.2.1 Single thumbnail image mosaic
The style that harboured the most creativity was when using a single image as the thumbnail and repeating it several times to make up the final image. For the purpose of readability and continuity these single images will from now on be referred to as “brushes”. Taking the inspiration from Chuck Close’s work, it was interesting to see what could be represented with different brushes.
5.2.1.1 Shader creation
The method for the creation and assigning of shaders for this style of mosaic was somewhat dictated by the possibility of introducing animation sequences into the plugin at some stage. Because of the need to key colour change for the animation to be interpreted as motion in the final images, each separate tile would need its own shader.
This was implanted by looping through the amount tiles and creating a standard Lambert shader for each. The interesting challenge came when the shader was “chosen” for a tile.
As the script executes, for example on a static image, it will step through the tiles and create and assign a shader to each one, connection the shaders transparency to be driven by the submitted thumbnail texture. It is then required to set the shaders ambient and diffuse colour to tally with its corresponding sample area on the reference image.
This was achieved, for example on the greyscale colour palette by stepping through the various shades
for ($j = 0; $j <= 255.0; $j += (255.0/$nVariations))
and comparing it with the sampled areas averaged pixel value.
if ($averagedGrey[($i * 4)] <= ($j + (255.0/$nVariations)) && $averagedGrey[($i * 4)] > $j)
Where ($averagedGrey[($i * 4)] is always the R component of the returned pixel data, and because in this example it is dealing with greyscale, each RGB component will be equal.
When it finds that the sampled data from the reference image is within a particular bound it will, because Maya works with colour ranges between 1 and 0, divide the matching shade by 255 to get the equivalent colour value and assign it to the shader ambient and diffuse colour attributes. The reasoning behind this was that some of the earlier results that were being achieved were lacking in contrast and by adding in the ambient colour as well gave the various shades more definition.
$colourValue = <<($j/255.0),($j/255.0),($j/255.0)>>;
.color") -type double3 ($colourValue.x) ($colourValue.y) ($colourValue.z);
.ambientColor") -type double3 ($colourValue.x) ($colourValue.y) ($colourValue.z);
5.2.1.2 Colour palette
It was interesting to explore what difference the colour palette made. Its ability to accurately portray the reference image accurately became less important as it began to develop its own styles. The aesthetic qualities that often resulted seemed to make the represented image redundant and it became a piece in its own right.
To create these varying colours it necessitated the ability to specify the return values of the sampled pixels. It is important to note that at no stage in this process does the plugin alter the state of the original files and only the pixel data held in memory is changed.
With the ability to return the RGBA values of a given image, it was a similar process to returning the greyscale values. To calculate the greyscale from RGB values, the component channels (excluding alpha) can added together and then divided by three (RGB), for example the RGB values R - 144, G - 214, B - 65 can be converted to greyscale equivalent by calculating (144 + 214 + 65) / 3) = 141 and then assigning this value to each of the RGB components
Thus, by passing in a specific flag (“-g”) when the averageFunction command is called, the plugin can return either the RGBA values or Grey (with alpha) values.
5.2.1.3 Brush styles
As explained above this style of mosaic was the biggest fascination throughout the project. The idea behind the process was simple - to use one file texture, driving the transparency of multiple Lambert shaders. However the results that can be achieved with just one image are incredibly diverse.
At the beginning of the implementation this style began with using a simple “squiggle brush” shown in figure 14 on the left. As figure 17 demonstrates, it is possible to create some interesting images even with a small sample area.
Figure 14. Squiggle brush
These brushes, if saved with an alpha mask, can then be assigned to the string - $thumbnailImage = ("C:/pathname/thumb.tif");. This is then connected to the transparency of a file texture.
This technique is great fun and as can be seen, can create some fantastic images and with infinite styles simply by using a different brush.
This could also be seen as early investigation into painterly rendering styles and techniques, using any of the textures brushes from Adobe Photoshop or GIMP any number of oil or acrylic outputs could be achieved.
5.2.1.4 Colour controlling height
Alongside the depth map functionality it was interesting to see what could be achieved aesthetically by simply driving the height of a tile by its shade.
5.2.2 Animation sequences
Dealing with animation sequences had always been at the heart of the project even throughout the shifts in goals. As previously stated the shader creation techniques were paramount to the success of this area.
This system can be used with either computer generated imagery, captured footage or merely a group of images to be morphed between. However with the ability to consider alpha channels and depth maps in this solution it is generally better to use computer generated images.
Each tile is assigned its own shader that can then key the colour change as the reference image changes as the sequence animates. The next image in the sequence is read in and queried by the plugin returning the RGBA or greyscale values ready for the next iteration of the loop, and it follows this cycle, key the colour of the shader at each iteration until the last image in the sequence is reached.
Used in conjunction with the colour affects height method described above, the animation element of this solution is at its best. Keying the heights and colours of the tiles as the animation is recreated in a totally different light. This could be used to create some very interesting visuals especially put to music with the rise and fall of the tiles keyed to the animation that was driving it.
5.2.3 Multiple thumbnail image mosaic
The other style to be explored was the creation of mosaics from multiple thumbnails. Predominantly photos were used to test the functionality but it is entirely feasible to use any variation of images or even geometry to make up the montage. Although the tool is capable of producing these photo mosaic effects, they felt rather stagnant when compared with the dynamic feel of the same image mosaic results.
A separate script is included that controls this aspect of the project. Whilst it is based upon the same foundations as the proceeding styles it requires, as it stands the implementations of the Mel script are slightly different. The premise of this method is that it creates a shader and file texture for every single thumbnail specified in the resized thumbnail directory.
This is obviously a slow and cumbersome way to do this, but it does allow easy comparison between the available thumbnails and the reference image.
Most photo mosaic solutions will use colour correction to aid in the tolerance levels when looking for a thumbnail to use to represent the sampled area of the reference image. This feature has not been implemented in this solution because of the preference to the previous style of mosaic; however this could have been achieved by connecting the file out colour to either the ambient or diffuse connection on the shader and then controlling the colour of the shader when the thumbnail was in a certain tolerance. This would allow for fewer thumbnails to be required in total with obviously less overhead.
The results demonstrated here show that the functionality is there to create the photo montage styles but it soon began to get tiresome and slow computationally dealing with all the image textures.
The method of execution is interesting though because it is apparent how the technique could be used to compare and select other attributes to be controlled by the script, such as geometry.
// for every plane
for ($i = 0; $i != ($planeFaces); $i++){
// get the colour of the tiles rgba from $referenceImage
$cR = $averagedRGBA[$i * 4];
$cG = $averagedRGBA[(($i * 4) + 1)];
$cB = $averagedRGBA[(($i * 4) + 2)];
// loop through every thumbnail
for ($k = 0; $k != (($numberOfThumbs * 4)); $k++){
// if the tiles colour is within a 10 of the tiles rgb
if (abs($cR - $avg[($k * 4)]) < 50 &&
abs($cG - $avg[(($k * 4) + 1)]) < 50 &&
abs($cB - $avg[(($k * 4) + 2)]) < 50){
if (rand(15) > 5){
// assign the plane
select -r ("pPlane" + ($i +1));
sets -e -forceElement ("thumbnail" + ($k) + "SG");
break;
}
}
}
}
For every tile in the scene the script runs through and loops through every thumbnail in the scene (by this stage already connected to a shader) and compares the sampled data from the reference image to an array of averaged values of each thumbnail. It has a tolerance range of fifty pixels for comparing which can be changed depending on the number of available thumbnails.
The image to the left demonstrates the tool working
well, with a large number of thumbnails available an adequate output image is
available. Even so it became apparent that unless a radical new approach was to
be undertaken, such as a geometry selection system, it was obvious this style
was becoming redundant other than for learning purposes.
5.3 GUI
The GUI that was in development for this hand in release of the project, became to difficult to implement in the time remaining and was beginning to prove more trouble than worth.
The several directory paths that need to be specified in the script were difficult to control with the GUI at it was difficult to make sure all paths specified were correct. It makes more sense at this stage to submit the GUI as a work in progress at this stage in the project.
The GUI should be an important and integral part of the user integration for the tool and has always been a key factor in the brief. It is disappointing to see it failing at the last stage but in terms of stability it would have never been ready for a release this version.
The structure was to be designed as intuitively as possible, whilst mimicking the look of the Alias scripts to increase the look of a professional product.
However it doesn’t render the scripts totally un-editable to the user, and after some quick, and in some cases rather major, changes to naming conventions and programming structure, the scripts are understandable for parameter changes and tweaks from the user.
All of these parameters are discussed in the User manual in the appendix 1.
6. Analysis
6.1 Conclusion
As the toolset nears only a certain level of completion and as it is only just starting to produce some exciting results, it seems a difficult time to conclude and analyse what has been accomplished, when there are many more areas that are left unexplored.
As most projects do over the implementation and design stage, this project was constantly changing as the knowledge to produce the results was growing along with the functionality of what the program could do. Opening up new ideas and artistic inspirations and techniques that would be exciting to try and implement.
At the beginning of the implementation stages, the project was proposed as a limited exploration into a certain type of photo mosaic. As it stands now it has turned into a much wider exploration into the several styles of photo montage and mosaics.
It is much more varied than was originally planned and this may have led to the final product being at a less polished standard than if the project had focused on the one style. However it feels much more of an exciting and innovative project as it is now.
Having this openness to the final outputted results of the tool allowed many more influences and experiments to be brought into the plugin allowing for exciting results and massively increasing the potential for development in future use.
The backbone of the project is the Maya plugin that is called from MEL, written especially for the project, its function was to handle all interaction between Maya and the images. Using the MImage class provided a very stable platform for image I/O and the functions it provided for image manipulation were sufficient for the requirements of this project.
As the functionality of the tool increased, the more image processing was required, which meant expanding the plugin to include query functions for both the reference images and a depth map image if available. If the system were in development for final client release it would have been a lot more efficient to tie together some of these functions to provide a smaller footprint, lower overheads and faster execution speeds. However as the project was more an exploration into what was achievable within the brief, speed and efficiency took a less prominent role at the earlier stages of development but as personal knowledge and technique increased; naturally the coding practices became more efficient and structured.
The tool also required work in MEL to achieve the interaction between the sampled image data and the geometry. MEL can be very powerful but the scripts are often tedious to write and managing the files and variables can become convoluted. However the MEL script written for this project was enjoyable to code because of the variation in output that could be achieved with tweaks to the parameters.
The combination of the plugin, its internal commands and the Mel scripts were a powerful solution and allowed a wide range of out put images to be experimented with quickly.
In general the project has been a successful one in terms of knowledge gained and results achieved. Although the level of polish on the “final” tool to be submitted is less than perfect it demonstrates the achievements that have been pioneered and that it could be developed into a successful, fully integrated plugin with which extremely interesting effects could be achieved and integrated into a geometry based scene for object interaction or simply for creating exciting montages from pre-rendered frames or captured footage.
In a project as open as this one, with so many styles to be inspired by it feels as if my project has fallen somewhat short of what it could have been, but purely because there is so many variations on the theme that could be looked into. However the feeling at the concluding stages of the project is one of satisfaction at having developed a tool that is capable of producing some great visuals based on a powerful and stable code base. One of the major aims thought-out the project was for the tool to be totally artist friendly. On completion it has been achieved by giving the user total control, albeit at a basic level at this stage, over the style of the output image. Allowing the user to control a vast range of parameters gives the tool almost infinite possibilities for what could be created through user control.
The subsequent section goes on to explore what some of these might be and where the plugin will be taken next.
6.2 Next stage
It is often too easy to discuss conclusions in terms of time constraints placed upon the project and how restrictive they were to the successful completion of the project. However only certain things are possible in a finite amount of time and it is important to outline those areas that were not completed or those that are planned for future releases of the toolset.
The system as it now stands is still very much in its early stages of final development, with the emphasis having been on background development and the main functions being developed. For it to be implemented into any sort of production pipeline it will needs a lot of work in terms of overall speed and scene management alongside enhanced features for image output. The user interface is adequate for its current stage but to make it usable to artists with perhaps little Maya experience, it will need to be improved.
As the project has developed and it became more and more a vehicle for exploring the various styles associated with photomontage rather than a rigid photo mosaic creator. However there are still more aesthetic styles that could be explored and integrated.
6.2.1 Further studies into aesthetic styles
The style that would be next to be integrated would be along the lines of Hockney’s photomontages, using multiple parts of the same image to create a disjointed, larger version.
These could be approached in several ways; one method would use geometry and the second would use images.
A 3D object could be constructed of, for example, the desk in Hockney’s piece figured in the research section of this report, and a camera created and its render area set to a certain size, this could then be controlled, by code, to move gradationally around the object rendering it from different angles and positions until the whole object had been covered. The rendered images could then be renamed with a naming convention that stipulates the position they should be reassembled. This could work well utilising the integrated depth buffer functions that the plugin.
To support this function the plugin would need to be able to control the camera’s changing position and rotation. It would also need to call the rendering algorithm at the correct locations. The number of sampled renders would dictate the continuity of the final image and could be controlled to achieve some very interesting results.
The second method of tackling this approach would be based on existing render sequences, still images or captured footage. The image would be opened and handled using the MImage class in Maya API and the pixel array accessed and stored in a custom created array that could then resize or modify the pixel data and then write it to a new thumbnail image file, this could then be reintegrated with the existing plugin and handled as usual.
This would require only small modifications to the existing code to allow it to write the pixel array back to a new file using the MImage function: writeToFile( ).
Extending the features that I would like to implement in the future versions of the toolset would look into working with actual 3D geometry, and moving the normally 2D world of photographic manipulation and montage into the 3D computer graphics platform.
To conclude on a personal note, the project has been an excellent vehicle for personal learning and development. It saddens me that the project as it is handed in is not in a user friendly form and it is difficult to demonstrate any real output as it was just beginning to produce some exciting results.
It feels somewhat that for all his hard work and effort at the end of it all I have little to show for my efforts but it is comforting to know that the functionality and the test images I have produced express what is capable in the next version of the tool.
A0. User Manual
This appendix in the report is a copy of the README file included with the project submission.
The lack of GUI presently available for the system makes this task more challenging than would be had the user been able to interact with the code more gracefully.
For the convenience of the user a sample directory has been included with the submission demonstrating a good way to set up a folder hierarchy for effective file and data management. Included within this is a selection of images that have already been tested on the system and are know to produce good results.
The program is available for both Win32 systems and Linux OS.
The plugin mosaicPlugin.mll/.so must be loaded in to Maya using the plug in manager to register and deregister the commands officially.
With the plugin successfully loaded the user can then open the required script into the script editor. To explore the main script based on same image thumbnails, load the sameThumb.mel file.
// choose a style...
// 0 = black and white - same image thumbnail...
// 1 = greyscale - same image thumbnail...
// 2 = full colour - same image thumbnail...
int $style = 2;
// choose whether the thumbnails have any randomness to them...
// 0 = NONE (defualt)...
// 1 = random rotation...
// 2 = random scale(small changes)...
// 3 = BOTH scale AND rotation...
int $randomStyle = 3;
// choose whether the thumbnails height is affected by its colour
// (cheap depth map affect if there is no depth map available)...
// 0 = NO (defualt)...
// 1 = YES...
int $colourAffectHeight = 1;
// choose whether there are multiple reference images...
// 0 = NO (defualt)...
// 1 = YES...
int $isAnim = 1;
// set the time inbetween frames
int $frameDurationBetween = 5;
Here are the main parameters to control the final output of the plugin, the values here are fairly self explanatory with the help of the comments. However the main variable to change is $style this controls the colour palette of the image.
At the moment it is only style 1 that supports depth maps and colour affect height so take note when selecting the options
References
Bibliography
Gould, D.A., 2003. Complete Maya Programming - An Extensive Guide to Mel and the C++API
OpenGL Review Board, OpenGL programming guide - 4th addition