Reflection Scene rendered by our combined renderer

I’ve been studying computer sciences for three years and recently I finished writing my bachelor thesis. The topic was implementing reflections and refractions in the context of mixed reality. Basically, mixed reality means adding virtual objects into a video stream (of real environment). One of the goals is to make the resulting images look as realistic as possible. Naturally this includes reflection and refraction effects.

In my work I utilise the framework implemented in the RESHADE project. There was already a mixed reality renderer based on a Direct3D rasterizer. My task was to work on a ray tracer using OptiX and merge the resulting image with the rasterizer’s image. OptiX provides a ray tracing framework for the graphics card.

Glass Bunny Scene

Generally a ray tracer is slower than a rasterizer, but it has better quality with reflections and refractions. Because of that, we used a combined method. The ray tracer is working only on transparent and reflecting surfaces. This is achieved by a mask, which is created by the rasterizer in a separate render pass. This mask indicates, where reflecting or refracting objects are.

Especially with virtual flat mirrors there are very strong artefacts (see reflection scene, right mirror). This is due to lack of information. We are using the video frame, the environment map and the textured model of the real scene to gather information for the reflected ray.

Other problems we encountered were artefacts due to different shading algorithms in the rasterizer and ray tracer, and various issues with implementation due to the tone mapper.

When I was more or less finished with my work, I discovered that somebody else in the project was working on the implementation of reflections and refractions using the rasterizer. This was cool, because we were able to compare the results. Usually rasterizing is much faster than ray tracing. Surprisingly this was not the case with all scenarios. The reflection scene (first image) was faster with the combined method, the transparent bunny was slower. Especially with the bunny scene, I believe that the main drawback in speed was accuracy. The ray tracer also computes inner reflections, which causes a lot of additional work. I didn’t manage to benchmark this (rendering without inner reflections), because I was running out of time, but maybe I will do it yet..

Thanks to Michael Wimmer for making this work possible and special thanks to Martin Knecht for being an excellent supervisor and helping with all the questions .

Finally, if you are interested into more details, here you can find the thesis..

I quickly coded something together, so that I was able to prove, that my idea would work.

Lindenmayer systems are some kind of a grammar, they work like this: You have an alphabet of letters, in this case these letters are simple short lines, then you have productions and a rule: Always produce all letters. These productions eat one letter and produce ether no, one or several new letters. There are several classes of such productions, one of the simplest would be “A -> AA”. More sophisticated productions, like in the brush, can contain parameters, if clauses, calculations, random values etc. Basically it would be possible to have several different productions, but I chose to have only one and in turn make it more powerful.

Here is an example of a production, it is used in one of the brushes:

if{letter[branched] == false} {
newLetter = newLetter();
newLetter[position] = variant(letter[endPosition]);
newLetter[angle] = variant(newLetter[angleToSun]);
newLetter[length] = 5;
letter[branched] = bool(true);
}

if{letter[age] > 1} {
letter.delete();
}


It actually creates just a line.

At first the productions were plain C++ code, but that is very cumbersome, as you have to build, install and start krita for testing. The next thing I tried, was to use QtScript and write the productions in ECMAScript (also called JavaScript). It took about 5 to 7 hours until I proved, that it was way to slow.

So I decided to implement my own little scripting language. The interpreter is about 700 lines of code, including some tests. It’s certainly not as powerful, as ECMAScript, but it’s fast and okayish for this use case.

There is an if clause, which supports &&, there is a rand() function, a function for mixing angles, some very basic math, bool and float data types, you can create new letters and delete old ones. It’s possible to write new parameters (“branched” in the code above), edit some default ones (position, angle, length..), which will be used for drawing and access some computed ones (endPosition, angleToSun, distanceToSun, age..).

I have to write some documentation, expand the possibilities of the language and work on the configuration interface. But for now you can already test the current code, it’s in the krita-lbrush-adamc branch in the calligra repository.

Here is a final picture..

The presets for this brushes are included in the repository

Well, we added loads of new features, mainly graphical ones:

• lens  flares and sun dazzle effect
• moving space dust (white circles), which are moving randomly, this is an simple particle system.
• explosion and fire stream effects (more complicated particle systems)
• environment mapping (on the ships and station)
• bloom
• music and sound effects
• some power ups (armor, rockets, boost)
• …

Here is a video, we made for the presentation. Unfortunately it’s low quality, I hope to get a better one tomorrow, but it’s probably gone..

Update: We found a better HD quality Video

I’ve made a short video, where you can see bullet physics in action (at the end of the video you can see bullet debug mode enabled).

Music by Bézèd’h, music and video are CC licensed.

This term i’m attending the second computer graphics course in my university. The goal is to program a 3d game in OpenGL. We (Felix and me) chose to program a split screen flight duel game, as this is a simple game concept, but it’s still very extensible graphics and game logic wise.

The first step was to setup OpenGL etc. (GLEW, GLFW), which produced this first screen shot in begin of April.

We have worked hard and now we have implemented model loading (Assimp), shading, texturing, controls and split screen. Looks good for our first deadline

Next steps will be to implement some more game logic, in game display of hull state, frags and menu and then physics. We will use CEGUI and Bullet for these tasks..

There are no icons yet, but enkithan promised to create some, thanks . It’s currently not yet possible to extract the curves as data and i still have to port the current krita curve (some kind of function curve made by Dmitry Kazakov) to this new widget and integrate it in Krita.

I was told, that Dmitry has created an extendible api. If i can reuse that, it would be possible to replace all widgets at once, which in turn would mean, that the new curve could appear in Krita 2.3. Unfortunately i broke my ankle last week, while climbing in Croatia, and i have to undergo a surgery next Thursday or Friday, so maybe i won’t be able to code in the next week.

I was doing very much small changes, tweaks, bug fixes, layouting code and much more. The last bigger addition was the minimal shade selector, I will talk about it in a few lines.

But first here is a small summary, of what i was doing since the last blog post: I’ve connect the colour selectors to the resource manager, made the settings save to kritarc, added a tab in krita preferences (thanks Boud for the new registry, that was needed for that), added a colour history, moved the settings button to a more space efficient place, implemented drag and drop for colours, shortcuts for quick access and an api for setting a colour. The last point was actually much more difficult, than it sounds, because of the way, the selectors work with colour spaces. The selector has to search for the colour, that generates the user visible colour by using the colour spaces. Here is an attempt to explain it graphically.

Lets turn to a slightly more interesting thing, the minimal shade selector. Probably some pictures are better than 1000 words

It’s the bottom part with the gradients / patches.

Here is the configuration dialog.

And the combo box for selecting the gradients.

You can select one of the predefined gradients or configure one in the highlighted area.

That’s it for now..

Next Tuesday I’m going on a motorcycle trip to Constanta, which certainly will take a few days. So in that time I will be offline. Hopefully the weather will be ok

All selector types from this site are implemented now. All of them support colour spaces, which is important, if using cmyk. There is quite a difference in what you are probably used to, as you can see here:

This is of course still work in progress. I plan to change the layout of the selectors dynamically according to the space provided. For instance the horizontal slider of the wheel selector will be a vertical one, if the width is greater than the height. This way the widget will be very space efficient.

I also made the MyPaint shade selector colour space aware. Here is a direct comparison between cmyk at the top and rgb at the bottom.

As you probably see, this widget is not antialiased. It’s not as simple as setting a flag in Qt, but during programming the standard selectors, I’ve learned, how to make it antialiased. So in the final version there will certainly be a MyPaint selector with antialiased edges. If I have time, I will also happily implement some of these ideas.

I approached this from two sides: a space efficient and configurable layout, so that the user can select whichever size fits him best and an optional zoom, so that the user can make the widget small and yet simply make the chooser bigger, if he needs it.

Edit2:
The colour patches below the selectors represent common colours from the image and on the right there will be a history of last used colours..

So here are some screenshots, on how it looks like currently.
This is a ‘big’ layout,

‘small’ and space efficient,

a popup with a bigger selector,

and finally a page from the settings window.

Not all of the settings are connected already, the colour selector (triangle in the images) is just a dummy and the whole thing is not connected to the painting colour.
But I hope to get some comments on the layouting, maybe some more ideas and so on. I’ve just enabled building of this plugin in trunk, if you want to test it, you probably also have to enable this docker in settings menu -> dockers -> Color Selector Ng.

Edit:
This is still work in progress, next steps will be to gather some comments on usability, refine layouting and move on to implement pigment, colour selectors and so on. It’s not even half time of GSoC

The algorithm is a slightly modified version of median cut and it is quite the same as the one used in JFIF jpeg library for reducing colours and indexing images. There is a quite good and easy to understand paper from leptonica on this topic. Thanks to pippin for guiding me to median cut .

I also ported the shade selector algorithm from MyPaint to Krita. This is already a docker in Krita, but it isn’t connected yet:
.

Also, Pigment is not yet used, which is important to support colour spaces properly. Anyway, the next step will be to implement all widgets with dummy graphics, so Pigment and connection must wait.