Painting to Texture on iOS

Draw Something has continued to do very well in the App Store and now we’re seeing more derivatives — apps and games with basic painting and sketching capabilities. Last weekend I had some fun playing around with a basic painting setup, just to see how much brush (pun intended) I’d have to go through to get to the painting picnic.

On iOS, there are really only a couple of ways to implement a painting app — Quartz or OpenGL ES (there’s a nice little walk-through using Quartz here, and Apple put together a cool little OGL example called GLPaint here).

It should be relatively clear that the OGL approach is cleaner, more flexible and a bit faster. But while GLPaint is a nice place to start, it’s not very app- or engine-friendly in the context of a full-fledged OGL app. Since the point of GLPaint’s example code is demonstrate how to do basic painting, it has no need to consider the rest of your OGL surface’s render loop, nor does it concern itself with other important engine pieces, like your OGL redundancy state checker, sorting and synchronization issues between render and update calls, and most important, clearing the buffer.

That last bit really is most important because a nicely-performing painting app should never clear the buffer. Doing so will quickly slow things down to a slide-show. This should be obvious: In order to paint to the screen — whether you’re using GL_POINT_SPRITE_OES or rolling your own quads — you’ll need to draw a ton of sprites on-screen to get a continuous line of color and/or texture. If you clear every frame, you have to re-draw every frame, and voila, you’ll have molasses in less time than it takes to launch the simulator. If you don’t clear, you’re only drawing a handful of new sprites each frame.

The GLPaint example does this — it doesn’t clear the buffer. However in a real-world app, you must clear every frame in order for anything else — GUI elements/textures, mesh rendering, camera changes, etc. — to work. Hence the conundrum — you need a nice, normal clear/render loop but you also need a render-only call each time you want to paint.

Luckily there’s a straightforward solution: painting to a texture, then render that texture in your normal render loop. And setting up a texture to paint to is easy, by attaching it to an FBO. For example, a full-screen texture buffer:

- (GLenum) CreateRenderTexture
{
    m_texW = [[UIScreen mainScreen] bounds].size.width;
    m_texH = [[UIScreen mainScreen] bounds].size.height;

    glGenFramebuffers(1, &m_texFrameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, m_texFrameBuffer);

    glGenTextures(1, &m_texTexture);
    glBindTexture(GL_TEXTURE_2D, m_texTexture);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_texW, m_texH,
        0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
        GL_TEXTURE_2D, m_texTexture, 0);

    return glCheckFramebufferStatus(GL_FRAMEBUFFER);
}

From there it’s simply a matter of drawing to the texture and only clearing the texture when you need to, e.g., by calling a function like this:

- (void) StartTextureRender:(BOOL)clear color:(COLOR)color
{
    glBindFramebuffer(GL_FRAMEBUFFER, m_texFrameBuffer);
    if (clear)
    {
        glClearColor(color.r, color.g, color.b, color.a);
        glClear(GL_COLOR_BUFFER_BIT);
    }
    glViewport(0, 0, m_texW, m_texH);
    // setup ortho matrix, render and client states here...
}

One of the cool things about Draw Something is that it records and replays your drawing. This is relatively straightforward functionality to implement, and GLPaint kinda-sorta does it as a nice bonus. However their implementation is on the oddball side and a bit shy of more readable, that-makes-sense-to-me production code. A clearer way to implement it is to do a standard 2D lerp between the current touch (as you move your finger on the screen) and the last touch. Then record the time it took between finger-down and finger-up for playback later. For instance:

- (void) Draw:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    end         = Vec2(x, y);
    dist        = Vec2Dist(start, end);
    num         = (int)dist;
    if (num > 0)
    {
        startVert   = numVerts;
        numVerts    = MaxInt(numVerts + num, MAX_PATH_VERTS);
        for (int i = startVert; i < numVerts; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        time += [Engine GetTimeElapsed];
    }
    start = end;
    [self DrawRender];
}

Below is the entire class (note that several of the vars are structs elsewhere in the engine, but you get the gist).

// INTERFACE
#define MAX_PATH_VERTS  20000
@interface Path : NSObject
{
@public
    VEC2            verts[MAX_PATH_VERTS];
    int             numVerts;
    int             startVert;
    VEC2            start;
    VEC2            end;
    VEC2            cur;
    Texture*        texture;
    COLOR           color;
    float           size;
    float           tick;
    float           time;
    int             num;
    float           dist;
    BOOL            replaying;
    int             vertCount;
    int             curVert;
    int             endVert;
}
@property (nonatomic, readwrite) BOOL replaying;
- (id)   initWithColorTextureSize:(COLOR)c texture:(Texture*)t size:(float)s;
- (void) DrawStart:(float)x y:(float)y;
- (void) Draw:(float)x y:(float)y;
- (BOOL) Replay;
- (void) ReplayStart;
@end

// IMPLEMENTATION
@implementation Path
@synthesize replaying;
- (id) initWithColorTextureSize:(COLOR)c texture:(Texture*)t size:(float)s
{
    if (!(self == [super init])) return nil;
    numVerts        = 0;
    texture         = t;
    color           = c;
    size            = s;
    return self;
}
- (void) DrawRender
{
    if (num > 0)
    {
        glEnablePointSprite(GL_TRUE, size);
        glSetTexture(texture.index);
        glSetColor(color.r, color.g, color.b, color.a);
        glSetVertexPointerEx(&verts[0], sizeof(VEC2), 2);
        glDrawArrays(GL_POINTS, startVert, num);
    }
}
- (void) DrawStart:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    verts[0]    = start = end = Vec2(x, y);
    numVerts    = num = 1;
    startVert   = 0;
    time        = 0;
    [self DrawRender];
}
- (void) Draw:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    end         = Vec2(x, y);
    dist        = Vec2Dist(start, end);
    num         = (int)dist;
    if (num > 0)
    {
        startVert   = numVerts;
        numVerts    = IEMaxInt(numVerts + num, MAX_PATH_VERTS);
        for (int i = startVert; i < numVerts; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        time += [Engine GetTimeElapsed];
    }
    start = end;
    [self DrawRender];
}
- (BOOL) Replay
{
    if (replaying)
    {
        tick    = Max(tick + [Engine GetTimeElapsed], time);
        curVert = endVert;
        endVert = (int)Max(Lerp(0, numVerts, tick / time), numVerts);
        dist    = Vec2Dist(start, end);
        end     = verts[endVert];
        for (int i = startVert; i < endVert; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        start = end;
        int count = MinInt(endVert - curVert, 1);
        if (count > 0)
        {
            glEnablePointSprite(GL_TRUE, size);
            glSetTexture(texture.index);
            glSetColor(color.r, color.g, color.b, color.a);
            glSetVertexPointerEx(&verts[0], sizeof(VEC2), 2);
            glDrawArrays(GL_POINTS, curVert, count);
        }
        replaying = (endVert != numVerts);
    }
    return replaying;
}
- (void) ReplayStart
{
    curVert   = 0;
    endVert   = 0;
    replaying = (curVert < numVerts);
    if (replaying)
    {
        tick  = 0;
        time  = Min(time, 0.001);
        start = verts[0];
        end   = verts[1];
    }
}
@end

An NSMutableArray of multiple instances of this class is kept by the caller; each instance is born on finger-down (where we set color, brush texture and size) and dies on finger-up. Replay is easy — essentially just a programmatic rendering of the verts that were previously recorded while iterating over the NSMutableArray, handled with a flag in the Render() function. Below is the basic idea.

- (void) TouchDown:(float)x y:(float)y
{
    if (curSize == -1) // we're erasing here
    {
        [Engine StartTextureRender:YES color:curBackColor];
    }
    else
    {
        [Engine StartTextureRender:NO color:curBackColor];
        curPath = [[Path alloc] initWithColorTextureSize:curColor
                                                 texture:curTexture
                                                    size:curSize];
        [paths addObject:curPath], [curPath release];
        [curPath DrawStart:x y:y];
    }
}
- (void) TouchMove:(float)x y:(float)y
{
    [Engine StartTextureRender:NO color:curBackColor];
    [curPath Draw:x y:y];
}
- (void) TouchUp:(float)x y:(float)y
{
    curPath = nil;
}
- (void) Render
{
    [Engine RenderToTexture];
    if (replaying)
    {
        [Engine StartTextureRender:NO color:curBackColor];
        if (![curPath Replay])
        {
            curPath = nil;
            for (Path* m in paths) { if (m.replaying) { curPath = m; break; } }
            replaying = (curPath != nil);
        }
    }
}

One of the cool things about using OGL for painting and sketching is that you can very easily change up the brush texture, for nice Photoshop-like texture brushes (care should be paid to how you setup the blending, however, due to pre-multiplied alpha on iOS). While it’s possible to do this with Quartz, it’s much easier to grok using OGL. And of course you can do silly/fun stuff like paint a background behind your 3D orc model (maybe there’s a game idea in there somewhere, hmm — ok, maybe not).

Racing to the Bottom in Flatland

Racing to the Bottom in Flatland

Like a lot of Americans, my wife and I love movies. I’d say we’re above average film consumers, due in no small part to my love of theatre and film from an early age, as well as her years as a performance artist documenting much of her work on video and/or film.

So a few years ago when we first signed up for streaming movies via NetFlix, we were excited. At the time we were renting a house with a Blockbuster less than a mile away and had plenty of DVD options for re-viewing, but the idea of thousands upon thousands of movies at our fingertips every month, for less than the price of a single DVD or a couple of movie tickets, had us salivating.

We wound up streaming fewer movies than we anticipated, and it was a reminder of the power of perceived value as well as a lesson on the deceptive nature of choice. After the first couple of months the routine degenerated into:

Wife, in the kitchen, working on some incredible dish for dinner (she’s a serious gourmet cook): “Honey, what do you want to watch?”
Me, on the sofa already, flipping through all the crap on cable: “Oh, I dunno. What are you in the mood for?”
Her: “How about something on Netflix?”
Me: “I’ll check.”
Her, five minutes later: “How’s it going?”
Me: “Well, say, are you in he mood for something in particular?”
Her: “Oh not really, whatever you’d like.”
Me: “Maybe a campy 80’s flick.”
Her: “Sure!”
Her, ten minutes later: “So, are you finding anything?”
Me: “Hmmm, well, there’s a bad Tom Cruise movie, nah, I’m not in the mood. How about Willow? Oh, wait we have that on DVD, right?”
Her: “I believe so, but anything’s fine for me. I could maybe go for a disaster movie.”
Me: “Ok, that’ll work.”
Her, 10 minutes later, dinner’s almost ready: “So are you finding something?”
Me: “Well I don’t know, not really. There are plenty of action and disaster flicks. What do you think about Woody Allen?”
Her: “That’s not exactly an action movie. But sure, ok. Anything’s fine at this point.”
Me: “Alrighty, let me just look a little more.”
Her: 15 minutes later, after one or two more rounds of Q&A: “Dinner’s ready. What are we watching?”
Me: “Yeah, hmmm. Nothing’s really jumping out at me. Why don’t we catch up on Stewart and Colbert instead?”

This was the script for many an evening until it dawned on me that I was horrible, like most people, at choosing from too many options. The movies were all right there. There was an image of a movie poster, a star rating, a description, more info on the film if I wanted to view it. It wasn’t difficult to decide but the number of choices, low cost and easy access made it impossibly difficult. And it was Flatland (double entendre intended) — the structure of the presentation was not designed to lead me to a decision, it was all there in a single dimension of indistinguishable value.

My realization was not novel — there are a ton of complex psychological, sociological and cultural fractures in the bedrock of choice, some of which are discussed in books like The Paradox of Choice, The Tipping Point and more recently, The Myth of Choice (which was, by the way, written by Kent Greenfield, who hails from my hometown in Kentucky and was a couple of years ahead of me in school. Kent is a brilliant thinker and the real deal).

Choice, and in particular the relationship between choice and value, can be so tricky. For example, I know that if have to get up off my arse, get in the car, drive to Blockbuster, browse physical shelves, pay several bucks for a movie, then bring it home with the understanding that I’d better return it tomorrow or be penalized, I’m going to damn well watch the movie, and probably enjoy (or hate) it more. I will value the movie more (even if I don’t like it), because I had to give more to get it.

Perhaps the most important thing about the physical store is that it’s not Flatland. There are plenty of cues in an intuitive 3D world guiding my decision. As I walk the aisles, glance at posters, watch previews on LCD screens, browse end-caps, consider specials, drill-down by category and alphabetical then read the backs of boxes, I am continually operating in a sort of risk/reward environment at a granular, unconscious level. I have to put physical and mental effort into finding a movie, but it (usually) doesn’t feel like work. There is a natural discovery mechanism in play.

In the digital/virtual world of Netflix, discovery is more complicated and less intuitive. My instincts are muddied by a 2D world of flat images in a layout that poorly mimics the real world. The trade-offs between effort, choice and value are less clear. In terms of risk/reward, I start out risking little as I navigate a Flatland of seemingly endless choices. As time goes on, I realize that time is what is at risk. Choices have less value to me. Quality (the reward) becomes more and more important in order to mitigate risk. But my choice, or expected choice, has less and less value the more I risk because the choices are all flatly presented without curation or authority. It’s almost recursive, and instead of a race to a top choice, it’s a race to the bottom. If I pick something, it’s because my need to see any movie now outweighs my desire to be delighted by a particular movie.

The same is true for non-traditional games — games that are delivered digitally and have no real world discovery process (aside from word-of-mouth). This is just about everything in gaming right now except for the big-budget, massive IP-holding holdouts, on consoles and some PC games, in the traditional retail space. Discovery is truly a Flatland, for all the same reasons as streaming videos. It’s a race to bottom in terms of choice, but perhaps even more interesting, content — the games themselves — have also been racing to the bottom.

Discovery is a huge problem in the industry right now. Because mobile devices and digital distribution have eclipsed the traditional retail game business, we’re struggling to understand it all – fundamentally because our game-playing customers have too many choices and, for all the reasons above, this causes the perceived value of the content to do down. But the kicker is this: lower perceived value results in lower profits, which in turn results in lower-quality content. It’s a race to the bottom.

Discovery has become the domain of companies with enough money to sell content in Flatland. Since there is little hierarchy or thoughtful segmentation at point of purchase (not said lightly, by the way, because this is a huge problem to overcome), it’s like going into the physical store with a row of titles a mile long, one end-cap and the “best-sellers” shelved within the first 20 feet. Few customers will walk a mile to discover something new; more important, the sheer size and number of titles devalues the entire shelf.

As noted in Jeremy Liew’s recent post, Discovery is the problem in gaming, game design is incrementally improving and we are starting to see bigger budgets and somewhat higher quality, while distribution continues to be easy (the rote barrier-to-entry is still low, and the industry is not competing on distribution). But as Jeremy indisputably states, the outcome of easy development and distribution has been a massive explosion in the number of games.

Can quality keep up with quantity? The industry competes on discovery, but it does so in Flatland. So its methods have inevitably involved larger and larger marketing and PR spends, bigger brand spending, internal and external cross-promotion, [over] extending IP and game re-skinning, and certainly paid acquisition. Since every one of these things require deeper pockets than most developers have, not surprisingly the industry has begun to consolidate into a network of discovery-focused “publishers”. This is not too different from the traditional industry except the emphasis is on the publishers’ ability to segment and cross-promote by increasing the size of their catalogs very, very quickly. This makes sense, since discovery at the platform distribution level is so ineffective (unlike a traditional retail store). Therefore a big catalog is critical to overcoming the inherent obstacles in Flatland.

For developers, this has resulted in an unprecedented lack of funds from a growing number of non-traditional publishers who simply don’t have the business model to finance development. This new breed of publisher is looking to plug content into their pipeline without the more traditional dose of production help,  putting most of the risk into the hands of the developers, who must figure out how to fund from other sources. In Flatland, they’re a better choice than being buried on the shelf a half-mile down the aisle, but the extra risk leads to cutting corners on quality. In the short-term at least, this means the quality curve will continue to look more like the rolling hills of Tuscany than a profile of Half Dome.

While the problem of discovery is thorny and there are no easy answers, at Kineplay we’re continuing to research the viability of location, specifically location-based and map-based discovery mechanisms for games, and we believe location (ironically, as a better, non-Flat virtual substitute for the real world) is a promising solution. Location also has the built-in benefit of enabling content to be re-skinned or re-themed based on physical coordinates, which is a way to differentiate extended content and might help bridge the gap between where we are now and high-quality gaming.

Whatever the solutions that present themselves over the next several years, the industry (and its customers) can’t keep racing to the bottom in a Flatland which, ironically, has no real bottom.