Painting to Texture on iOS

Jun 30, 2012

Draw Something has continued to do very well in the App Store and now we’re seeing more derivatives — apps and games with basic painting and sketching capabilities. Last weekend I had some fun playing around with a basic painting setup, just to see how much brush (pun intended) I’d have to go through to get to the painting picnic.

On iOS, there are really only a couple of ways to implement a painting app — Quartz or OpenGL ES (there’s a nice little walk-through using Quartz here, and Apple put together a cool little OGL example called GLPaint here).

It should be relatively clear that the OGL approach is cleaner, more flexible and a bit faster. But while GLPaint is a nice place to start, it’s not very app- or engine-friendly in the context of a full-fledged OGL app. Since the point of GLPaint’s example code is demonstrate how to do basic painting, it has no need to consider the rest of your OGL surface’s render loop, nor does it concern itself with other important engine pieces, like your OGL redundancy state checker, sorting and synchronization issues between render and update calls, and most important, clearing the buffer.

That last bit really is most important because a nicely-performing painting app should never clear the buffer. Doing so will quickly slow things down to a slide-show. This should be obvious: In order to paint to the screen — whether you’re using GL_POINT_SPRITE_OES or rolling your own quads — you’ll need to draw a ton of sprites on-screen to get a continuous line of color and/or texture. If you clear every frame, you have to re-draw every frame, and voila, you’ll have molasses in less time than it takes to launch the simulator. If you don’t clear, you’re only drawing a handful of new sprites each frame.

The GLPaint example does this — it doesn’t clear the buffer. However in a real-world app, you must clear every frame in order for anything else — GUI elements/textures, mesh rendering, camera changes, etc. — to work. Hence the conundrum — you need a nice, normal clear/render loop but you also need a render-only call each time you want to paint.

Luckily there’s a straightforward solution: painting to a texture, then render that texture in your normal render loop. And setting up a texture to paint to is easy, by attaching it to an FBO. For example, a full-screen texture buffer:

- (GLenum) CreateRenderTexture
{
    m_texW = [[UIScreen mainScreen] bounds].size.width;
    m_texH = [[UIScreen mainScreen] bounds].size.height;

    glGenFramebuffers(1, &m_texFrameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, m_texFrameBuffer);

    glGenTextures(1, &m_texTexture);
    glBindTexture(GL_TEXTURE_2D, m_texTexture);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_texW, m_texH,
        0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
        GL_TEXTURE_2D, m_texTexture, 0);

    return glCheckFramebufferStatus(GL_FRAMEBUFFER);
}

From there it’s simply a matter of drawing to the texture and only clearing the texture when you need to, e.g., by calling a function like this:

- (void) StartTextureRender:(BOOL)clear color:(COLOR)color
{
    glBindFramebuffer(GL_FRAMEBUFFER, m_texFrameBuffer);
    if (clear)
    {
        glClearColor(color.r, color.g, color.b, color.a);
        glClear(GL_COLOR_BUFFER_BIT);
    }
    glViewport(0, 0, m_texW, m_texH);
    // setup ortho matrix, render and client states here...
}

One of the cool things about Draw Something is that it records and replays your drawing. This is relatively straightforward functionality to implement, and GLPaint kinda-sorta does it as a nice bonus. However their implementation is on the oddball side and a bit shy of more readable, that-makes-sense-to-me production code. A clearer way to implement it is to do a standard 2D lerp between the current touch (as you move your finger on the screen) and the last touch. Then record the time it took between finger-down and finger-up for playback later. For instance:

- (void) Draw:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    end         = Vec2(x, y);
    dist        = Vec2Dist(start, end);
    num         = (int)dist;
    if (num > 0)
    {
        startVert   = numVerts;
        numVerts    = MaxInt(numVerts + num, MAX_PATH_VERTS);
        for (int i = startVert; i < numVerts; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        time += [Engine GetTimeElapsed];
    }
    start = end;
    [self DrawRender];
}

Below is the entire class (note that several of the vars are structs elsewhere in the engine, but you get the gist).

// INTERFACE
#define MAX_PATH_VERTS  20000
@interface Path : NSObject
{
@public
    VEC2            verts[MAX_PATH_VERTS];
    int             numVerts;
    int             startVert;
    VEC2            start;
    VEC2            end;
    VEC2            cur;
    Texture*        texture;
    COLOR           color;
    float           size;
    float           tick;
    float           time;
    int             num;
    float           dist;
    BOOL            replaying;
    int             vertCount;
    int             curVert;
    int             endVert;
}
@property (nonatomic, readwrite) BOOL replaying;
- (id)   initWithColorTextureSize:(COLOR)c texture:(Texture*)t size:(float)s;
- (void) DrawStart:(float)x y:(float)y;
- (void) Draw:(float)x y:(float)y;
- (BOOL) Replay;
- (void) ReplayStart;
@end

// IMPLEMENTATION
@implementation Path
@synthesize replaying;
- (id) initWithColorTextureSize:(COLOR)c texture:(Texture*)t size:(float)s
{
    if (!(self == [super init])) return nil;
    numVerts        = 0;
    texture         = t;
    color           = c;
    size            = s;
    return self;
}
- (void) DrawRender
{
    if (num > 0)
    {
        glEnablePointSprite(GL_TRUE, size);
        glSetTexture(texture.index);
        glSetColor(color.r, color.g, color.b, color.a);
        glSetVertexPointerEx(&verts[0], sizeof(VEC2), 2);
        glDrawArrays(GL_POINTS, startVert, num);
    }
}
- (void) DrawStart:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    verts[0]    = start = end = Vec2(x, y);
    numVerts    = num = 1;
    startVert   = 0;
    time        = 0;
    [self DrawRender];
}
- (void) Draw:(float)x y:(float)y
{
    if (numVerts == MAX_PATH_VERTS) return;
    end         = Vec2(x, y);
    dist        = Vec2Dist(start, end);
    num         = (int)dist;
    if (num > 0)
    {
        startVert   = numVerts;
        numVerts    = IEMaxInt(numVerts + num, MAX_PATH_VERTS);
        for (int i = startVert; i < numVerts; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        time += [Engine GetTimeElapsed];
    }
    start = end;
    [self DrawRender];
}
- (BOOL) Replay
{
    if (replaying)
    {
        tick    = Max(tick + [Engine GetTimeElapsed], time);
        curVert = endVert;
        endVert = (int)Max(Lerp(0, numVerts, tick / time), numVerts);
        dist    = Vec2Dist(start, end);
        end     = verts[endVert];
        for (int i = startVert; i < endVert; i++)
        {
            Vec2Lerp(&verts[i], start, end, (i - startVert) / dist);
        }
        start = end;
        int count = MinInt(endVert - curVert, 1);
        if (count > 0)
        {
            glEnablePointSprite(GL_TRUE, size);
            glSetTexture(texture.index);
            glSetColor(color.r, color.g, color.b, color.a);
            glSetVertexPointerEx(&verts[0], sizeof(VEC2), 2);
            glDrawArrays(GL_POINTS, curVert, count);
        }
        replaying = (endVert != numVerts);
    }
    return replaying;
}
- (void) ReplayStart
{
    curVert   = 0;
    endVert   = 0;
    replaying = (curVert < numVerts);
    if (replaying)
    {
        tick  = 0;
        time  = Min(time, 0.001);
        start = verts[0];
        end   = verts[1];
    }
}
@end

An NSMutableArray of multiple instances of this class is kept by the caller; each instance is born on finger-down (where we set color, brush texture and size) and dies on finger-up. Replay is easy — essentially just a programmatic rendering of the verts that were previously recorded while iterating over the NSMutableArray, handled with a flag in the Render() function. Below is the basic idea.

- (void) TouchDown:(float)x y:(float)y
{
    if (curSize == -1) // we're erasing here
    {
        [Engine StartTextureRender:YES color:curBackColor];
    }
    else
    {
        [Engine StartTextureRender:NO color:curBackColor];
        curPath = [[Path alloc] initWithColorTextureSize:curColor
                                                 texture:curTexture
                                                    size:curSize];
        [paths addObject:curPath], [curPath release];
        [curPath DrawStart:x y:y];
    }
}
- (void) TouchMove:(float)x y:(float)y
{
    [Engine StartTextureRender:NO color:curBackColor];
    [curPath Draw:x y:y];
}
- (void) TouchUp:(float)x y:(float)y
{
    curPath = nil;
}
- (void) Render
{
    [Engine RenderToTexture];
    if (replaying)
    {
        [Engine StartTextureRender:NO color:curBackColor];
        if (![curPath Replay])
        {
            curPath = nil;
            for (Path* m in paths) { if (m.replaying) { curPath = m; break; } }
            replaying = (curPath != nil);
        }
    }
}

One of the cool things about using OGL for painting and sketching is that you can very easily change up the brush texture, for nice Photoshop-like texture brushes (care should be paid to how you setup the blending, however, due to pre-multiplied alpha on iOS). While it’s possible to do this with Quartz, it’s much easier to grok using OGL. And of course you can do silly/fun stuff like paint a background behind your 3D orc model (maybe there’s a game idea in there somewhere, hmm — ok, maybe not).