From UTF-8 to Retained Text in Cheryl Engine 4

A UTF-8 string does not directly describe a list of things to draw. Several bytes can encode one scalar, several scalars can form one grapheme, and shaping can combine or reposition glyphs. Bidirectional text adds another ordering. A useful engine pipeline has to retain those distinctions while producing something the renderer can consume.

The Unicode path in Cheryl Engine 4 separates layout, CPU preparation, upload, and retained submission. HarfBuzz supplies OpenType shaping, ICU supplies boundary and bidirectional analysis, and FreeType supplies glyph loading and rasterization. Cheryl’s integration owns the font snapshots, source mappings, wrapping decisions, resource generations, and update scheduling described here.

Start with a source that is not one character per byte

Consider a label containing Cafe followed by U+0301 COMBINING ACUTE ACCENT, a slash, Hebrew text, and digits. The e and accent occupy three UTF-8 bytes and two scalars, but form one grapheme. Their eventual glyph count depends on the selected font and shaping.

Assume font is an already-selected CE::Text::Font. This example uses explicit UTF-8 bytes, a right-to-left paragraph base, and a width in local pixels:

#include <cheryl/text/layout.h>
#include <cheryl/assets/types/2d/unicode-text.h>
#include <string>
#include <utility>

const std::string label =
    "Cafe\xCC\x81 / "
    "\xD7\xA9\xD7\x9C\xD7\x95\xD7\x9D 123";

CE::Text::LayoutOptions options;
options.pixel_height = 24;
options.maximum_width = 180.0f;
options.direction = CE::Text::ParagraphDirection::RightToLeft;

auto shaped = CE::Text::layout_text(font, label, options);
auto prepared = CE::Text::Rendering::prepare(std::move(shaped));

The decoder retains each scalar’s original byte range. Malformed sequences produce replacement scalars with the consumed byte ranges recorded. Encoding errors remain distinguishable from missing font coverage. The result retains decoded records and font bytes, rather than borrowing the caller’s string; an application wanting the original text itself must still own that text.

Choose fonts for graphemes, then shape runs

ICU’s character-boundary iterator establishes the grapheme units used for fallback. Cheryl selects the first retained face covering the visible scalars of a whole unit. This keeps a combining mark with its base. An uncovered grapheme uses an embedded replacement and records that font coverage was missing.

ICU also resolves the paragraph’s bidirectional structure. The explicit RTL option sets the paragraph base; Latin text and digits still receive their own run directions. Cheryl derives the visual runs for each accepted line, separates them by face and script, and passes direction, script, language, and source-cluster indices to HarfBuzz. See grapheme selection and shaping setup and paragraph and visual-run handling.

Glyphs retain both their face-qualified identity and their logical source range. Several glyphs can share a shaping cluster; a ligature can cover several source scalars. These associations explain which source produced a glyph, but do not supply a complete caret-navigation or selection policy. Shaping clusters and editing positions need separate treatment.

Wrapping changes the shaping problem

Measuring an entire paragraph and slicing its glyph array would lose the actual line-boundary context. Cheryl asks ICU for line-break opportunities and shapes candidate lines at their proposed boundaries. A candidate is accepted using its resulting advance, while paragraph bidi state carries across the wrapped lines.

If a word cannot fit, the fallback wrapping loop tries whole-grapheme prefixes. A single too-wide grapheme is admitted and marks the line as overflowing, allowing layout to advance. Clipping or scaling that result belongs to the caller. Trailing ASCII spaces and tabs contribute no visual advance, while line records retain their source membership.

This is a greedy fitter with repeated candidate shaping. Long paragraphs under narrow widths can repeat substantial work; the implementation alone supplies no performance result. It also does not provide automatic hyphenation, justification, vertical writing, or a text editor. Those are separate requirements from shaping and fitting a label.

Prepare pixels, then publish matching resources

prepare() rasterizes each distinct face-qualified glyph used by the message through FreeType. It packs grayscale coverage into padded CPU pages and creates glyph geometry. Empty outlines retain their advance without adding a quad. The preparation loop owns its buffers and requires no resource provider.

upload() runs on the provider’s graphics owner and creates fresh geometry and atlas handles. It validates the returned resources before returning a complete Generation. If an upload fails partway through, no replacement generation is returned; any transient resources already created follow their backend cleanup path. The caller can publish a complete value after success; that does not make the individual native uploads atomic.

Text submission selects the geometry and atlas from that same generation. Packets retain those handles, so replacing a label does not pair an old glyph range with a new atlas. That is the connection to prepared frames and resource lifetimes.

Keep the last complete label during replacement

CE::Text::Rendering::Updater owns asynchronous replacement. request() records desired content; update() advances the work without waiting. CPU layout/preparation runs through a context-tracked worker group, followed by upload through its platform dispatcher. Its jobs retain requests instead of borrowing the updater or game.

Construct the updater during init_platform(), then serialize request() and update() calls on the simulation owner. The initial-request constructor prepares and uploads synchronously on the graphics owner; later replacements use the worker and dispatcher path.

One active replacement and one newest desired request bound the queue. Intermediate desired values coalesce, but a successful active result is adopted before its successor starts. Preparation, submission, or upload failure preserves current(). An unchanged failed request stays settled until explicit retry(); resubmitting the same request does not create a busy retry loop. These transitions are visible in the updater implementation.

Updater calls require serialization, and retained text still needs its original live graphics domain for native use. The runtime lifecycle supplies the worker/dispatcher shutdown boundary. Together, these pieces turn a changing Unicode label into owned values at each stage while preserving the information needed to explain the pixels back to their source.

Cheryl Engine 4 reading path

1. Runtime ownership and lifecycle
2. Prepared frames and rendering handoff
3. Retained resources and cleanup
4. Input routing and editable menus
5. Unicode layout and retained text (this article)
6. Optional modules and the installed SDK