Optimize terrain rendering for large exports
This commit is contained in:
@@ -17,4 +17,6 @@ MINEGEO_FACE_SIZE=1024
|
||||
MINEGEO_WORKER_HEAP_MB=4096
|
||||
# Fail with an actionable error if rendering makes no reported progress for this long.
|
||||
MINEGEO_RENDER_IDLE_SECONDS=600
|
||||
# Allow Chromium to use the GPU; software forces SwiftShader for headless hosts.
|
||||
MINEGEO_RENDER_BACKEND=auto
|
||||
MINEGEO_MAX_UPLOAD_BYTES=268435456
|
||||
|
||||
@@ -50,6 +50,14 @@ The worker renders the map and panoramas. Open the private draft in your dashboa
|
||||
|
||||
Both the dashboard and draft show completed-work progress: downloading, reading/indexing terrain, loading models, building geometry, rendering panorama faces, rendering the surface map, and saving images. Percentages are not a time estimate. The terminal reports block/face counts as work advances. A large export can take longer on a Mac using software rendering.
|
||||
|
||||
The renderer compiles each block state's model transforms, textures and UVs once, then builds blocks synchronously into bounded typed buffers. Hidden faces use a compact bitset index. Meshes are split into 64 × 64-block regions so panorama cameras can cull distant/off-screen geometry while the surface map still covers the complete area. No terrain is downsampled by this optimization. Model loading and image uploads use bounded parallel requests, and panorama faces are encoded directly as WebP.
|
||||
|
||||
`MINEGEO_RENDER_BACKEND=auto` (the default) allows Chromium to use the GPU. If graphics initialization fails, the worker falls back to SwiftShader. To force the original software backend on a host with graphics driver issues, use `MINEGEO_RENDER_BACKEND=software` in `.env.local` and restart. The log names the actual graphics renderer; `auto` does not guarantee that the host has GPU acceleration.
|
||||
|
||||
Building logs now include blocks per second. Each job's local `render-report.json` records indexing/model/build times, emitted and hidden face counts, geometry bytes, mesh count and actual graphics backend; the worker prints this report when rendering finishes. Use those measurements to compare runs on the same Mac and export. An existing running worker must be restarted after pulling the optimizer to use the new builder.
|
||||
|
||||
Stopping the updated worker cancels its renderer and returns its job to the queue without using up a retry. The next worker renders that export from the beginning; completed published maps are unaffected.
|
||||
|
||||
If the worker stops renewing its lease, the page warns that it has stopped reporting. Renderer cancellation closes its browser, and `MINEGEO_RENDER_IDLE_SECONDS` (default 600) fails a renderer that reports no progress for that period. This is an inactivity limit, not a limit on total rendering time. Use a smaller export area or lower `MINEGEO_FACE_SIZE` if rendering one face exceeds it.
|
||||
|
||||
## Retry, cancel, and configure
|
||||
|
||||
@@ -20,3 +20,9 @@ The real Storage test caught two additional issues that a mock endpoint did not:
|
||||
Reports: `artifacts-plugin-regression.json` and `artifacts-api-smoke.json`. Temporary test datasets were deleted and the temporary servers stopped.
|
||||
|
||||
This pass did not launch a Minecraft client, use Playwright, or use the Computer plugin. The browser appearance, an actual Fabric capture session, Spigot loading, and a fresh plugin panorama rendering run on macOS still need runtime verification. The existing renderer has not been replaced; earlier renderer proof is recorded separately in `artifacts-renderer-proof.json`.
|
||||
|
||||
## Large-export optimization
|
||||
|
||||
The subsequent renderer changes replace per-block model transformation and texture loading with cached state templates, numeric section bitsets, bounded typed geometry buffers and spatial meshes. WebP output avoids an intermediate PNG encode, and processed-image uploads run in a four-request pool. GPU selection is automatic with software fallback on initialization failure.
|
||||
|
||||
No tests, browsers or benchmarks were run for this change at the user's request. Its speedup and image fidelity on the 2,651,888-block Mac export are unverified. The new per-stage timings, blocks-per-second logs and geometry counts provide measurements for the next real run.
|
||||
|
||||
132
renderer/mesh-buffers.mjs
Normal file
132
renderer/mesh-buffers.mjs
Normal file
@@ -0,0 +1,132 @@
|
||||
// Numeric section keys and one bit per opaque voxel avoid millions of coordinate
|
||||
// strings. Bounds checks prevent key aliasing outside the indexed sections.
|
||||
export class OpaqueIndex {
|
||||
constructor(bounds, minY, maxY) {
|
||||
this.minX = Math.floor(bounds.minX / 16);
|
||||
this.maxX = Math.floor(bounds.maxX / 16);
|
||||
this.minZ = Math.floor(bounds.minZ / 16);
|
||||
this.maxZ = Math.floor(bounds.maxZ / 16);
|
||||
this.minY = Math.floor(minY / 16);
|
||||
this.maxY = Math.floor(maxY / 16);
|
||||
this.width = this.maxX - this.minX + 1;
|
||||
this.depth = this.maxZ - this.minZ + 1;
|
||||
this.sections = new Map();
|
||||
}
|
||||
key(x, y, z) {
|
||||
const cx = Math.floor(x / 16),
|
||||
cy = Math.floor(y / 16),
|
||||
cz = Math.floor(z / 16);
|
||||
if (
|
||||
cx < this.minX ||
|
||||
cx > this.maxX ||
|
||||
cz < this.minZ ||
|
||||
cz > this.maxZ ||
|
||||
cy < this.minY ||
|
||||
cy > this.maxY
|
||||
)
|
||||
return -1;
|
||||
return (
|
||||
cx -
|
||||
this.minX +
|
||||
(cz - this.minZ) * this.width +
|
||||
(cy - this.minY) * this.width * this.depth
|
||||
);
|
||||
}
|
||||
add(x, y, z) {
|
||||
const key = this.key(x, y, z);
|
||||
if (key < 0) throw new Error("Terrain block outside indexed bounds");
|
||||
let section = this.sections.get(key);
|
||||
if (!section) {
|
||||
section = new Uint32Array(128);
|
||||
this.sections.set(key, section);
|
||||
}
|
||||
const bit = (x & 15) | ((z & 15) << 4) | ((y & 15) << 8);
|
||||
section[bit >>> 5] |= 1 << (bit & 31);
|
||||
}
|
||||
has(x, y, z) {
|
||||
const key = this.key(x, y, z);
|
||||
if (key < 0) return false;
|
||||
const section = this.sections.get(key);
|
||||
if (!section) return false;
|
||||
const bit = (x & 15) | ((z & 15) << 4) | ((y & 15) << 8);
|
||||
return (section[bit >>> 5] & (1 << (bit & 31))) !== 0;
|
||||
}
|
||||
clear() {
|
||||
this.sections.clear();
|
||||
}
|
||||
}
|
||||
|
||||
// Each buffer is bounded, keeps 16-bit triangle indices, and writes directly
|
||||
// into typed arrays. No per-vertex objects or growable JS number arrays.
|
||||
export class QuadBuffer {
|
||||
constructor(maxQuads = 4096) {
|
||||
this.maxQuads = maxQuads;
|
||||
this.capacity = Math.min(32, maxQuads);
|
||||
this.count = 0;
|
||||
this.positions = new Float32Array(this.capacity * 12);
|
||||
this.normals = new Float32Array(this.capacity * 12);
|
||||
this.uvs = new Float32Array(this.capacity * 8);
|
||||
this.indices = new Uint16Array(this.capacity * 6);
|
||||
this.min = [Infinity, Infinity, Infinity];
|
||||
this.max = [-Infinity, -Infinity, -Infinity];
|
||||
}
|
||||
get full() {
|
||||
return this.count === this.maxQuads;
|
||||
}
|
||||
add(face, x, y, z) {
|
||||
if (this.full) throw new Error("Geometry batch is full");
|
||||
if (this.count === this.capacity) {
|
||||
this.capacity = Math.min(this.maxQuads, this.capacity * 2);
|
||||
for (const [key, components, Type] of [
|
||||
["positions", 12, Float32Array],
|
||||
["normals", 12, Float32Array],
|
||||
["uvs", 8, Float32Array],
|
||||
["indices", 6, Uint16Array],
|
||||
]) {
|
||||
const next = new Type(this.capacity * components);
|
||||
next.set(this[key]);
|
||||
this[key] = next;
|
||||
}
|
||||
}
|
||||
const offset = this.count * 12;
|
||||
for (let i = 0; i < 12; i += 3) {
|
||||
this.positions[offset + i] = face.positions[i] + x;
|
||||
this.positions[offset + i + 1] = face.positions[i + 1] + y;
|
||||
this.positions[offset + i + 2] = face.positions[i + 2] + z;
|
||||
// Bounds must enclose the actual Float32 positions, including rounding
|
||||
// at large Minecraft coordinates, or frustum culling can cut off faces.
|
||||
const px = this.positions[offset + i],
|
||||
py = this.positions[offset + i + 1],
|
||||
pz = this.positions[offset + i + 2];
|
||||
this.normals[offset + i] = face.normal[0];
|
||||
this.normals[offset + i + 1] = face.normal[1];
|
||||
this.normals[offset + i + 2] = face.normal[2];
|
||||
if (px < this.min[0]) this.min[0] = px;
|
||||
if (py < this.min[1]) this.min[1] = py;
|
||||
if (pz < this.min[2]) this.min[2] = pz;
|
||||
if (px > this.max[0]) this.max[0] = px;
|
||||
if (py > this.max[1]) this.max[1] = py;
|
||||
if (pz > this.max[2]) this.max[2] = pz;
|
||||
}
|
||||
this.uvs.set(face.uvs, this.count * 8);
|
||||
const base = this.count * 4,
|
||||
index = this.count * 6;
|
||||
this.indices[index] = base;
|
||||
this.indices[index + 1] = base + 1;
|
||||
this.indices[index + 2] = base + 2;
|
||||
this.indices[index + 3] = base + 2;
|
||||
this.indices[index + 4] = base + 1;
|
||||
this.indices[index + 5] = base + 3;
|
||||
this.count++;
|
||||
}
|
||||
finish() {
|
||||
return {
|
||||
positions: this.positions.slice(0, this.count * 12),
|
||||
normals: this.normals.slice(0, this.count * 12),
|
||||
uvs: this.uvs.slice(0, this.count * 8),
|
||||
indices: this.indices.slice(0, this.count * 6),
|
||||
min: this.min,
|
||||
max: this.max,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -56,7 +56,7 @@ export async function renderSnapshot(
|
||||
throw new Error("Invalid path");
|
||||
res.setHeader(
|
||||
"Content-Type",
|
||||
path.endsWith(".js")
|
||||
path.endsWith(".js") || path.endsWith(".mjs")
|
||||
? "text/javascript"
|
||||
: path.endsWith(".json")
|
||||
? "application/json"
|
||||
@@ -92,6 +92,11 @@ export async function renderSnapshot(
|
||||
await new Promise((r) => server.close(r));
|
||||
throw new Error("MINEGEO_RENDER_IDLE_SECONDS must be at least 60");
|
||||
}
|
||||
const backend = process.env.MINEGEO_RENDER_BACKEND || "auto";
|
||||
if (!["auto", "software"].includes(backend)) {
|
||||
await new Promise((r) => server.close(r));
|
||||
throw new Error("MINEGEO_RENDER_BACKEND must be auto or software");
|
||||
}
|
||||
const watchdog = setInterval(() => {
|
||||
if (Date.now() - lastActivity > idleSeconds * 1000) {
|
||||
failure = new Error(
|
||||
@@ -104,20 +109,52 @@ export async function renderSnapshot(
|
||||
try {
|
||||
if (signal?.aborted) throw new Error("Rendering cancelled");
|
||||
await report({ percent: 24, message: "Starting terrain renderer" });
|
||||
browser = await chromium.launch({
|
||||
headless: true,
|
||||
args: ["--use-angle=swiftshader", "--enable-unsafe-swiftshader"],
|
||||
let page;
|
||||
let selectedBackend = backend;
|
||||
async function openRenderer(mode) {
|
||||
browser = await chromium.launch({
|
||||
headless: true,
|
||||
args:
|
||||
mode === "software"
|
||||
? ["--use-angle=swiftshader", "--enable-unsafe-swiftshader"]
|
||||
: ["--enable-gpu", "--enable-unsafe-swiftshader"],
|
||||
});
|
||||
if (signal?.aborted || failure)
|
||||
throw failure || new Error("Rendering cancelled");
|
||||
const rendererPage = await browser.newPage();
|
||||
await rendererPage.exposeFunction("minegeoProgress", report);
|
||||
rendererPage.on("pageerror", (err) =>
|
||||
console.error("Renderer:", err.message),
|
||||
);
|
||||
await rendererPage.goto(`http://127.0.0.1:${server.address().port}`);
|
||||
await rendererPage.waitForFunction(() => window.rendererReady);
|
||||
return rendererPage;
|
||||
}
|
||||
try {
|
||||
page = await openRenderer(backend);
|
||||
} catch (error) {
|
||||
if (backend === "software" || signal?.aborted || failure || !browser)
|
||||
throw error;
|
||||
await browser.close();
|
||||
console.warn(
|
||||
"GPU renderer initialization failed; falling back to software rendering.",
|
||||
);
|
||||
selectedBackend = "software";
|
||||
page = await openRenderer("software");
|
||||
}
|
||||
const graphics = await page.evaluate(() => window.rendererInfo);
|
||||
console.log(`MineGeo graphics (${selectedBackend}): ${graphics.renderer}`);
|
||||
await report({
|
||||
percent: 24,
|
||||
message: `Graphics backend: ${graphics.renderer}`,
|
||||
});
|
||||
const page = await browser.newPage();
|
||||
await page.exposeFunction("minegeoProgress", report);
|
||||
page.on("pageerror", (err) => console.error("Renderer:", err.message));
|
||||
await page.goto(`http://127.0.0.1:${server.address().port}`);
|
||||
await page.waitForFunction(() => window.rendererReady);
|
||||
const stats = await page.evaluate(async () => {
|
||||
const response = await fetch("/terrain.json");
|
||||
if (!response.ok) throw new Error("Unable to load terrain snapshot");
|
||||
return window.buildTerrain(await response.json());
|
||||
});
|
||||
stats.graphics = graphics;
|
||||
stats.backend = selectedBackend;
|
||||
await report({
|
||||
percent: 60,
|
||||
message: `Terrain ready: ${stats.blocks} blocks; rendering ${viewpoints.length} panoramas`,
|
||||
@@ -131,12 +168,17 @@ export async function renderSnapshot(
|
||||
({ p, face, size }) => window.renderFace(p, face, size),
|
||||
{ p, face, size },
|
||||
);
|
||||
const png = Buffer.from(data.split(",")[1], "base64");
|
||||
await sharp(png)
|
||||
.webp({ quality: 85 })
|
||||
.toFile(`${output}/${p.id}/${face}.webp`);
|
||||
const encoded = Buffer.from(data.split(",")[1], "base64");
|
||||
// Chromium encodes WebP once; retain a PNG fallback for canvas runtimes
|
||||
// without WebP support rather than silently writing the wrong MIME type.
|
||||
if (data.startsWith("data:image/webp;"))
|
||||
await writeFile(`${output}/${p.id}/${face}.webp`, encoded);
|
||||
else
|
||||
await sharp(encoded)
|
||||
.webp({ quality: 85 })
|
||||
.toFile(`${output}/${p.id}/${face}.webp`);
|
||||
if (face === "nz")
|
||||
await sharp(png)
|
||||
await sharp(encoded)
|
||||
.resize(512, 512)
|
||||
.webp({ quality: 76 })
|
||||
.toFile(`${output}/${p.id}/preview.webp`);
|
||||
@@ -148,9 +190,11 @@ export async function renderSnapshot(
|
||||
}
|
||||
await report({ percent: 85, message: "Rendering the surface map" });
|
||||
const map = await page.evaluate(() => window.renderMap(1024));
|
||||
await sharp(Buffer.from(map.split(",")[1], "base64"))
|
||||
.webp({ quality: 90 })
|
||||
.toFile(`${output}/map.webp`);
|
||||
const mapBytes = Buffer.from(map.split(",")[1], "base64");
|
||||
if (map.startsWith("data:image/webp;"))
|
||||
await writeFile(`${output}/map.webp`, mapBytes);
|
||||
else
|
||||
await sharp(mapBytes).webp({ quality: 90 }).toFile(`${output}/map.webp`);
|
||||
await report({
|
||||
percent: 90,
|
||||
message: "Rendered all panoramas and the surface map",
|
||||
@@ -166,7 +210,7 @@ export async function renderSnapshot(
|
||||
} finally {
|
||||
clearInterval(watchdog);
|
||||
signal?.removeEventListener("abort", cancel);
|
||||
await browser?.close();
|
||||
await browser?.close().catch(() => {});
|
||||
await new Promise((r) => server.close(r));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import * as THREE from "/three.js";
|
||||
import { buildTerrain } from "./terrain-mesh.js";
|
||||
// This renderer consumes snapshots, never connects to a Minecraft server.
|
||||
const renderer = new THREE.WebGLRenderer({
|
||||
antialias: true,
|
||||
@@ -8,6 +9,16 @@ renderer.outputColorSpace = THREE.SRGBColorSpace;
|
||||
renderer.setPixelRatio(1);
|
||||
document.body.append(renderer.domElement);
|
||||
const scene = new THREE.Scene();
|
||||
const gl = renderer.getContext();
|
||||
const gpuInfo = gl.getExtension("WEBGL_debug_renderer_info");
|
||||
window.rendererInfo = {
|
||||
vendor: gpuInfo
|
||||
? gl.getParameter(gpuInfo.UNMASKED_VENDOR_WEBGL)
|
||||
: "unavailable",
|
||||
renderer: gpuInfo
|
||||
? gl.getParameter(gpuInfo.UNMASKED_RENDERER_WEBGL)
|
||||
: "unavailable",
|
||||
};
|
||||
scene.background = new THREE.Color("#95c4eb");
|
||||
scene.add(new THREE.HemisphereLight("#e9f3ff", "#73704e", 2.0));
|
||||
const sun = new THREE.DirectionalLight("#fff4d6", 2.2);
|
||||
@@ -53,11 +64,20 @@ function textureRef(m, ref) {
|
||||
async function texture(ref) {
|
||||
if (!ref) return null;
|
||||
if (!textureCache.has(ref)) {
|
||||
const t = await loader.loadAsync(resourcePath(ref, "textures") + ".png");
|
||||
t.magFilter = THREE.NearestFilter;
|
||||
t.minFilter = THREE.NearestFilter;
|
||||
t.colorSpace = THREE.SRGBColorSpace;
|
||||
textureCache.set(ref, t);
|
||||
// Cache the in-flight promise and failures as well as successful textures.
|
||||
// Missing vanilla assets must not trigger another request for every face.
|
||||
textureCache.set(
|
||||
ref,
|
||||
loader
|
||||
.loadAsync(resourcePath(ref, "textures") + ".png")
|
||||
.then((t) => {
|
||||
t.magFilter = THREE.NearestFilter;
|
||||
t.minFilter = THREE.NearestFilter;
|
||||
t.colorSpace = THREE.SRGBColorSpace;
|
||||
return t;
|
||||
})
|
||||
.catch(() => null),
|
||||
);
|
||||
}
|
||||
return textureCache.get(ref);
|
||||
}
|
||||
@@ -152,40 +172,9 @@ const faceData = {
|
||||
],
|
||||
},
|
||||
};
|
||||
const nonOpaque = (s) =>
|
||||
/air|water|lava|leaves|glass|grass(?!_block)|fern|flower|sapling|torch|fence|wall|slab|stairs|door|pane|rail|vine|carpet/.test(
|
||||
s,
|
||||
);
|
||||
window.buildTerrain = async (snapshot) => {
|
||||
const started = performance.now(),
|
||||
batches = new Map(),
|
||||
unsupported = new Set();
|
||||
const occupancy = new Map();
|
||||
const distinctStates = new Set();
|
||||
let lastProgress = 0;
|
||||
async function progress(percent, message, force = false) {
|
||||
if (!force && performance.now() - lastProgress < 3000) return;
|
||||
lastProgress = performance.now();
|
||||
// Yield to browser tasks so bindings, cancellation and timers can run.
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
await window.minegeoProgress?.({ percent, message });
|
||||
}
|
||||
await progress(
|
||||
25,
|
||||
`Indexing ${snapshot.blocks.length.toLocaleString()} terrain blocks`,
|
||||
true,
|
||||
);
|
||||
for (let i = 0; i < snapshot.blocks.length; i++) {
|
||||
const b = snapshot.blocks[i];
|
||||
occupancy.set(`${b[0]},${b[1]},${b[2]}`, b[3]);
|
||||
distinctStates.add(b[3]);
|
||||
if (i % 16384 === 0)
|
||||
await progress(
|
||||
25 + (5 * i) / snapshot.blocks.length,
|
||||
`Indexing blocks ${i}/${snapshot.blocks.length}`,
|
||||
);
|
||||
}
|
||||
const stateCache = new Map();
|
||||
const unsupported = new Set(),
|
||||
stateCache = new Map();
|
||||
async function stateModels(s) {
|
||||
if (stateCache.has(s)) return stateCache.get(s);
|
||||
const { name, props } = parseState(s);
|
||||
@@ -200,193 +189,15 @@ window.buildTerrain = async (snapshot) => {
|
||||
stateCache.set(s, variants);
|
||||
return variants;
|
||||
}
|
||||
// Preload each distinct model once before constructing batched geometry.
|
||||
await progress(30, "Loading block models and textures", true);
|
||||
let modelCount = 0;
|
||||
for (const s of distinctStates) {
|
||||
await stateModels(s);
|
||||
await progress(
|
||||
30 + (5 * ++modelCount) / distinctStates.size,
|
||||
`Loading block model ${modelCount}/${distinctStates.size}`,
|
||||
);
|
||||
}
|
||||
async function emit(block, m, v = {}) {
|
||||
const [bx, by, bz, s] = block;
|
||||
for (const e of m.elements || []) {
|
||||
const from = e.from.map((n) => n / 16),
|
||||
to = e.to.map((n) => n / 16);
|
||||
const transform = new THREE.Matrix4();
|
||||
const r = new THREE.Euler(
|
||||
(-(v.x || 0) * Math.PI) / 180,
|
||||
(-(v.y || 0) * Math.PI) / 180,
|
||||
0,
|
||||
"YXZ",
|
||||
);
|
||||
transform.makeRotationFromEuler(r);
|
||||
for (const [dir, f] of Object.entries(e.faces)) {
|
||||
const fd = faceData[dir];
|
||||
if (!fd) continue;
|
||||
const n = new THREE.Vector3(...fd.normal).applyMatrix4(transform);
|
||||
const neighbor = occupancy.get(
|
||||
`${bx + Math.round(n.x)},${by + Math.round(n.y)},${bz + Math.round(n.z)}`,
|
||||
);
|
||||
if (f.cullface && neighbor && !nonOpaque(neighbor)) continue;
|
||||
const ref = textureRef(m, f.texture),
|
||||
tex = await texture(ref).catch(() => null);
|
||||
if (!tex) {
|
||||
unsupported.add(s);
|
||||
continue;
|
||||
}
|
||||
const tint =
|
||||
f.tintindex !== undefined
|
||||
? s.includes("water")
|
||||
? "#5b9eb5"
|
||||
: s.includes("leaves")
|
||||
? "#7ba54c"
|
||||
: "#90b650"
|
||||
: "#ffffff";
|
||||
const key = ref + "|" + tint;
|
||||
let batch = batches.get(key);
|
||||
if (!batch) {
|
||||
batch = {
|
||||
positions: [],
|
||||
uvs: [],
|
||||
normals: [],
|
||||
indices: [],
|
||||
tex,
|
||||
tint,
|
||||
};
|
||||
batches.set(key, batch);
|
||||
}
|
||||
const uv = f.uv || [0, 0, 16, 16],
|
||||
coords = [
|
||||
[uv[0] / 16, 1 - uv[3] / 16],
|
||||
[uv[2] / 16, 1 - uv[3] / 16],
|
||||
[uv[0] / 16, 1 - uv[1] / 16],
|
||||
[uv[2] / 16, 1 - uv[1] / 16],
|
||||
];
|
||||
const base = batch.positions.length / 3;
|
||||
for (let i = 0; i < 4; i++) {
|
||||
let p = new THREE.Vector3(
|
||||
...fd.corners[i].map((c, j) => from[j] + c * (to[j] - from[j])),
|
||||
);
|
||||
if (e.rotation) {
|
||||
const er = e.rotation,
|
||||
origin = new THREE.Vector3(...er.origin.map((a) => a / 16)),
|
||||
axis = new THREE.Vector3(
|
||||
er.axis === "x" ? 1 : 0,
|
||||
er.axis === "y" ? 1 : 0,
|
||||
er.axis === "z" ? 1 : 0,
|
||||
);
|
||||
p.sub(origin).applyAxisAngle(axis, (er.angle * Math.PI) / 180);
|
||||
if (er.rescale) {
|
||||
const scale = 1 / Math.cos((er.angle * Math.PI) / 180);
|
||||
for (const a of ["x", "y", "z"]) if (a !== er.axis) p[a] *= scale;
|
||||
}
|
||||
p.add(origin);
|
||||
}
|
||||
p.subScalar(0.5)
|
||||
.applyMatrix4(transform)
|
||||
.addScalar(0.5)
|
||||
.add(new THREE.Vector3(bx, by, bz));
|
||||
batch.positions.push(p.x, p.y, p.z);
|
||||
batch.normals.push(n.x, n.y, n.z);
|
||||
let index = i;
|
||||
if (f.rotation) {
|
||||
const order = [0, 1, 3, 2],
|
||||
rank = order.indexOf(i);
|
||||
index = order[(rank + f.rotation / 90) % 4];
|
||||
}
|
||||
batch.uvs.push(...coords[index]);
|
||||
}
|
||||
batch.indices.push(
|
||||
base,
|
||||
base + 1,
|
||||
base + 2,
|
||||
base + 2,
|
||||
base + 1,
|
||||
base + 3,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
await progress(35, "Building terrain geometry", true);
|
||||
for (let i = 0; i < snapshot.blocks.length; i++) {
|
||||
const b = snapshot.blocks[i];
|
||||
if (i % 256 === 0)
|
||||
await progress(
|
||||
35 + (20 * i) / snapshot.blocks.length,
|
||||
`Building blocks ${i}/${snapshot.blocks.length}`,
|
||||
);
|
||||
if (/minecraft:(water|lava)(\[|$)/.test(b[3])) {
|
||||
await emit(b, {
|
||||
textures: {
|
||||
all:
|
||||
"minecraft:block/" +
|
||||
(b[3].includes("water") ? "water_still" : "lava_still"),
|
||||
},
|
||||
elements: [
|
||||
{
|
||||
from: [0, 0, 0],
|
||||
to: [16, 14, 16],
|
||||
faces: Object.fromEntries(
|
||||
Object.keys(faceData).map((k) => [
|
||||
k,
|
||||
{
|
||||
texture: "#all",
|
||||
tintindex: b[3].includes("water") ? 0 : undefined,
|
||||
cullface: k,
|
||||
},
|
||||
]),
|
||||
),
|
||||
},
|
||||
],
|
||||
});
|
||||
continue;
|
||||
}
|
||||
for (const { m, v } of await stateModels(b[3])) await emit(b, m, v);
|
||||
}
|
||||
occupancy.clear();
|
||||
await progress(55, "Preparing the scene", true);
|
||||
let batchCount = 0;
|
||||
for (const batch of batches.values()) {
|
||||
const g = new THREE.BufferGeometry();
|
||||
g.setAttribute(
|
||||
"position",
|
||||
new THREE.Float32BufferAttribute(batch.positions, 3),
|
||||
);
|
||||
g.setAttribute(
|
||||
"normal",
|
||||
new THREE.Float32BufferAttribute(batch.normals, 3),
|
||||
);
|
||||
g.setAttribute("uv", new THREE.Float32BufferAttribute(batch.uvs, 2));
|
||||
g.setIndex(batch.indices);
|
||||
const mat = new THREE.MeshLambertMaterial({
|
||||
map: batch.tex,
|
||||
color: batch.tint,
|
||||
side: THREE.DoubleSide,
|
||||
alphaTest: 0.3,
|
||||
transparent: batch.tex.image.src.includes("water"),
|
||||
opacity: batch.tex.image.src.includes("water") ? 0.8 : 1,
|
||||
});
|
||||
scene.add(new THREE.Mesh(g, mat));
|
||||
// Drop the JS number arrays as each GPU geometry is completed.
|
||||
batch.positions.length = 0;
|
||||
batch.normals.length = 0;
|
||||
batch.uvs.length = 0;
|
||||
batch.indices.length = 0;
|
||||
await progress(
|
||||
55 + (5 * ++batchCount) / batches.size,
|
||||
`Preparing geometry ${batchCount}/${batches.size}`,
|
||||
);
|
||||
}
|
||||
window.terrainBounds = snapshot.bounds;
|
||||
return {
|
||||
blocks: snapshot.blocks.length,
|
||||
drawCalls: batches.size,
|
||||
unsupported: [...unsupported],
|
||||
buildMs: performance.now() - started,
|
||||
};
|
||||
return buildTerrain(
|
||||
snapshot,
|
||||
scene,
|
||||
faceData,
|
||||
stateModels,
|
||||
textureRef,
|
||||
texture,
|
||||
unsupported,
|
||||
);
|
||||
};
|
||||
const directions = {
|
||||
px: [
|
||||
@@ -422,7 +233,7 @@ window.renderFace = (p, face, size) => {
|
||||
camera.up.set(...up);
|
||||
camera.lookAt(p.x + dir[0], p.y + dir[1], p.z + dir[2]);
|
||||
renderer.render(scene, camera);
|
||||
return renderer.domElement.toDataURL("image/png");
|
||||
return renderer.domElement.toDataURL("image/webp", 0.85);
|
||||
};
|
||||
window.renderMap = (size) => {
|
||||
const b = window.terrainBounds;
|
||||
@@ -439,6 +250,6 @@ window.renderMap = (size) => {
|
||||
camera.up.set(0, 0, -1);
|
||||
camera.lookAt(camera.position.x, 0, camera.position.z);
|
||||
renderer.render(scene, camera);
|
||||
return renderer.domElement.toDataURL("image/png");
|
||||
return renderer.domElement.toDataURL("image/webp", 0.9);
|
||||
};
|
||||
window.rendererReady = true;
|
||||
|
||||
329
renderer/terrain-mesh.js
Normal file
329
renderer/terrain-mesh.js
Normal file
@@ -0,0 +1,329 @@
|
||||
import * as THREE from "/three.js";
|
||||
import { OpaqueIndex, QuadBuffer } from "./mesh-buffers.mjs";
|
||||
|
||||
const nonOpaque =
|
||||
/air|water|lava|leaves|glass|grass(?!_block)|fern|flower|sapling|torch|fence|wall|slab|stairs|door|pane|rail|vine|carpet/;
|
||||
const fluid = /^minecraft:(water|lava)(\[|$)/;
|
||||
const uvOrder = [0, 1, 3, 2];
|
||||
|
||||
// Transform each distinct state's faces once, including element rotations,
|
||||
// variant rotations, tint and UV orientation. The block loop only translates.
|
||||
async function compileState(
|
||||
state,
|
||||
variants,
|
||||
faceData,
|
||||
textureRef,
|
||||
texture,
|
||||
materials,
|
||||
unsupported,
|
||||
) {
|
||||
const faces = [];
|
||||
for (const { m, v = {} } of variants) {
|
||||
const rotation = new THREE.Matrix4().makeRotationFromEuler(
|
||||
new THREE.Euler(
|
||||
(-(v.x || 0) * Math.PI) / 180,
|
||||
(-(v.y || 0) * Math.PI) / 180,
|
||||
0,
|
||||
"YXZ",
|
||||
),
|
||||
);
|
||||
for (const e of m.elements || []) {
|
||||
const from = e.from.map((n) => n / 16),
|
||||
to = e.to.map((n) => n / 16);
|
||||
for (const [dir, f] of Object.entries(e.faces)) {
|
||||
const fd = faceData[dir];
|
||||
if (!fd) continue;
|
||||
const ref = textureRef(m, f.texture);
|
||||
const tex = await texture(ref);
|
||||
if (!tex) {
|
||||
unsupported.add(state);
|
||||
continue;
|
||||
}
|
||||
const tint =
|
||||
f.tintindex !== undefined
|
||||
? state.includes("water")
|
||||
? "#5b9eb5"
|
||||
: state.includes("leaves")
|
||||
? "#7ba54c"
|
||||
: "#90b650"
|
||||
: "#ffffff";
|
||||
const key = ref + "|" + tint;
|
||||
let material = materials.get(key);
|
||||
if (!material) {
|
||||
const water = ref?.includes("water");
|
||||
material = {
|
||||
id: materials.size,
|
||||
mesh: new THREE.MeshLambertMaterial({
|
||||
map: tex,
|
||||
color: tint,
|
||||
side: THREE.DoubleSide,
|
||||
alphaTest: 0.3,
|
||||
transparent: water,
|
||||
opacity: water ? 0.8 : 1,
|
||||
}),
|
||||
};
|
||||
materials.set(key, material);
|
||||
}
|
||||
const normal = new THREE.Vector3(...fd.normal).applyMatrix4(rotation);
|
||||
const positions = new Float64Array(12),
|
||||
uvs = new Float32Array(8);
|
||||
const uv = f.uv || [0, 0, 16, 16];
|
||||
const coords = [
|
||||
[uv[0] / 16, 1 - uv[3] / 16],
|
||||
[uv[2] / 16, 1 - uv[3] / 16],
|
||||
[uv[0] / 16, 1 - uv[1] / 16],
|
||||
[uv[2] / 16, 1 - uv[1] / 16],
|
||||
];
|
||||
for (let i = 0; i < 4; i++) {
|
||||
const p = new THREE.Vector3(
|
||||
...fd.corners[i].map((c, j) => from[j] + c * (to[j] - from[j])),
|
||||
);
|
||||
if (e.rotation) {
|
||||
const er = e.rotation,
|
||||
origin = new THREE.Vector3(...er.origin.map((a) => a / 16));
|
||||
const axis = new THREE.Vector3(
|
||||
er.axis === "x" ? 1 : 0,
|
||||
er.axis === "y" ? 1 : 0,
|
||||
er.axis === "z" ? 1 : 0,
|
||||
);
|
||||
p.sub(origin).applyAxisAngle(axis, (er.angle * Math.PI) / 180);
|
||||
if (er.rescale) {
|
||||
const scale = 1 / Math.cos((er.angle * Math.PI) / 180);
|
||||
for (const a of ["x", "y", "z"]) if (a !== er.axis) p[a] *= scale;
|
||||
}
|
||||
p.add(origin);
|
||||
}
|
||||
p.subScalar(0.5).applyMatrix4(rotation).addScalar(0.5);
|
||||
positions.set([p.x, p.y, p.z], i * 3);
|
||||
const index = f.rotation
|
||||
? uvOrder[(uvOrder.indexOf(i) + f.rotation / 90) % 4]
|
||||
: i;
|
||||
uvs.set(coords[index], i * 2);
|
||||
}
|
||||
faces.push({
|
||||
positions,
|
||||
uvs,
|
||||
normal: [normal.x, normal.y, normal.z],
|
||||
cull: f.cullface
|
||||
? [Math.round(normal.x), Math.round(normal.y), Math.round(normal.z)]
|
||||
: null,
|
||||
material,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
return faces;
|
||||
}
|
||||
|
||||
function fluidModel(state, faceData) {
|
||||
return {
|
||||
m: {
|
||||
textures: {
|
||||
all:
|
||||
"minecraft:block/" +
|
||||
(state.includes("water") ? "water_still" : "lava_still"),
|
||||
},
|
||||
elements: [
|
||||
{
|
||||
from: [0, 0, 0],
|
||||
to: [16, 14, 16],
|
||||
faces: Object.fromEntries(
|
||||
Object.keys(faceData).map((k) => [
|
||||
k,
|
||||
{
|
||||
texture: "#all",
|
||||
tintindex: state.includes("water") ? 0 : undefined,
|
||||
cullface: k,
|
||||
},
|
||||
]),
|
||||
),
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function buildTerrain(
|
||||
snapshot,
|
||||
scene,
|
||||
faceData,
|
||||
stateModels,
|
||||
textureRef,
|
||||
texture,
|
||||
unsupported,
|
||||
) {
|
||||
const started = performance.now(),
|
||||
states = new Map(),
|
||||
materials = new Map();
|
||||
let minY = Infinity,
|
||||
maxY = -Infinity,
|
||||
lastProgress = 0;
|
||||
async function progress(percent, message, force = false) {
|
||||
if (!force && performance.now() - lastProgress < 3000) return;
|
||||
lastProgress = performance.now();
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
await window.minegeoProgress?.({ percent, message });
|
||||
}
|
||||
// Time-sliced passes keep progress/cancellation responsive for large exports.
|
||||
if (!snapshot.blocks.length)
|
||||
throw new Error("Terrain snapshot contains no blocks");
|
||||
await progress(
|
||||
25,
|
||||
`Indexing ${snapshot.blocks.length.toLocaleString()} terrain blocks`,
|
||||
true,
|
||||
);
|
||||
for (let i = 0; i < snapshot.blocks.length; i++) {
|
||||
const b = snapshot.blocks[i];
|
||||
if (b[1] < minY) minY = b[1];
|
||||
if (b[1] > maxY) maxY = b[1];
|
||||
if (!states.has(b[3]))
|
||||
states.set(b[3], { opaque: !nonOpaque.test(b[3]), faces: null });
|
||||
if (i % 16384 === 0)
|
||||
await progress(
|
||||
25 + (2 * i) / snapshot.blocks.length,
|
||||
`Indexing blocks ${i}/${snapshot.blocks.length}`,
|
||||
);
|
||||
}
|
||||
const occupancy = new OpaqueIndex(snapshot.bounds, minY, maxY);
|
||||
for (let i = 0; i < snapshot.blocks.length; i++) {
|
||||
const b = snapshot.blocks[i];
|
||||
if (states.get(b[3]).opaque) occupancy.add(b[0], b[1], b[2]);
|
||||
if (i % 16384 === 0)
|
||||
await progress(
|
||||
27 + (3 * i) / snapshot.blocks.length,
|
||||
`Indexing hidden faces ${i}/${snapshot.blocks.length}`,
|
||||
);
|
||||
}
|
||||
const indexMs = performance.now() - started;
|
||||
await progress(30, "Compiling block models and textures", true);
|
||||
const entries = [...states.entries()];
|
||||
let completedModels = 0;
|
||||
// Bounded parallel I/O; failed textures are negatively cached by the loader.
|
||||
// Keep progress reporting serialized while compiler tasks perform I/O.
|
||||
for (let i = 0; i < entries.length; i += 8) {
|
||||
const end = Math.min(i + 8, entries.length);
|
||||
await Promise.all(
|
||||
entries.slice(i, end).map(async ([state, entry]) => {
|
||||
const variants = fluid.test(state)
|
||||
? [fluidModel(state, faceData)]
|
||||
: await stateModels(state);
|
||||
entry.faces = await compileState(
|
||||
state,
|
||||
variants,
|
||||
faceData,
|
||||
textureRef,
|
||||
texture,
|
||||
materials,
|
||||
unsupported,
|
||||
);
|
||||
completedModels++;
|
||||
}),
|
||||
);
|
||||
await progress(
|
||||
30 + (5 * completedModels) / entries.length,
|
||||
`Compiled block models ${completedModels}/${entries.length}`,
|
||||
);
|
||||
}
|
||||
const modelMs = performance.now() - started - indexMs;
|
||||
await progress(35, "Building cached block faces", true);
|
||||
const batches = new Map(),
|
||||
meshList = [],
|
||||
materialCount = materials.size || 1;
|
||||
const originX = Math.floor(snapshot.bounds.minX / 64),
|
||||
originZ = Math.floor(snapshot.bounds.minZ / 64);
|
||||
const regionWidth = Math.floor(snapshot.bounds.maxX / 64) - originX + 1;
|
||||
let emittedFaces = 0,
|
||||
hiddenFaces = 0;
|
||||
function flush(buffer, material) {
|
||||
if (!buffer.count) return;
|
||||
const data = buffer.finish(),
|
||||
g = new THREE.BufferGeometry();
|
||||
g.setAttribute("position", new THREE.BufferAttribute(data.positions, 3));
|
||||
g.setAttribute("normal", new THREE.BufferAttribute(data.normals, 3));
|
||||
g.setAttribute("uv", new THREE.BufferAttribute(data.uvs, 2));
|
||||
g.setIndex(new THREE.BufferAttribute(data.indices, 1));
|
||||
g.boundingBox = new THREE.Box3(
|
||||
new THREE.Vector3(...data.min),
|
||||
new THREE.Vector3(...data.max),
|
||||
);
|
||||
g.boundingSphere = g.boundingBox.getBoundingSphere(new THREE.Sphere());
|
||||
const mesh = new THREE.Mesh(g, material.mesh);
|
||||
scene.add(mesh);
|
||||
meshList.push(mesh);
|
||||
}
|
||||
for (let i = 0; i < snapshot.blocks.length; i++) {
|
||||
const b = snapshot.blocks[i],
|
||||
x = b[0],
|
||||
y = b[1],
|
||||
z = b[2];
|
||||
const entry = states.get(b[3]);
|
||||
const region =
|
||||
Math.floor(x / 64) -
|
||||
originX +
|
||||
(Math.floor(z / 64) - originZ) * regionWidth;
|
||||
for (const face of entry.faces) {
|
||||
const n = face.cull;
|
||||
if (n && occupancy.has(x + n[0], y + n[1], z + n[2])) {
|
||||
hiddenFaces++;
|
||||
continue;
|
||||
}
|
||||
const key = region * materialCount + face.material.id;
|
||||
let batch = batches.get(key);
|
||||
if (!batch) {
|
||||
batch = { buffer: new QuadBuffer(), material: face.material };
|
||||
batches.set(key, batch);
|
||||
}
|
||||
batch.buffer.add(face, x, y, z);
|
||||
emittedFaces++;
|
||||
if (batch.buffer.full) {
|
||||
flush(batch.buffer, batch.material);
|
||||
batch.buffer = new QuadBuffer();
|
||||
}
|
||||
}
|
||||
if (i % 4096 === 0)
|
||||
await progress(
|
||||
35 + (20 * i) / snapshot.blocks.length,
|
||||
`Building blocks ${i}/${snapshot.blocks.length}; ${Math.round(i / Math.max(0.001, (performance.now() - started - indexMs - modelMs) / 1000)).toLocaleString()} blocks/s; ${hiddenFaces.toLocaleString()} hidden faces skipped`,
|
||||
);
|
||||
}
|
||||
occupancy.clear();
|
||||
const meshMs = performance.now() - started - indexMs - modelMs;
|
||||
await progress(55, "Preparing spatial meshes", true);
|
||||
let prepared = 0;
|
||||
for (const [key, batch] of batches) {
|
||||
flush(batch.buffer, batch.material);
|
||||
batches.delete(key);
|
||||
await progress(
|
||||
55 + (5 * ++prepared) / (prepared + batches.size),
|
||||
`Preparing geometry ${prepared}/${prepared + batches.size}`,
|
||||
);
|
||||
}
|
||||
window.terrainBounds = snapshot.bounds;
|
||||
window.terrainMeshes = meshList;
|
||||
const totalMs = performance.now() - started;
|
||||
return {
|
||||
blocks: snapshot.blocks.length,
|
||||
states: states.size,
|
||||
meshes: meshList.length,
|
||||
drawCalls: meshList.length,
|
||||
emittedFaces,
|
||||
hiddenFaces,
|
||||
unsupported: [...unsupported],
|
||||
indexMs,
|
||||
modelMs,
|
||||
meshMs,
|
||||
buildMs: totalMs,
|
||||
blocksPerSecond: Math.round(
|
||||
snapshot.blocks.length / Math.max(0.001, meshMs / 1000),
|
||||
),
|
||||
geometryBytes: meshList.reduce(
|
||||
(sum, mesh) =>
|
||||
sum +
|
||||
mesh.geometry.attributes.position.array.byteLength +
|
||||
mesh.geometry.attributes.normal.array.byteLength +
|
||||
mesh.geometry.attributes.uv.array.byteLength +
|
||||
mesh.geometry.index.array.byteLength,
|
||||
0,
|
||||
),
|
||||
};
|
||||
}
|
||||
@@ -15,8 +15,13 @@ const client = createClient<Database>(
|
||||
);
|
||||
const worker = "worker-" + randomUUID();
|
||||
let stopping = false;
|
||||
process.on("SIGINT", () => (stopping = true));
|
||||
process.on("SIGTERM", () => (stopping = true));
|
||||
let activeAbort: AbortController | null = null;
|
||||
function stopWorker() {
|
||||
stopping = true;
|
||||
activeAbort?.abort();
|
||||
}
|
||||
process.on("SIGINT", stopWorker);
|
||||
process.on("SIGTERM", stopWorker);
|
||||
function check<T>(r: {
|
||||
data: T;
|
||||
error: { message: string } | null;
|
||||
@@ -31,6 +36,7 @@ async function processJob(job: {
|
||||
}) {
|
||||
const abort = new AbortController(),
|
||||
directory = `.cache/jobs/${job.id}`;
|
||||
activeAbort = abort;
|
||||
await mkdir(directory, { recursive: true });
|
||||
let lastProgressWrite = 0;
|
||||
let lastLoggedAt = 0;
|
||||
@@ -146,18 +152,30 @@ async function processJob(job: {
|
||||
)
|
||||
)
|
||||
throw new Error("Snapshot bounds disagree with manifest");
|
||||
for (const b of terrain.blocks) {
|
||||
for (let i = 0; i < terrain.blocks.length; i++) {
|
||||
const b = terrain.blocks[i];
|
||||
if (
|
||||
!Array.isArray(b) ||
|
||||
b.length < 4 ||
|
||||
!b.slice(0, 3).every((v) => Number.isInteger(v)) ||
|
||||
!Number.isSafeInteger(b[0]) ||
|
||||
!Number.isSafeInteger(b[1]) ||
|
||||
!Number.isSafeInteger(b[2]) ||
|
||||
b[0] < terrain.bounds.minX ||
|
||||
b[0] > terrain.bounds.maxX ||
|
||||
b[2] < terrain.bounds.minZ ||
|
||||
b[2] > terrain.bounds.maxZ ||
|
||||
b[1] < -4096 ||
|
||||
b[1] > 4096 ||
|
||||
typeof b[3] !== "string" ||
|
||||
!/^minecraft:[a-z0-9_]+(\[[a-z0-9_=,]+\])?$/.test(b[3])
|
||||
)
|
||||
throw new Error("Invalid block record");
|
||||
// Avoid starving lease heartbeats while validating millions of records.
|
||||
if (i % 65536 === 0)
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
}
|
||||
console.log(
|
||||
`Job ${job.id}: ${terrain.blocks.length.toLocaleString()} blocks, ${manifest.viewpoints.length} panoramas; software rendering may take time.`,
|
||||
`Job ${job.id}: ${terrain.blocks.length.toLocaleString()} blocks, ${manifest.viewpoints.length} panoramas; starting cached geometry builder.`,
|
||||
);
|
||||
const report = await renderSnapshot(
|
||||
terrain,
|
||||
@@ -217,28 +235,43 @@ async function processJob(job: {
|
||||
const images = (await readdir(output, { recursive: true })).filter(
|
||||
(entry) => entry.endsWith(".webp"),
|
||||
);
|
||||
let uploaded = 0;
|
||||
for (const entry of images) {
|
||||
if (abort.signal.aborted) throw new Error("Lease lost or cancelled");
|
||||
const normalized = entry.replaceAll("\\", "/");
|
||||
check(
|
||||
await client.storage
|
||||
.from("minegeo-media")
|
||||
.upload(
|
||||
`${job.dataset_id}/${normalized}`,
|
||||
await readFile(`${output}/${entry}`),
|
||||
{
|
||||
contentType: "image/webp",
|
||||
upsert: true,
|
||||
cacheControl: "31536000",
|
||||
},
|
||||
),
|
||||
);
|
||||
await progress(
|
||||
90 + (9 * ++uploaded) / images.length,
|
||||
`Saved ${uploaded}/${images.length} processed images`,
|
||||
);
|
||||
}
|
||||
let uploaded = 0,
|
||||
nextImage = 0;
|
||||
let uploadError: unknown;
|
||||
// Small bounded pool avoids paying Storage round-trip latency serially.
|
||||
// Finish every in-flight operation before cleanup/retry can touch the files.
|
||||
await Promise.all(
|
||||
Array.from({ length: Math.min(4, images.length) }, async () => {
|
||||
try {
|
||||
while (nextImage < images.length && !uploadError) {
|
||||
if (abort.signal.aborted)
|
||||
throw new Error("Lease lost or cancelled");
|
||||
const entry = images[nextImage++],
|
||||
normalized = entry.replaceAll("\\", "/");
|
||||
check(
|
||||
await client.storage
|
||||
.from("minegeo-media")
|
||||
.upload(
|
||||
`${job.dataset_id}/${normalized}`,
|
||||
await readFile(`${output}/${entry}`),
|
||||
{
|
||||
contentType: "image/webp",
|
||||
upsert: true,
|
||||
cacheControl: "31536000",
|
||||
},
|
||||
),
|
||||
);
|
||||
await progress(
|
||||
90 + (9 * ++uploaded) / images.length,
|
||||
`Saved ${uploaded}/${images.length} processed images`,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
uploadError ??= error;
|
||||
}
|
||||
}),
|
||||
);
|
||||
if (uploadError) throw uploadError;
|
||||
if (abort.signal.aborted) throw new Error("Lease lost or cancelled");
|
||||
check(
|
||||
await client.rpc("complete_job", {
|
||||
@@ -253,11 +286,18 @@ async function processJob(job: {
|
||||
const message =
|
||||
error instanceof Error ? error.message : "Processing failed";
|
||||
console.error(`Job ${job.id}: ${message}`);
|
||||
const state = job.attempts >= 3 ? "failed" : "queued";
|
||||
const state = stopping ? "queued" : job.attempts >= 3 ? "failed" : "queued";
|
||||
const rows = check(
|
||||
await client
|
||||
.from("upload_jobs")
|
||||
.update({ state, error: message, lease_until: null, progress: 0 })
|
||||
.update({
|
||||
state,
|
||||
error: stopping ? null : message,
|
||||
lease_owner: null,
|
||||
lease_until: null,
|
||||
progress: 0,
|
||||
attempts: stopping ? Math.max(0, job.attempts - 1) : job.attempts,
|
||||
})
|
||||
.eq("id", job.id)
|
||||
.eq("lease_owner", worker)
|
||||
.eq("state", "running")
|
||||
@@ -270,6 +310,7 @@ async function processJob(job: {
|
||||
.eq("id", job.dataset_id)
|
||||
.eq("state", "processing");
|
||||
} finally {
|
||||
if (activeAbort === abort) activeAbort = null;
|
||||
clearInterval(heartbeat);
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user