diff --git a/addons/addon-image/README.md b/addons/addon-image/README.md index a18ce18bb..80c88de8e 100644 --- a/addons/addon-image/README.md +++ b/addons/addon-image/README.md @@ -1,6 +1,7 @@ ## @xterm/addon-image -Inline image output in xterm.js. Supports SIXEL and iTerm's inline image protocol (IIP). +Inline image output in xterm.js. Supports SIXEL, iTerm's inline image protocol (IIP) +and partially Kitty's terminal graphics prtocol (TGP). ![](fixture/example.png) @@ -43,9 +44,6 @@ terminal.loadAddon(imageAddon); ### General Notes -- *IMPORTANT:* The worker approach as done in previous releases got removed. - The addon contructor no longer expects a worker path as first argument. - - By default the addon will activate these `windowOptions` reports on the terminal: - getWinSizePixels (CSI 14 t) - getCellSizePixels (CSI 16 t) @@ -73,7 +71,7 @@ terminal.loadAddon(imageAddon); - **Cursor Positioning** If scrolling is set, the cursor will be placed at the first image column of the last image row (VT340 mode). - Other cursor positioning modes as used by xterm or mintty are not supported. + Other cursor positioning modes as used by xterm or mintty are not supported for sixel. - **SIXEL Palette Handling** By default the addon limits the palette size to 256 registers (as demanded by the DEC specification). @@ -100,17 +98,24 @@ terminal.loadAddon(imageAddon); Set by default, change it with `{iipSupport: true}`. The IIP implementation has the following features / restrictions (sequence will silently fail for unmet conditions): - - Supported formats: PNG, JPEG and GIF + - Supported formats: PNG, JPEG, GIF and QOI - No animation support. - Image type hinting is not supported (always deducted from data header). - File download is not supported. - Filename gets parsed but not used. - Strict base64 handling as of RFC4648 §4 (standard alphabet, optional padding, no separator bytes allowed). - - Payload size may not exceed CEIL(sizeParameter * 4 / 3). - Image scaling beyond terminal viewport size is allowed (e.g. `width=200%`). - Image pixel size is restricted by `pixelLimit` (pre- and post resizing). - - Size parameter is restricted by `iipSizeLimit`. - - Cursor positioning behaves the same as for sixel (see above). + - Playload size is restricted by `iipSizeLimit`. + - size parameter is not mandatory anymore. + - Cursor gets positioned at the next cell of the last image line. + - ReportCellSize sequence is supported. + - Hi-res display support is currently limited (image gets blurred). + +- **Kitty Graphics Support (TGP)** + Set by default, change it with `{kittySupport: true}`. + + Note that the kitty graphics support is still WIP. ### Storage and Drawing Settings @@ -127,6 +132,7 @@ The addon exposes two properties to interact with the storage limits at runtime: By default the addon will show a placeholder pattern for evicted images that are still part of the terminal (e.g. in the scrollback). The pattern can be deactivated by toggling `showPlaceholder`. + ### Image Data Retrieval The addon provides the following API endpoints to retrieve raw image data as canvas: @@ -140,6 +146,7 @@ The addon provides the following API endpoints to retrieve raw image data as can The buffer position is the 0-based absolute index (including scrollback at top). Note that the canvas gets created and data copied over for every call, thus it is not suitable for performance critical actions. + ### Memory Usage The addon does most image processing in Javascript and therefore can occupy a rather big amount of memory. To get an idea where the memory gets eaten, lets look at the data flow and processing steps: @@ -180,8 +187,8 @@ _How can I adjust the memory usage?_ A constructor setting, thus you would have to anticipate, whether (multiple) terminals in your page gonna do lots of concurrent decoding. Since this is normally not the case and the memory usage is only temporarily peaking, a rather high value should work even with multiple terminals in one page. - `storageLimit` A constructor and a runtime setting. In conjunction with `storageUsage` you can do runtime checks and adjust the limit to your needs. If you have to lower the limit below the current usage, images will be removed in FIFO order and may turn into a placeholder in the terminal's scrollback (if `showPlaceholder` is set). When adjusting keep in mind to leave enough room for memory peaking for decoding. -- `sixelSizeLimit` - A constructor setting. This has only a small impact on peaking memory during decoding. It is meant to avoid processing of overly big or broken SIXEL sequences at an earlier phase, thus may stop the decoder from entering its memory intensive task for potentially invalid data. +- `sixelSizeLimit | iipSizeLimit | kittySizeLimit` + Constructor settings. This has only a small impact on peaking memory during decoding. It is meant to avoid processing of overly big or broken image sequences at an earlier phase, thus may stop the invoked decoders from entering memory intensive tasks for potentially invalid data. ### Terminal Interaction @@ -211,23 +218,5 @@ _How can I adjust the memory usage?_ ### Status -- Sixel support and image handling in xterm.js is considered beta quality. -- IIP support is in alpha stage. Please file a bug for any awkwardities. - - -### Changelog - -- 0.5.0 integrate with xtermjs base repo (at v0.4.3) -- 0.4.3 defer canvas creation -- 0.4.2 fix image canvas resize -- 0.4.1 compat release for xterm.js 5.2.0 -- 0.4.0 IIP support -- 0.3.1 compat release for xterm.js 5.1.0 -- 0.3.0 important change: worker removed from addon -- 0.2.0 compat release for xterm.js 5.0.0 -- 0.1.3 bugfix: avoid striping -- 0.1.2 bugfix: reset clear flag -- 0.1.1 bugfixes: - - clear sticky image tiles on render - - create folder from bootstrap.sh - - fix peer dependency in package.json +- Sixel and IIP support and image handling in xterm.js is considered beta quality. +- Kitty support is in alpha stage. Please file a bug for any awkwardities or missing features. diff --git a/addons/addon-image/fixture/testimages/dice.qoi b/addons/addon-image/fixture/testimages/dice.qoi new file mode 100644 index 000000000..bbc8154fc Binary files /dev/null and b/addons/addon-image/fixture/testimages/dice.qoi differ diff --git a/addons/addon-image/src/IIPHandler.ts b/addons/addon-image/src/IIPHandler.ts index e35687c10..5e15c5277 100644 --- a/addons/addon-image/src/IIPHandler.ts +++ b/addons/addon-image/src/IIPHandler.ts @@ -8,7 +8,7 @@ import { IIPImageStorage } from './IIPImageStorage'; import { CELL_SIZE_DEFAULT } from './ImageStorage'; import Base64Decoder from 'xterm-wasm-parts/lib/base64/Base64Decoder.wasm'; import QoiDecoder from 'xterm-wasm-parts/lib/qoi/QoiDecoder.wasm'; -import { HeaderParser, IHeaderFields, HeaderState } from './IIPHeaderParser'; +import { HeaderParser, IHeaderFields, HeaderState, SequenceType } from './IIPHeaderParser'; import { imageType, UNSUPPORTED_TYPE } from './IIPMetrics'; // Local const enum mirror - esbuild can't inline const enums from external packages @@ -23,6 +23,7 @@ const enum DecoderConst { // default IIP header values const DEFAULT_HEADER: IHeaderFields = { + type: SequenceType.INVALID, name: 'Unnamed file', size: 0, width: 'auto', @@ -39,6 +40,8 @@ export class IIPHandler implements IOscHandler, IResetHandler { private _dec: Base64Decoder; private _qoiDec: QoiDecoder; private _metrics = UNSUPPORTED_TYPE; + private _isMultipart = false; + private _abortMulti = false; constructor( private readonly _opts: IImageAddonOptions, @@ -52,11 +55,14 @@ export class IIPHandler implements IOscHandler, IResetHandler { this._qoiDec = new QoiDecoder(DecoderConst.KEEP_DATA); } - public reset(): void {} + public reset(): void { + this._hp.reset(); + this._dec.release(); + this._qoiDec.release(); + } public start(): void { this._aborted = false; - this._header = DEFAULT_HEADER; this._metrics = UNSUPPORTED_TYPE; this._hp.reset(); } @@ -76,15 +82,27 @@ export class IIPHandler implements IOscHandler, IResetHandler { return; } if (dataPos > 0) { - this._header = Object.assign({}, DEFAULT_HEADER, this._hp.fields); - if (!this._header.inline || !this._header.size || this._header.size > this._opts.iipSizeLimit) { + const seqType = this._hp.fields.type; + if (seqType === SequenceType.FILE) { + if (this._isMultipart) { + this._isMultipart = false; + this._abortMulti = false; + this._dec.release(); + } + this._header = Object.assign({}, DEFAULT_HEADER, this._hp.fields); + if (!this._header.inline) { + this._aborted = true; + return; + } + this._dec.init(); + } else if (this._abortMulti) { this._aborted = true; return; } - this._dec.init(); if ((this._dec.put(data.subarray(dataPos, end)) as number) !== DecoderConst.OK) { this._dec.release(); this._aborted = true; + if (this._isMultipart) this._abortMulti = true; } } } @@ -93,6 +111,44 @@ export class IIPHandler implements IOscHandler, IResetHandler { public end(success: boolean): boolean | Promise { if (this._aborted) return true; + if (this._hp.state !== HeaderState.END) { + if (this._hp.end()) return true; + } + const seqType = this._hp.fields.type; + + if (seqType === SequenceType.FILEPART) return true; + + if (seqType === SequenceType.REPORTCELLSIZE) { + // OSC 1337 ; ReportCellSize=[height];[width];[scale] ST + let width = CELL_SIZE_DEFAULT.width; + let height = CELL_SIZE_DEFAULT.height; + if (this._renderer.dimensions) { + width = this._renderer.dimensions.css.canvas.width / this._coreTerminal.cols; + height = this._renderer.dimensions.css.canvas.height / this._coreTerminal.rows; + } + const scale = this._coreTerminal._core._coreBrowserService?.dpr ?? 1; + const report = `\x1b]1337;ReportCellSize=${height.toFixed(3)};${width.toFixed(3)};${scale.toFixed(3)}\x1b\\`; + this._coreTerminal.input(report, false); + return true; + } + + if (seqType === SequenceType.MULTIPARTFILE) { + this._header = Object.assign({}, DEFAULT_HEADER, this._hp.fields); + this._isMultipart = true; + this._abortMulti = false; + this._dec.release(); + this._dec.init(); + return true; + } + + if (seqType === SequenceType.FILEEND) { + if (!this._isMultipart) return true; + this._isMultipart = false; + if (this._abortMulti || this._header.type !== SequenceType.MULTIPARTFILE) return true; + } + + // fallthrough for SequenceType.FILE & SequenceType.FILEEND + let w = 0; let h = 0; @@ -100,15 +156,13 @@ export class IIPHandler implements IOscHandler, IResetHandler { let cond: number | boolean; if (cond = success) { if (cond = !this._dec.end()) { - if (cond = this._dec.data8.length === this._header.size) { - this._metrics = imageType(this._dec.data8); - if (cond = this._metrics.mime !== 'unsupported') { - w = this._metrics.width; - h = this._metrics.height; - if (cond = w && h && w * h < this._opts.pixelLimit) { - [w, h] = this._resize(w, h).map(Math.floor); - cond = w && h && w * h < this._opts.pixelLimit; - } + this._metrics = imageType(this._dec.data8); + if (cond = this._metrics.mime !== 'unsupported') { + w = this._metrics.width; + h = this._metrics.height; + if (cond = w && h && w * h < this._opts.pixelLimit) { + [w, h] = this._resize(w, h).map(Math.floor); + cond = w && h && w * h < this._opts.pixelLimit; } } } diff --git a/addons/addon-image/src/IIPHeaderParser.test.ts b/addons/addon-image/src/IIPHeaderParser.test.ts index e42782c14..ba6d5f052 100644 --- a/addons/addon-image/src/IIPHeaderParser.test.ts +++ b/addons/addon-image/src/IIPHeaderParser.test.ts @@ -4,19 +4,20 @@ */ import { assert } from 'chai'; -import { HeaderParser, HeaderState, IHeaderFields } from './IIPHeaderParser'; +import { HeaderParser, HeaderState, IHeaderFields, SequenceType } from './IIPHeaderParser'; const CASES: [string, IHeaderFields][] = [ - ['File=size=123456;name=dGVzdA==:', {name: 'test', size: 123456}], - ['File=size=123456;name=dGVzdA:', {name: 'test', size: 123456}], + ['File=size=123456;name=dGVzdA==:', {type: SequenceType.FILE, name: 'test', size: 123456}], + ['File=size=123456;name=dGVzdA:', {type: SequenceType.FILE, name: 'test', size: 123456}], // utf-8 encoding in name - ['File=size=123456;name=w7xtbMOkdXTDnw==:', {name: 'ümläutß', size: 123456}], - ['File=size=123456;name=w7xtbMOkdXTDnw:', {name: 'ümläutß', size: 123456}], + ['File=size=123456;name=w7xtbMOkdXTDnw==:', {type: SequenceType.FILE, name: 'ümläutß', size: 123456}], + ['File=size=123456;name=w7xtbMOkdXTDnw:', {type: SequenceType.FILE, name: 'ümläutß', size: 123456}], // full header spec [ 'File=inline=1;width=10px;height=20%;preserveAspectRatio=1;size=123456;name=w7xtbMOkdXTDnw:', { + type: SequenceType.FILE, inline: 1, width: '10px', height: '20%', @@ -28,6 +29,7 @@ const CASES: [string, IHeaderFields][] = [ [ 'File=inline=1;width=auto;height=20;preserveAspectRatio=1;size=123456;name=w7xtbMOkdXTDnw:', { + type: SequenceType.FILE, inline: 1, width: 'auto', height: '20', @@ -101,7 +103,7 @@ describe('IIPHeaderParser', () => { let res = hp.parse(inp, 0, inp.length); assert.strictEqual(res, inp.length); assert.strictEqual(hp.state, HeaderState.END); - assert.deepEqual(hp.fields, {name: 'test', size: 0}); + assert.deepEqual(hp.fields, {type: SequenceType.FILE, name: 'test', size: 0}); hp.reset(); inp = fromBs(CASES[0][0]); res = hp.parse(inp, 0, inp.length); @@ -115,7 +117,7 @@ describe('IIPHeaderParser', () => { let res = hp.parse(inp, 0, inp.length); assert.strictEqual(res, inp.length); assert.strictEqual(hp.state, HeaderState.END); - assert.deepEqual(hp.fields, {name: '', size: 123456}); + assert.deepEqual(hp.fields, {type: SequenceType.FILE, name: '', size: 123456}); hp.reset(); inp = fromBs(CASES[0][0]); res = hp.parse(inp, 0, inp.length); @@ -135,4 +137,124 @@ describe('IIPHeaderParser', () => { assert.strictEqual(hp.state, HeaderState.END); assert.deepEqual(hp.fields, CASES[0][1]); }); + it('FilePart sequence', () => { + const hp = new HeaderParser(); + const inp = fromBs('FilePart=w7xtbMOkdXTDnw'); + const res = hp.parse(inp, 0, inp.length); + assert.strictEqual(res, 9); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields.type, SequenceType.FILEPART); + }); + it('FilePart sequence - bytewise', () => { + const hp = new HeaderParser(); + const inp = fromBs('FilePart=w7xtbMOkdXTDnw'); + let res = -2; + let pos = 0; + while (res === -2 && pos < inp.length) { + res = hp.parse(new Uint32Array([inp[pos++]]), 0, 1); + } + assert.strictEqual(res, 1); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields.type, SequenceType.FILEPART); + }); + it('MultipartFile sequence', () => { + const hp = new HeaderParser(); + const inp = fromBs('MultipartFile=inline=1;width=10px;height=20%;preserveAspectRatio=1;size=123456;name=w7xtbMOkdXTDnw'); + let res = hp.parse(inp, 0, inp.length); + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.VALUE); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual( + hp.fields, + { + type: SequenceType.MULTIPARTFILE, + inline: 1, + width: '10px', + height: '20%', + preserveAspectRatio: 1, + size: 123456, + name: 'ümläutß' + } + ); + }); + it('MultipartFile sequence - bytewise', () => { + const hp = new HeaderParser(); + const inp = fromBs('MultipartFile=inline=1;width=10px;height=20%;preserveAspectRatio=1;size=123456;name=w7xtbMOkdXTDnw'); + let res = -2; + let pos = 0; + while (res === -2 && pos < inp.length) { + res = hp.parse(new Uint32Array([inp[pos++]]), 0, 1); + } + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.VALUE); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual( + hp.fields, + { + type: SequenceType.MULTIPARTFILE, + inline: 1, + width: '10px', + height: '20%', + preserveAspectRatio: 1, + size: 123456, + name: 'ümläutß' + } + ); + }); + it('FileEnd sequence', () => { + const hp = new HeaderParser(); + const inp = fromBs('FileEnd'); + let res = hp.parse(inp, 0, inp.length); + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.START); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields.type, SequenceType.FILEEND); + }); + it('FileEnd sequence - bytewise', () => { + const hp = new HeaderParser(); + const inp = fromBs('FileEnd'); + let res = -2; + let pos = 0; + while (res === -2 && pos < inp.length) { + res = hp.parse(new Uint32Array([inp[pos++]]), 0, 1); + } + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.START); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields.type, SequenceType.FILEEND); + }); + it('ReportCellSize sequence', () => { + const hp = new HeaderParser(); + const inp = fromBs('ReportCellSize'); + let res = hp.parse(inp, 0, inp.length); + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.START); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields, { type: SequenceType.REPORTCELLSIZE }); + }); + it('ReportCellSize sequence - bytewise', () => { + const hp = new HeaderParser(); + const inp = fromBs('ReportCellSize'); + let res = -2; + let pos = 0; + while (res === -2 && pos < inp.length) { + res = hp.parse(new Uint32Array([inp[pos++]]), 0, 1); + } + assert.strictEqual(res, -2); + assert.strictEqual(hp.state, HeaderState.START); + res = hp.end(); + assert.strictEqual(res, 0); + assert.strictEqual(hp.state, HeaderState.END); + assert.deepEqual(hp.fields, { type: SequenceType.REPORTCELLSIZE }); + }); }); diff --git a/addons/addon-image/src/IIPHeaderParser.ts b/addons/addon-image/src/IIPHeaderParser.ts index af4e9a68d..9c20262b1 100644 --- a/addons/addon-image/src/IIPHeaderParser.ts +++ b/addons/addon-image/src/IIPHeaderParser.ts @@ -6,9 +6,27 @@ // eslint-disable-next-line declare const Buffer: any; +export const enum HeaderState { + START = 0, + ABORT = 1, + KEY = 2, + VALUE = 3, + END = 4 +} + +export const enum SequenceType { + INVALID = 0, + FILE = 1, + MULTIPARTFILE = 2, + FILEPART = 3, + FILEEND = 4, + REPORTCELLSIZE = 5 +} export interface IHeaderFields { [key: string]: number | string | Uint32Array | null | undefined; + // sequence type + type: SequenceType; // base-64 encoded filename. Defaults to "Unnamed file". name: string; // File size in bytes. The file transfer will be canceled if this size is exceeded. @@ -29,14 +47,6 @@ export interface IHeaderFields { inline?: number; } -export const enum HeaderState { - START = 0, - ABORT = 1, - KEY = 2, - VALUE = 3, - END = 4 -} - // field value decoders // ASCII bytes to string @@ -92,7 +102,19 @@ const DECODERS: {[key: string]: (v: Uint32Array) => number | string} = { }; +// sequence type markers +// File const FILE_MARKER = [70, 105, 108, 101]; +// MultipartFile +const MULTIPARTFILE_MARKER = [77, 117, 108, 116, 105, 112, 97, 114, 116, 70, 105, 108, 101]; +// FilePart +const FILEPART_MARKER = [70, 105, 108, 101, 80, 97, 114, 116]; +// FileEnd +const FILEEND_MARKER = [70, 105, 108, 101, 69, 110, 100]; +// ReportCellSize +const REPORTCELLSIZE_MARKER = [82, 101, 112, 111, 114, 116, 67, 101, 108, 108, 83, 105, 122, 101]; + +// max allowed chars for sequence header const MAX_FIELDCHARS = 1024; @@ -111,12 +133,43 @@ export class HeaderParser { this._key = ''; } + public end(): number { + if (this.state === HeaderState.START) { + if (this._position === FILEEND_MARKER.length) { + for (let k = 0; k < FILEEND_MARKER.length; ++k) { + if (this._buffer[k] !== FILEEND_MARKER[k]) return this._a(); + } + this.fields['type'] = SequenceType.FILEEND; + this.state = HeaderState.END; + return 0; + } + if (this._position === REPORTCELLSIZE_MARKER.length) { + for (let k = 0; k < REPORTCELLSIZE_MARKER.length; ++k) { + if (this._buffer[k] !== REPORTCELLSIZE_MARKER[k]) return this._a(); + } + this.fields['type'] = SequenceType.REPORTCELLSIZE; + this.state = HeaderState.END; + return 0; + } + return this._a(); + } + if (this.state === HeaderState.END) return 0; + if (this.state === HeaderState.VALUE + && this.fields.type === SequenceType.MULTIPARTFILE + ) { + if (!this._storeValue(this._position)) return this._a(); + this.state = HeaderState.END; + return 0; + } + return this._a(); + } + public parse(data: Uint32Array, start: number, end: number): number { let state = this.state; let pos = this._position; const buffer = this._buffer; if (state === HeaderState.ABORT || state === HeaderState.END) return -1; - if (state === HeaderState.START && pos > 6) return -1; + if (state === HeaderState.START && pos > 14) return -1; for (let i = start; i < end; ++i) { const c = data[i]; switch (c) { @@ -127,8 +180,29 @@ export class HeaderParser { break; case 61: // = if (state === HeaderState.START) { - for (let k = 0; k < FILE_MARKER.length; ++k) { - if (buffer[k] !== FILE_MARKER[k]) return this._a(); + if (buffer[0] === 70) { + // 'File' or 'FilePart' + let k = 0; + for (; k < FILE_MARKER.length; ++k) { + if (buffer[k] !== FILE_MARKER[k]) return this._a(); + } + this.fields['type'] = SequenceType.FILE; + if (pos === FILEPART_MARKER.length) { + for (; k < FILEPART_MARKER.length; ++k) { + if (buffer[k] !== FILEPART_MARKER[k]) return this._a(); + } + this.fields['type'] = SequenceType.FILEPART; + this.state = HeaderState.END; + return i + 1; + } + } else if (buffer[0] === 77) { + // 'MultipartFile' + for (let k = 0; k < MULTIPARTFILE_MARKER.length; ++k) { + if (buffer[k] !== MULTIPARTFILE_MARKER[k]) return this._a(); + } + this.fields['type'] = SequenceType.MULTIPARTFILE; + } else { + return this._a(); } state = HeaderState.KEY; pos = 0; @@ -158,6 +232,7 @@ export class HeaderParser { } private _a(): number { + this.fields.type = SequenceType.INVALID; this.state = HeaderState.ABORT; return -1; } diff --git a/addons/addon-image/src/IIPImageStorage.ts b/addons/addon-image/src/IIPImageStorage.ts index 0da91607f..282fea9f0 100644 --- a/addons/addon-image/src/IIPImageStorage.ts +++ b/addons/addon-image/src/IIPImageStorage.ts @@ -3,6 +3,7 @@ * @license MIT */ +import { IAddImageOpts } from 'Types'; import { ImageStorage } from './ImageStorage'; /** @@ -12,6 +13,7 @@ import { ImageStorage } from './ImageStorage'; * - Always uses scrolling mode (cursor advances with image) */ export class IIPImageStorage { + private _addImageOpts: IAddImageOpts = { scrolling: true, layer: 'top', zIndex: 0, cursorPos: 'iip' }; constructor( private readonly _storage: ImageStorage ) {} @@ -21,6 +23,6 @@ export class IIPImageStorage { * Always uses scrolling mode — cursor advances past the image. */ public addImage(img: HTMLCanvasElement | ImageBitmap): void { - this._storage.addImage(img, true); + this._storage.addImage(img, this._addImageOpts); } } diff --git a/addons/addon-image/src/IIPMetrics.test.ts b/addons/addon-image/src/IIPMetrics.test.ts index 9fa476bf6..cc381e5cb 100644 --- a/addons/addon-image/src/IIPMetrics.test.ts +++ b/addons/addon-image/src/IIPMetrics.test.ts @@ -30,7 +30,8 @@ const TEST_IMAGES: [string, IMetrics][] = [ ['iphone_hdr_YES.jpg', { mime: 'image/jpeg', width: 3264, height: 2448 }], ['nikon-e950.jpg', { mime: 'image/jpeg', width: 800, height: 600 }], ['agfa-makernotes.jpg', { mime: 'image/jpeg', width: 8, height: 8 }], - ['sony-alpha-6000.jpg', { mime: 'image/jpeg', width: 6000, height: 4000 }] + ['sony-alpha-6000.jpg', { mime: 'image/jpeg', width: 6000, height: 4000 }], + ['dice.qoi', { mime: 'image/qoi', width: 800, height: 600 }], ]; diff --git a/addons/addon-image/src/ImageAddon.ts b/addons/addon-image/src/ImageAddon.ts index 7e0f29d3f..8fd395431 100644 --- a/addons/addon-image/src/ImageAddon.ts +++ b/addons/addon-image/src/ImageAddon.ts @@ -137,11 +137,6 @@ export class ImageAddon implements ITerminalAddon, IImageApi { // enable size reports if (this._opts.enableSizeReports) { - // const windowOptions = terminal.getOption('windowOptions'); - // windowOptions.getWinSizePixels = true; - // windowOptions.getCellSizePixels = true; - // windowOptions.getWinSizeChars = true; - // terminal.setOption('windowOptions', windowOptions); const windowOps = terminal.options.windowOptions ?? {}; windowOps.getWinSizePixels = true; windowOps.getCellSizePixels = true; @@ -260,7 +255,7 @@ export class ImageAddon implements ITerminalAddon, IImageApi { } private _report(s: string): void { - this._terminal?._core.coreService.triggerDataEvent(s); + this._terminal?._core.input(s, false); } private _decset(params: (number | number[])[]): boolean { diff --git a/addons/addon-image/src/ImageStorage.ts b/addons/addon-image/src/ImageStorage.ts index e5585c527..75851343c 100644 --- a/addons/addon-image/src/ImageStorage.ts +++ b/addons/addon-image/src/ImageStorage.ts @@ -5,7 +5,11 @@ import { IDisposable } from '@xterm/xterm'; import { ImageRenderer } from './ImageRenderer'; -import { ITerminalExt, IExtendedAttrsImage, IImageAddonOptions, IImageSpec, IBufferLineExt, BgFlags, Cell, Content, ICellSize, ExtFlags, Attributes, UnderlineStyle, ImageLayer } from './Types'; +import { + ITerminalExt, IExtendedAttrsImage, IImageAddonOptions, IImageSpec, + IBufferLineExt, BgFlags, Cell, Content, ICellSize, ExtFlags, Attributes, + UnderlineStyle, IAddImageOpts +} from './Types'; // fallback default cell size @@ -235,13 +239,15 @@ export class ImageStorage implements IDisposable { /** * Method to add an image to the storage. * @param img - The image to add (canvas or bitmap). - * @param scrolling - When true, cursor advances with the image (lineFeed per row). - * When false, image is placed at (0,0) and cursor is restored (DECSET 80 / sixel origin mode). - * @param layer - Which canvas layer to render on ('top' or 'bottom'). - * @param zIndex - Z-index for image layering within the same layer. + * @param opts - Options for addImage: + * - scrolling: When true, cursor advances with the image. + * When false, image is placed at ORIGIN and cursor does not move. + * - layer: Which canvas layer to render on ('top' or 'bottom'). + * - zIndex: Z-index for image layering within the same layer. + * - cursorPos: 'vt340' for bottom-left, 'iip' for bottom.right. * @returns The internal image ID assigned to the stored image. */ - public addImage(img: HTMLCanvasElement | ImageBitmap, scrolling: boolean, layer: ImageLayer = 'top', zIndex: number = 0): number { + public addImage(img: HTMLCanvasElement | ImageBitmap, opts: IAddImageOpts): number { // never allow storage to exceed memory limit this._evictOldest(img.width * img.height); @@ -263,7 +269,7 @@ export class ImageStorage implements IDisposable { let offset = originX; let tileCount = 0; - if (!scrolling) { + if (!opts.scrolling) { buffer.x = 0; buffer.y = 0; offset = 0; @@ -277,7 +283,7 @@ export class ImageStorage implements IDisposable { this._writeToCell(line as IBufferLineExt, offset + col, imageId, row * cols + col); tileCount++; } - if (scrolling) { + if (opts.scrolling) { if (row < rows - 1) this._terminal._core._inputHandler.lineFeed(); } else { if (++buffer.y >= termRows) break; @@ -287,8 +293,12 @@ export class ImageStorage implements IDisposable { this._terminal._core._inputHandler._dirtyRowTracker.markDirty(buffer.y); // cursor positioning modes - if (scrolling) { - buffer.x = offset; + if (opts.scrolling) { + if (opts.cursorPos === 'iip') { + buffer.x = Math.min(offset + cols, termCols); + } else { + buffer.x = offset; + } } else { buffer.x = originX; buffer.y = originY; @@ -331,8 +341,8 @@ export class ImageStorage implements IDisposable { marker: endMarker || undefined, tileCount, bufferType: this._terminal.buffer.active.type, - layer, - zIndex + layer: opts.layer, + zIndex: opts.zIndex }; // finally add the image diff --git a/addons/addon-image/src/SixelImageStorage.ts b/addons/addon-image/src/SixelImageStorage.ts index 4c5f555bb..c2376a68a 100644 --- a/addons/addon-image/src/SixelImageStorage.ts +++ b/addons/addon-image/src/SixelImageStorage.ts @@ -4,7 +4,7 @@ */ import { ImageStorage, CELL_SIZE_DEFAULT } from './ImageStorage'; -import { IImageAddonOptions, ITerminalExt } from './Types'; +import { IImageAddonOptions, ITerminalExt, IAddImageOpts } from './Types'; import { ImageRenderer } from './ImageRenderer'; /** @@ -15,6 +15,7 @@ import { ImageRenderer } from './ImageRenderer'; * - advanceCursor for empty sixels carrying only height */ export class SixelImageStorage { + private _addImageOpts: IAddImageOpts = { scrolling: true, layer: 'top', zIndex: 0, cursorPos: 'vt340' }; constructor( private readonly _storage: ImageStorage, private readonly _opts: IImageAddonOptions, @@ -27,7 +28,8 @@ export class SixelImageStorage { * Cursor behavior depends on the sixelScrolling option (DECSET 80). */ public addImage(img: HTMLCanvasElement | ImageBitmap): void { - this._storage.addImage(img, this._opts.sixelScrolling); + this._addImageOpts.scrolling = this._opts.sixelScrolling; + this._storage.addImage(img, this._addImageOpts); } /** diff --git a/addons/addon-image/src/Types.ts b/addons/addon-image/src/Types.ts index 80cec9e24..19a1acca1 100644 --- a/addons/addon-image/src/Types.ts +++ b/addons/addon-image/src/Types.ts @@ -100,6 +100,14 @@ export interface ICellSize { } export type ImageLayer = 'top' | 'bottom'; +export type CursorPos = 'vt340' | 'iip'; + +export interface IAddImageOpts { + scrolling: boolean; + layer: ImageLayer; + zIndex: number; + cursorPos: CursorPos; +} export interface IImageSpec { orig: HTMLCanvasElement | ImageBitmap | undefined; diff --git a/addons/addon-image/src/kitty/KittyImageStorage.ts b/addons/addon-image/src/kitty/KittyImageStorage.ts index 95ea594c7..1f5c09ec9 100644 --- a/addons/addon-image/src/kitty/KittyImageStorage.ts +++ b/addons/addon-image/src/kitty/KittyImageStorage.ts @@ -5,7 +5,7 @@ import { IDisposable } from '@xterm/xterm'; import { ImageStorage } from '../ImageStorage'; -import { ImageLayer } from '../Types'; +import { ImageLayer, IAddImageOpts } from '../Types'; import { IKittyImageData } from './KittyGraphicsTypes'; // Kitty-specific image storage controller. @@ -42,6 +42,7 @@ export class KittyImageStorage implements IDisposable { this._images.delete(kittyId); } }; + private _addImageOpts: IAddImageOpts = { scrolling: true, layer: 'top', zIndex: 0, cursorPos: 'iip' }; constructor( private readonly _storage: ImageStorage @@ -98,7 +99,10 @@ export class KittyImageStorage implements IDisposable { if (oldStorageId !== undefined) { this._storageIdToKittyId.delete(oldStorageId); } - const storageId = this._storage.addImage(image, scrolling, layer, zIndex); + this._addImageOpts.scrolling = scrolling; + this._addImageOpts.layer = layer; + this._addImageOpts.zIndex = zIndex; + const storageId = this._storage.addImage(image, this._addImageOpts); this._kittyIdToStorageId.set(kittyId, storageId); this._storageIdToKittyId.set(storageId, kittyId); } diff --git a/addons/addon-image/test/ImageAddon.test.ts b/addons/addon-image/test/ImageAddon.test.ts index dc729ec63..f3cde06a8 100644 --- a/addons/addon-image/test/ImageAddon.test.ts +++ b/addons/addon-image/test/ImageAddon.test.ts @@ -290,6 +290,67 @@ test.describe('ImageAddon', () => { }); }); + test.describe('IIP support', () => { + test('size params is optional/informative', async () => { + // too small size param + await ctx.proxy.write(`\x1b]1337;File=inline=1;size=5:${PALETTE_PNG_BASE64}\x07`); + deepStrictEqual(await getOrigSize(1), [640, 80]); + // no size aparm at all + await ctx.proxy.write(`\x1b[10H\x1b]1337;File=inline=1:${PALETTE_QOI_BASE64}\x07`); + deepStrictEqual(await getOrigSize(2), [640, 80]); + }); + test.skip('FilePart support - bytewise', async () => { + // opener + await ctx.proxy.write(`\x1b]1337;MultipartFile=inline=1\x07`); + // byte-wise + for (let i = 0; i < PALETTE_PNG_BASE64.length; ++i) { + await ctx.proxy.write(`\x1b]1337;FilePart=${PALETTE_PNG_BASE64[i]}\x07`); + } + // finalizer + await ctx.proxy.write(`\x1b]1337;FileEnd\x07`); + deepStrictEqual(await getOrigSize(1), [640, 80]); + }); + test('FilePart support - chunks', async () => { + // opener + await ctx.proxy.write(`\x1b]1337;MultipartFile=inline=1\x07`); + // chunks + for (let i = 0; i < PALETTE_PNG_BASE64.length; i += 256) { + await ctx.proxy.write(`\x1b]1337;FilePart=${PALETTE_PNG_BASE64.slice(i, i + 256)}\x07`); + } + // finalizer + await ctx.proxy.write(`\x1b]1337;FileEnd\x07`); + deepStrictEqual(await getOrigSize(1), [640, 80]); + }); + test('FilePart support - edgecases', async () => { + // multiple opener do not harm + for (let i = 0; i < 10; ++i) { + await ctx.proxy.write(`\x1b]1337;MultipartFile=inline=1\x07`); + } + // multiple finalizer do not harm + for (let i = 0; i < 5; ++i) { + await ctx.proxy.write(`\x1b]1337;FileEnd\x07`); + } + // chunks w'o opener are ignored + for (let i = 0; i < PALETTE_PNG_BASE64.length; i += 256) { + await ctx.proxy.write(`\x1b]1337;FilePart=${PALETTE_PNG_BASE64.slice(i, i + 256)}\x07`); + } + await ctx.proxy.write(`\x1b]1337;FileEnd\x07`); + pollFor(ctx.page, 'window.imageAddon._storage._images.size', 0); + }); + test('FilePart support - other IIP sequence cancels chunked image', async () => { + // opener + await ctx.proxy.write(`\x1b]1337;MultipartFile=inline=1\x07`); + // chunks + for (let i = 0; i < PALETTE_PNG_BASE64.length; i += 256) { + await ctx.proxy.write(`\x1b]1337;FilePart=${PALETTE_PNG_BASE64.slice(i, i + 256)}\x07`); + } + // finalizer skipped + // write spinfox.png + await ctx.proxy.write(TESTDATA_IIP[1][0]); + deepStrictEqual(await getOrigSize(1), TESTDATA_IIP[1][1]); + }); + }); + test.describe('IIP support - testimages', () => { test('palette.png', async () => { await ctx.proxy.write(TESTDATA_IIP[0][0]); diff --git a/demo/client/components/window/addonImageWindow.ts b/demo/client/components/window/addonImageWindow.ts index 392b7177b..e5402182a 100644 --- a/demo/client/components/window/addonImageWindow.ts +++ b/demo/client/components/window/addonImageWindow.ts @@ -65,7 +65,8 @@ export class AddonImageWindow extends BaseWindow implements IControlWindow { const dtIip = document.createElement('dt'); dtIip.textContent = 'IIP (iTerm)'; dlIip.appendChild(dtIip); - this._addDdWithButton(dlIip, 'image-demo3', 'palette'); + this._addDdWithButton(dlIip, 'image-demo3', 'palette (png File)'); + this._addDdWithButton(dlIip, 'image-demo4', 'dice (qoi MultipartFile)'); container.appendChild(dlIip); // Kitty demos @@ -141,6 +142,21 @@ export class AddonImageWindow extends BaseWindow implements IControlWindow { this._terminal.write(`\x1b]1337;File=inline=1;size=${data.length}:${btoa(sdata)}\x1b\\`); }); + const iipDemoMulti = (url: string) => () => fetch(url) + .then(resp => resp.arrayBuffer()) + .then(buffer => { + const data = new Uint8Array(buffer); + let sdata = ''; + for (let i = 0; i < data.length; ++i) sdata += String.fromCharCode(data[i]); + const encoded = btoa(sdata); + this._terminal.write('\r\n'); + this._terminal.write(`\x1b]1337;MultipartFile=inline=1\x1b\\`); + for (let i = 0; i < encoded.length; i += 100) { + this._terminal.write(`\x1b]1337;FilePart=${encoded.slice(i, i + 100)}\x1b\\`); + } + this._terminal.write(`\x1b]1337;FileEnd\x1b\\`); + }); + const kittyDemo = (url: string) => () => fetch(url) .then(resp => resp.arrayBuffer()) .then(buffer => { @@ -158,6 +174,8 @@ export class AddonImageWindow extends BaseWindow implements IControlWindow { sixelDemo('https://raw.githubusercontent.com/jerch/node-sixel/master/testfiles/test2.sixel')); document.getElementById('image-demo3')!.addEventListener('click', iipDemo('https://raw.githubusercontent.com/jerch/node-sixel/master/palette.png')); + document.getElementById('image-demo4')!.addEventListener('click', + iipDemoMulti('https://raw.githubusercontent.com/jerch/xterm-wasm-parts/master/fixtures/qoi/dice.qoi')); document.getElementById('image-demo-kitty1')!.addEventListener('click', kittyDemo('https://raw.githubusercontent.com/jerch/node-sixel/master/palette.png')); @@ -176,7 +194,7 @@ export class AddonImageWindow extends BaseWindow implements IControlWindow { // } // } - const pos = (this._terminal as any)._core._mouseService!.getCoords(ev, (this._terminal as any)._core.screenElement!, this._terminal.cols, this._terminal.rows); + const pos = (this._terminal as any)._core._mouseCoordsService!.getCoords(ev, (this._terminal as any)._core.screenElement!, this._terminal.cols, this._terminal.rows); const x = pos[0] - 1; const y = pos[1] - 1; const canvas = ev.shiftKey