Internationalizing Copy and Paste

[chschnell]

While working on graphical text mode I stumbled upon Copy and Paste which are still both tied to the en-US locale. It took a bit, but I think I here's how to internationalize these two functions.

Internationalizing Copy is relatively simply, but somewhere in the V86 settings the user will have to manually configure the 8-Bit codepage that is in use by the guest OS. I have published codepage tables here before.

Internationalizing Paste is a different beast though, and it is what this post is about.

I think the only proper way to implement a Paste function is by emulating key strokes on the keyboard, as is already the case in KeyboardAdapter.simulate_char() but without support for non-US keyboards. What I believe is missing is a "keyboard charset", that is the set of Unicode codepoints that can be generated by a specific keyboard (without resorting to Alt+3*Digit) and the mapping of scancodes (keystrokes) that produce them.

The website https://kbdlayout.info/ documents 217 different keyboard layouts in various formats, amongst them are KLC text files. A KLC text file is the output of Microsoft's Keyboard Layout Creator (MSKLC) encoded in Unicode, for example see the KLC code of the US-keyboard. Unfortunately, the KLC file format is not documented, but it is practically self-explaining. It's possible to reliably generate the "keyboard charset" of any keyboard given its KLC file.

So I wrote a pyhton script import_kbd.py that downloads a given set of KLC files, parses and extracts the keyboard charset, and then transforms that into a Javascript representation, see keyboard_tables.js for the resulting output (contains mappings for 16 of the 217 known keyboards).

As that turned out to be slightly dense, I wrote a little HTML page that visualizes these mappings for debugging. On the top you can select one out of 16 keyboards (could be extended to all 217 keyboards or any subset of it except for CJK), on the left you see the repertoire of all typeable Unicode symbols and their scancode-mappings, and on the right you can enter some Unicode text into the top textarea that is transformed into the full scancode-sequence (including key-press and -release scancodes as well as handling dead keys) shown in the textarea below.

See method Keyboard.text_to_scancodes(text) in keyboard_lib.js for the transformation from a Unicode text to its PC scancode sequence including modifer keys (Shift, Ctrl, Alt, AltGr), it's straight-forward.

Integrating this into V86 simply means to pump the produced scancode sequence into the V86 bus using keyboard-code events.

I included this in my V86 demo, just select Window -> Paste from the menu. Here's a screenshot of the 149 (minus a few) German keyboard codepoints that I pasted into EDIT.EXE under guest OS FreeDOS:

I've pasted whole source code files into FreeDOS, it's stable and I believe that already says a lot, yet I still have to do a lot of testing under other OSes.

Like manually configuring the active codepage for Copy, a user would have to manually configure the active keyboard for Paste.

It's noteworthy that emulating keystrokes in this way does not require any knowledge of the codepage used by the guest OS, it's a straight Unicode-to-Scancode mapping.

As keyboard layouts are defined by international standards, all OSes should treat them alike (so it shouldn't matter that I source my infos from Windows). Yet, nothing is fixed, OSes typically support user-defined keyboard layouts, so all that can be supported here is constrained to international standards.

EDIT: Updated links.

EDIT 2: Updated links again.

[chschnell]

It works under Debian, too:

debian-paste-de-charset.webm

[copy]

This looks interesting, thanks. One suggestions I would have is to compress layout information, by only specifying the keys that are different from the en-us layout.

[chschnell]

Thanks for this!

I followed a bit into the Linux keyboard handler (I assume that it overrides the BIOS' handler). I'm using Debian 12 (32 bit) for my tests.

The main file is i8042.c, it registers a handler for IRQ 1 which is the keyboard IRQ. The IRQ handler reads a single byte from port 60h (keyboard data port) and forwards it to the serio subsystem here.

So up to this point this code does exactly what is expected, but I don't know a thing about that serio subsystem, I must assume it provides a much larger buffer.

While DOS is happy with a maximum of 15 scancode bytes (a "burst" of 15 bytes) per 100ms, Linux accepts way more, but it's not unlimited.

Pasting the source code of keyboard_lib.js (16142 bytes of scancode data) with a maximum of 1024 bytes per burst works:

debian-paste-1024.webm

But it overflows at 2048 bytes per burst:

debian-paste-2048.webm

So maximum burst size depends on the OS, I had hoped it wouldn't.

By the way, the past couple of days my keyboard code in keyboard_lib.js had a bug that surfaced only on international keyboards, it's fixed.

EDIT: Updated broken links.

[SuperMaxusa]

So maximum burst size depends on the OS, I had hoped it wouldn't.

I have some thoughts that modern OSes (Linux, OpenBSD etc.) hooks keyboard IRQ themselves, while legacy OSes (MS-DOS, FreeDOS, bootOS etc) uses a BIOS functions for input and this may be the reason for the different keyboard buffer sizes.

Here I compiled SeaBIOS with COM1 debug logging for testing: seabios-com1debug.bin.gz (config).

On QEMU, you can use -bios and -serial stdio options:

qemu-system-i386 -bios seabios-com1debug.bin -serial stdio <...>

On v86, you can use xterm.js-based serial terminal (first run make build/xterm.js and load the library into your HTML file) and load debug BIOS in V86 constructor.

---

For example, I tried to send 48 chars in MS-DOS and bootOS, it only shows the first 16 chars, but in SeaBIOS' log it receives all 48 commands:

i8042_command cmd=ae

i8042_wait_write

(47x)

Same behaviour on OpenBSD's boot phase (on UKC boot> prompt), but after loading it starts using own i8042 driver, so the problem goes away.

[chschnell]

So maximum burst size depends on the OS, I had hoped it wouldn't.

I have some thoughts that modern OSes (Linux, OpenBSD etc.) hooks keyboard IRQ themselves, while legacy OSes (MS-DOS, FreeDOS, bootOS etc) uses a BIOS functions for input and this may be the reason for the different keyboard buffer sizes.

That's 100% the case.

Here I compiled SeaBIOS with COM1 debug logging for testing: seabios-com1debug.bin.gz (config).

On QEMU, you can use -bios and -serial stdio options:

qemu-system-i386 -bios seabios-com1debug.bin.gz -serial stdio <...>

On v86, you can use xterm.js-based serial terminal (first run make build/xterm.js and load the library into your HTML file) and load debug BIOS in V86 constructor.

I didn't know this, very useful!

For example, I tried to send 48 chars in MS-DOS and bootOS, it only shows the first 16 chars, but in SeaBIOS' log it receives all 48 commands:

i8042_command cmd=ae

i8042_wait_write

(47x)

Same behaviour on OpenBSD's boot phase (on UKC boot> prompt), but after loading it starts using own i8042 driver, so the problem goes away.

Yup.

I just wish the small bufferes weren't that small, bandwith is superlow because of that. Maybe two paste functions should be considered if this should get integrated: "slow paste" and "fast paste", but that seems a bit silly.

I will see if I can lower the 100ms pause between bursts a bit, but this will not change much about this.

[SuperMaxusa]

I have found that paste works correctly on MS-DOS 1.25 + Bochs BIOS (on previous and current versions of this demo), in other versions it skips a part of the keystrokes.

Just "for fun" I tried typing the Akalabeth for GWBASIC source code (https://nanochess.org/akalabeth.html), which is 29415 bytes and it tooked around 8 minutes. Unfortunately, on this BASIC in MS-DOS 1.25 the game got stuck on a level select, but this "weird" idea mostly worked.

v86.mp4

[chschnell]

I love it! The time lapse is hilarious.

Actually this brings up old memories, 35 years ago we used to type these kind of games from printouts in PC magazines into our parent's 80286 or 80386 (can't remember), pages upon pages of code. Once a month a friend and I would pool our money and buy one of these, and then we'd had to meet for days to get it done, one was reading and checking while the other was typing. At least sometimes they had per-line checksums. Makes me complain less about the "slow" ~75 keys per second that we'll have to live with.

[chschnell]

I have made some progress and have a few preliminary results.

In order to make this a bit more presentable I moved all code into a new repository, see: https://github.com/chschnell/v86-i18n, the interactive keyboard demo is available there as a github page, see: https://chschnell.github.io/v86-i18n/keyboard-tables/. I've added example code to the repository.

The code has matured, I think I cannot make the representation of codepages and keyboard layouts any compacter than it is now, see codepage_tables.js and keyboard_tables.js.

To summarize, I think this is what we learned so far:

1. when scancode data is placed into the PS/2 ringbuffer, a tight loop between V86 and the guest OS is initiated until the ringbuffer is exhausted (V86 raises INT 01h directly after the guest OS has retrieved a byte from the ringbuffer through port 60h, and the guest OS must handle the interrupt ASAP and then will read port 60h again triggering the next interrupt, and so on)
2. older PC operating systems are very constrained in their keyboard buffer (15 or 16 bytes per 100ms, unclear)
3. newer systems are less constrained (limit unclear), but V86 imposes a limit of 1024 bytes in its PS/2 ringbuffer

So unless the limit of V86's ringbuffer gets lifted we will always run into an upper bound, it just depends on the amount of pasted text.

I've also noticed that when I paste while the Firefox debugger is open (which slows V86 down considerably), scancode data will get lost.

@SuperMaxusa noticed that some DOS/BIOS combinations may even be more constrained than for example FreeDOS/SeaBIOS which I use for tests (could you add which ones you tested?).

[SuperMaxusa]

Thanks for your info! I actually used Bochs Legacy BIOS for MS-DOS 1.25 because it has faster VGA text output than SeaBIOS in my eyes. About keyboard input, there are identical. While testing in MS-DOS, I found one interesting thing: keyboard input corruption can even depends on which program captures the keystrokes.

According to your previous tests, text input loses characters in FreeDOS EDIT, however, if VIm is used instead (you can click Floppy -> Clone -> FreeDOS 1.3 720K Copy in the demo and run a:vim), the text paste works fine.

Here is are some tests for this behavior, the original text is the first sentences of this post (literally):

As stated upper, in FreeDOS EDIT some characters are not sent, but some words can be recognized.

freedos-edit.webm

In VIm (FreeDOS) and MS-DOS Editor works properly.

demo

freedos-vim.webm

msdos-edit.webm

(I split the text into 80 chars lines for MS-DOS Editor because it doesn't have "Text Wrap" option)

About BASIC, version 1.12 from MS-DOS 1.25 works properly, but GW-BASIC 2.02 from MS-DOS 2.11 skips part of text.

demo

1.25:

msdos125-basic.webm

2.11:

msdos211-basic.webm

Last test is Windows 1.01 that very strangely leave only 42 characters and I don't know why, whereas on Windows 3.1 it worked properly.

windows101-notepad.webm

[chschnell]

Thanks for the extensive tests! Great effort, lots of new infos.

I'll repeat your tests and see what happens when pausing after each scancode (what I did initially), which is the slowest method to inject scancodes but of course also the most reliable one.