Sharing Windows OpenSSH keys for Linux Dual Boot

TL/DR: If you run into problems with opensshd permissions on windows, open a PowerShell Administrator prompt and run:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Previously I wrote about Installing OpenSSH on Windows. For my workflow, I actually prefer to dual-boot Linux and Windows even though WSL2 has come a long way.

I use Barrier (open source successor to synergy) to share my mouse (well trackball) and keyboard across my workstation and laptop, regardless of whether Linux or Windows is running – I securely share the same underlying keys, and have the dhcp server assign a fixed IP to each MAC address.

It’s actually quite tricky to get your OpenSSH keys from Linux’s /etc/ssh/ssh_host_*key to C:\ProgramData\ssh\ssh_host_*key because of ACL details, even though I only edited the files with nvim – I thought that should preserve the icacls status, but it doesn’t.

Windows iacls are a bit like selinux or AppArmor. Not a trivial subject, so be prepared if you’re going to wade in.

iacls have inheritance, removed with /inheritance:r

For me, the most confusing thing about icacls is that if you break the permissions in certain ways (for example removing inheritance before you’ve granted some individual permissions to that file), you can no longer use icacls to fix them! You have to use takeown to re-assert ownership, and then you can start using icacls again.

PS C:\ProgramData\ssh> net stop sshd
The OpenSSH SSH Server service was stopped successfully.

PS C:\ProgramData\ssh> net start sshd
The OpenSSH SSH Server service is starting.
The OpenSSH SSH Server service could not be started.

A system error has occurred.
System error 1067 has occurred.
The process terminated unexpectedly.

Because I am not a master of icacls, I completely hosed my entire C:\ProgramData permissions while trying to fix ssh

When trying to run sshd directly from the command line rather than via the windows service infrastructure, I actually got a bit more detail.

PS C:\WINDOWS\system32> sshd -dd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 158
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 158
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ecdsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

The PowerShell team provides a guide for exactly what ACL permissions are required to for your ssh_host_* files.

After several rounds of shooting myself in the foot with the not very memorable friendliness of icacls, I finally ran:

icacls "C:\ProgramData\ssh" /setowner system
icacls "C:\ProgramData\ssh" /q /c /t /reset
icacls "C:\ProgramData\ssh\ssh_host_*" /remove erwin

After that, I ran sshd -dd and finally was able to get OpenSSH to start up again on the command line without permissions errors, however running net start sshd still was failing to startup…

Turns out that running sshd -dd as just runs sshd in interactive mode under the currently logged on user (typically as an admin). To simulate the SYSTEM actually running sshd as a service you actually want to run:

psexec -s sshd.exe -ddd

(Note, psexec is part of sysinternals, probably easiest to install with shovel…)

PS C:\WINDOWS\system32> psexec -s sshd.exe -dd

PsExec v2.34 - Execute processes remotely
Copyright (C) 2001-2021 Mark Russinovich
Sysinternals -

debug2: load_server_config: filename PROGRAMDATA\ssh/sshd_config
debug2: load_server_config: done config len = 158
debug2: parse_server_config: config PROGRAMDATA\ssh/sshd_config len 158
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
Permissions for 'PROGRAMDATA\ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_rsa_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_rsa_key
Permissions for 'PROGRAMDATA\ssh/ssh_host_ecdsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ecdsa_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ecdsa_key
Permissions for 'PROGRAMDATA\ssh/ssh_host_ed25519_key' are too open.

It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ed25519_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.
sshd.exe exited on XPS with error code 1.

So even though I had fixed the permissions enough for my user to run sshd it was not enough for the system to run sshd, the way that net start sshd works.

I wasn’t even able to cd C:\ProgramData\ssh so I started with:

get-acl C: | set-acl C:\ProgramData

Then when I cd into C:\ProgramData\ssh turns out that the permissions are in fact way more open than what Window’s SSHD (or Linux sshd for that matter) permit.

PS C:\ProgramData\ssh> icacls ssh_host_*key
ssh_host_dsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)

ssh_host_ecdsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)

ssh_host_ed25519_key NT AUTHORITY\Authenticated Users:(I)(M)
                     NT AUTHORITY\SYSTEM:(I)(F)

ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)

Successfully processed 4 files; Failed processing 0 files

So the easy way to do this on Windows is just to focus on one file at a time… We know that sshd complained about ssh_host_rsa_key first, so we’ll start there.

icacls .\ssh_host_rsa_key /inheritance:r

So this removed the inheritance ACL from that file, and it’s basically impossible to re-add…

Windows improbable answer for removing inheritance from a single file is to use takeown

takeown /R /F C:\ProgramData\ssh

# Then reset the ACLs to their default values

icacls C:\ProgramData\ssh /T /Q /C /RESET

After takeown runs, you’ll be able to fix all the permissions again, but all the permissions will be messed up requiring them to be fixed 😉

Now we’ll try again:

icacls.exe .\ssh_host_rsa_key

.\ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)

Now I’m first going to explicitly grant Full Control to System.

icacls.exe .\ssh_host_rsa_key /grant SYSTEM:`(F`)
icacls.exe .\ssh_host_rsa_key /grant Administrators:`(F`)

Then I’m going to remove Inheritance

icacls.exe .\ssh_host_rsa_key /inheritance:r

However, unfortunately this still doesn’t work… Even though this gives us the exact values that the PowerShell team documents as required:

ssh_host_rsa_key BUILTIN\Administrators:(F)

When starting up sshd with psexec we’ll still get the error:

Permissions for '__PROGRAMDATA__\\ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "__PROGRAMDATA__\\ssh/ssh_host_rsa_key": bad permissions
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key

After far too long of a detour today, I finally solved it with:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Unlike Linux, the parent directory permissions don’t seem to matter.

Now, net start sshd works perfectly 🙂


Docker for Rails Developers in 2022

The pragprog team has a great book called “Docker for Rails Developers” however it’s about 3 year old now… So just some minor notes here on how to make everything work right in Ruby 3 / Rails 7 world.

Docker is primary designed for deploying finished images of your software to production. Docker can work great in development, but just keep in mind that it’s not really designed for development, so you’ll always have to jump through a few extra hoops in exchange. Things like sharing files as your building an image and caching to speed up development builds are extra complicated. If you’ve ever setup a chroot environment, it’s not as simply as just including the stuff you directly want to use — there’s overhead.

As I’m writing, Rails v7.0.3 with Ruby 3.1.2 is current.
Yet this book is based on Rails v5.2.2 and Ruby 2.6.

Docker makes these kind of version changes super easy. I recommend just sticking with exactly what the book uses and explicitly setting:

gem install rails -v 5.2.2

If you keep rails on v5.2.2 then all of the rest of the instructions should work, and when you start the rails server with:

docker run -p 3000:3000 ${my-image-id} bin/rails s -b

You should get the Rails 5 welcome page…

Your other also very easy option is to just switch to a Ruby 3.1.2 based image… So in your Dockerfile just use:

From: ruby:3.1.2
# instead of From: ruby:2.6

The examples are so simple from a Ruby perspective that you should be fine either way. Always comforting when your screenshots exactly match the authors though, so I would stick with Rails 5.2.2 and Ruby 2.6 and work though this short book.

Highly recommend this one though. Changed the way I think about Docker, especially all the docker-compose stuff.

Once you get to Chapter 7, Playing Nice with JavaScript, the Ruby 2.6 instructions really start to break down. Node 8 and Node 10 are deprecated, so you’ll need to move to Node 12, and the default packages for that aren’t compatible with the Debian base image that Ruby 2.6 is built on.

With Rails 7, we’ve kind of moved on from WebPacker anyway, but if you really want to do that part, I got it to build fine by using this as my Dockerfile.Basically, set ruby to 3.1.2, use setup_12.x for Node JS AND make sure to generate your rails app with Rails 7.

❯ cat Dockerfile        
FROM ruby:3.1.2

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \

RUN curl -sL | bash -

RUN curl -sS | apt-key add -
RUN echo "deb stable main" | \
    tee /etc/apt/sources.list.d/yarn.list

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \
    apt-utils \
    nodejs \
    yarn \

COPY . /usr/src/app

WORKDIR /usr/src/app
RUN bundle install

Webpacker comes up again in Advanced Gem Management and finally in Chapter 13, A Production-Like Playground.

The Rspec/testing chapter doesn’t use Node or webpacker, so you can tear the yarn and nodejs update stuff out of your Dockerfile and get rid of Webpacker in your Gemfile.

FROM ruby:2.6

LABEL maintainer=""
LABEL foo="bar"

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \
    apt-utils \
    nodejs \

COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN bundle install

COPY . /usr/src/app/

CMD ["bin/rails", "server", "-b", ""]

The automated testing with Capybara/Selenium didn’t work for me, but I’ve done some browser automation with Goland and Chrome so I didn’t want to spend a lot of time debugging it. If you want to do the RSpec/Capybara test examples in 2022, probably best to reference this post instead:

The authors tutorial approach of working though each step a piece at a time was extremely helpful for me, even being fairly familiar with Docker, but passively as a developer/user without having been excited enough to deep dive into Docker itself.


How to completely remove Snaps? Install voidlinux

Ubuntu Linux is on a dark path these days. I used Debian on a lot of servers in the early 2000s, and like RedHat Enterprise, the Debian Stable stuff is just too old… So having a team “stabilize” Debian’s Unstable distribution every six months was a fantastic idea and it worked well for a long time.

Unfortunately there is a big disconnect between who the Ubuntu team wants to serve and it’s regular users. Sadly, Wayland seems to have this same disconnect, but that’s another rant.

Snaps are just so awful. They break your themes, break your plugins and thus your workflow, block your access to files, clutter up your process list and mount list. Containerization is a great idea AS LONG AS you don’t reduce function… So far, containerization on the desktop is not working. I would much rather use a heaver QubesOS like model rather than kind of working but breaking tons of advanced use cases.

voidlinux is an awesome alternative to Ubuntu:

  • Gets rid of Systemd, using very simple runit scripts for services
  • Void is not a FORK of anything. It’s a new Linux developed from scratch.
  • It’s a “Stable rolling release” distro, while Arch just focuses on Bleeding Edge rolling releases (that occasionally blow up in my face)
  • Easily install any Debian package with xdeb
    • I used it for Zoom, Opera and Jumpapp
  • Enjoy the super simple xbps package system
    • sudo xbps-install -Svu to update the cache the first time
    • sudo xbps-install etckeeper to install etckeeper
    • sudo xbps-install xtools to install xlocate
  • xlocate is a very fast replacement for apt-file search, just type xlocate '/xlocate$' it accepts any regex and will help you find any file inside of any xbps package, even the ones you haven’t installed.
  • void is a minimal distro without any bloat

To get the install started, you just download the voidlinux ISO, which will startup a voidlinux live image. Open the terminal and run sudo void-installer – the password is “voidlinux“.

That’s basically it. Once you’re booted into your own voidlinux install, install xtools and run xlocate to find whatever you’re missing. Here’s my list of daily use tools if you need something to help get started.

# Refresh the XBPS-Cache
sudo xbps-install -Svu

# Always start with etckeeper
sudo xbps-install etckeeper

# Install generic utilities
sudo xbps-install htop neofetch python3 sqlite kitty kitty-terminfo 
sudo xbps-install ddcutil mlocate ntp
sudo xbps-install tree ranger tmux screen direnv mosh fzf fzy
sudo xbps-install bind-utils git curl wget whois netcat socat
sudo xbps-install flameshot zsh xtools syncthing cheat sxhkd
sudo xbps-install rofi aws-cli zathura tdrop mpg123 mp3info 
sudo xbps-install xsel xmodmap xev xrandr wmctrl jump
sudo xbps-install binutils tar curl xbps xz bash-completion
sudo xbps-install base-devel

# KDE Plasma Desktop
sudo xbps-install plasma-desktop
sudo xbps-install kde5-baseapps kde-cli-tools qt5-tools
sudo xbps-install kdegraphics-thumbnailers ffmpegthumbs kwallet-pam
sudo xbps-install ksshaskpass kwalletmanager 
sudo xbps-install pavucontrol-qt pavucontrol


Adding a `hyper` modifier key to your keyboard, because shift/ctrl/alt/meta/super aren’t enough

TL/DR: Try this in your ~/.Xmodmap log out of X11 and log back in. If you moved to Wayland, hopefully one day we’ll get stable APIs for 3rd party window utilities. Until that happens, I recommend switching back to X11 if you can.

clear mod1
clear mod2
clear mod3
clear mod4
clear mod5

keycode 248 = Hyper_L

add Mod1 = Alt_L Alt_R
add Mod2 = Num_Lock
add Mod3 = Hyper_L
add mod4 = Super_L Super_R
add mod5 = ISO_Level3_Shift Mode_switch

Note that you can’t blindly copy code 248… You need to find a key that will work for your keyboard.

Soft fn key on keyboard

Use xev -event keyboard to find the keycode for the key that you want to use. While xev is running, focus on the xev window and press the key you want to use. You’ll see the KeyRelease event show on your console. Best to use the xev -event keyboard filter so that all mouse events can be ignored.

In my example, this is AFTER I’ve use Xmodmap to map this key to Hyper_L… You won’t see this until your remapping is successful.

You need to make sure Xmodmap will actually start with your widow manager, and for me, it works best with a delay. So I have a file ~/.config/autostart/xmodmap.desktop

[Desktop Entry]
Comment=Xmodmap Keyboard Modification

And then delay_xmodmap is a trivial script that just sleeps for a bit before launching.


sleep 10;
xmodmap ~/.Xmodmap

Using a hot key manager like sxhkd you should be able to map shortcuts to the Hyper key now. Put something like this in your ~/.config/sxhkd/sxhkdrc to test that Hyper is working.

hyper + {a-z}
        notify-send "hyper {a-z}"

Every keyboard I’ve ever seen has a SHIFT key and CONTROL keys. Historical Unix vs Mac vs Windows are a bit different for the other keys:
Alt == Meta (Option in the Mac World)
Windows Key == Super Key == Command Key

So you basically get 3 modifiers on each key, but SHIFT is so common for text it’s dangerous for most of us to configure that.

If you’re on Linux, and you have a keyboard that generates an actual “keycode” (some fn labeled keys actually run software in the keyboard to send a different keycode to the OS, without the OS knowing that a modifier was involved).

mod1AltOptionAlt_L Alt_R / Meta
mod2Num LockNum Lock
mod4Windows KeyCommand KeySuper_L Super_R
X11 mod1 keys


Git Self Hosted: GitLab vs Gitea vs Gogs in 2022

Github is a double edged sword. Use it when you must, but you should self-host whenever possible.

I started with gitlab as it was the most popular github alternative at the time, and I think for medium-large businesses hosting their own local, dedicated git instance, gitlab is probably a good choice.

For me though, gitlab wad definitely overkill. The upgrades get very complicated, and I’m ultimately just not a big Ruby on Rails fan… I just wanted locally hosted git that I could also browse easily via http, especially to graphically view older commits and branches, because every once in a while I do leave the terminal…

After gitlab my next solution was Gitea, as I read it was a community fork of gogs so I assumed it must be better… In hindsight, I would say Gitea is working hard to compete with GitLab. That’s fine, but `GitLab` is actually what I wanted to get away from.

You can find a Gitea feature list here:

Here’s an old sysadmin rule of thumb: If it involves LDAP, run the other way!

Me, screaming… every time I do something with LDAP…

In all fairness, LDAP is necessary for directory based authorization and authentication. With lots of users, you’re kind of stuck with it and so you better know how it works. Everything connected to X.509 is a whole different mindset than the rest of the Linux and Open Source movement. If software had an attitude, ldap software and the zshell would be at opposite edges of the attitude spectrum.

I spent a long time actually getting gitea to work in my particular configuration, and just ended up dedicating way more time to it than I really wanted to. For me, git isn’t very much different from samba… I just want it to work so I can focus on problems that I’m more interested in.

I would recommend gitea to anyone looking at gitlab and vice versa. These two are very similar. I think that blindly running either of them in a container just because you can is asking for trouble though. Go through the manual instillation and know how to set things up from scratch. If you can’t do that, you shouldn’t run it, because you won’t be able to fix it when things go wrong. You want a sysadmin that knows how to set these up and how to manage them, back them up, and fix problems along the way.

So I spent far more hours than I’m proud to admit trying to get gitea working in my particular environment, but when I decided to switch over to gogs, I had it all working and my old repos migrated over within 30 minutes.

If you’re just looking for a simple, self hosted git server via `https` in 2022, I think that gogs is what you are looking for.

Another nice “self hosted” option is cgit but the default interface is very dated, and it’s even more minimal than my ideal feature set.

If neither gitlab, gitea, gogs or cgit are what you’re looking for, maybe check out the awesome self hosted list, but a lot of these are very niche and not well supported.


Configuring OpenSSH-Server (sshd) on Windows 11

OpenSSH Server on Windows is very cool, and very weird…

You need to open a PowerShell Prompt as Administrator

If you’re going to connect to Windows via OpenSSH, I think you’re really best off using PowerShell rather than cmd.exe, as PowerShell can do quite a bit more from the command line that’s impossible with cmd.exe.

# Install the OpenSSH Server
Add-WindowsCapability -Online -Name OpenSSH.Server

# Start the SSH server
Start-Service sshd
# or the even shorter and older...
net start sshd

32-bit versions of windows shipped with a ported version of edit.exe from DOS that would work in a command prompt. The 64-bit editor that works in every version of windows and is guaranteed to be installed is notepad.exe. Obviously that’s not going to get us very far via ssh so we need a way to edit in the terminal.

The best solution is to install neovim via scoop. If you’re not into neovim, regular vim or nano (doesn’t use vi keys) would also work.

# Allow PowerShell to run RemoteSigned code
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Install scoop
iwr -useb | iex

# Install git - required for scoop to operate
scoop install git

# Install neovim
scoop install neovim
# or `scoop install vim` if Lua is too fast for you
# or `scoop install nano` if hklj are too powerful...

# Install the VC++ runtime (recommended for neovim)
scoop install vcredist2015

Most regular *nix machines put the sshd configuration in /etc/ssh/sshd_config. For Windows, the /etc/ssh directory is:
C:\ProgramData\ssh. Inside of this directory you’ll see:

Directory: C:\ProgramData\ssh


At the end of the Windows sshd_config you’ll see:

Match Group administrators
       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

If you’re in a multi-user corporate environment I would strongly recommend changing this to:

AuthorizedKeysFile __PROGRAMDATA__/ssh/%u/authorized_keys

The point is that in a multi-admin environment, you’ll realistically want every admin to at least have their own home folder, and not effectively be sharing a login… Using Microsoft’s default, the username is set by user set in the ssh user@server command.

If you use any path under ProgramData for your keys, you’ll need to icalcls to set exactly these permissions, of course update your file name accordingly.

icacls.exe "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

In my case, I’m just running this as a virtual machine for development, so I just commented out the #match group Administrators Authorized Key File which simplifies permissions and behaves more like you expect of ssh. In a real server environment, I would definitely go with: PROGRAMDATA__/ssh/%u/authorized_keys

#Match Group administrators
#       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

Last, you need to add your ~/.ssh/ public key to the authorized_keys file that you’ve chosen, and then restart sshd.

net stop sshd
net start sshd

Now you should be able to log in via:

ssh myuser@windows-host -p 22 -i ~/.ssh/some_key

Once you’ve got SSH connected, I would recommend setting ssh to launch powershell as the default shell rather than cmd, so that you can edit the registry via SSH.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

# Check that it's set with:
Get-Item -Path "HKLM:\SOFTWARE\OpenSSH"

You can list the current version of windows that’s being used via cmd.exe’s ‘ver’ shell builtin.

cmd.exe /c ver

# Windows "11" is:
Microsoft Windows [Version 10.0.22000.318]

Windows 11 ships with WSLg allowing display of GUI apps, supporting both Wayland and X11 APIs…

wsl --install -d Ubuntu


Getting CTRL+TAB to work in neovim

If you’ve been using a terminal for a while, you’ve surely stumbled across the terminal’s legacy keyboard handling. Terminal escape codes were created in the 1970s and haven’t been updated much.

CTRL + J == newline
CTRL + I == tab

These two examples are some of the least annoying, but the implications aren’t at first obvious. Normal terminal apps can’t tell whether you actually typed “ENTER” on your keyboard or “CTRL” + “I”. From the perspective of the app, it’s the same. Sadly, that’s just the tip of the iceberg for these limitations.

CTRL + … Sequences starting with CTRL+… are basically used up by the way the terminal represents common data (CTRL + D == EOF) with “CTRL CODES”, and CTRL+SHIFT+letter is the same as CTRL+letter

ALT + C might be encoded as 0xc3, which collides with UTF-8 for characters like: é.

SHIFT + … The terminal doesn’t know the difference between “Shift + s” and “S”…

Hyper +… Hardly any apps are cool enough to be able to work with the awesome hyper modifier… Without an enhanced way to process keycodes, this isn’t possible.

Many terminals rely on timing to distinguish codes like Alt+C from Esc + C… This generates lag and errors when typed too slow or too fast. vim-fixkey has a nice list of workarounds to get full access to keypresses in the terminal, though even in the best case scenario, it would be limited by a timeout, and not work for Cmd, Hyper or various CTRL+x keys…

Fortunately Leonerd, the author of libtermkey (now replaced by libtickit), wrote the “fixterms” specification and the author of Kitty has extended the fixterms spec to cover Super, Hyper and other modifier keys. iTerm 2 is also being extended to cover the new Kitty version of the fixterms spec.

Some of this is “worked around” with the XTerm ModifyOtherKeys option that’s also supported in gnome-terminal and Konsole. Unfortunately ModifyOtherKeys is not complete.

There is discussion on the NeoVim github issues about adding fixterms support in neovim, but at this time (Nov 2021) it seems that no support is yet complete. It looks like neovim/src/nvim/keymap.c probably holds most of the code that needs to change.

Fix ambigous terminal key strokes?

Someone may have worked on an actual code change for this, tracking at:

TUI: enable/disable modifyOtherKeys automatically
TUI: distinguish Tab, CTRL-i (S8C1T mode)
CTRL-Alt-Space isn’t recognized even though terminal sends ^[^@, works in Vim

Binding <M-S-Tab>

TUI: S8C1T (8-bit) mode, v:termresponse

See: “xterm-8-bit” Nvim does not use 8-bit sequence detection, and always uses 7-bit sequences (for now)

One day, hopefully Neovim will support these sequences by default, but in the mean time it’s possible to map these sequences manually in Kitty and Neovim, it’s possible to manually use a specific mapping by configuring Kitty’s map ... send_text

# In Kitty.conf Example
map ctrl+enter send_text normal,application \x1b[13;5u

# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-cr>', ':echom "Hello C + R"<CR>')

Remember, the Kitty chart for progressive enhancement is:

0b1 (1)Disambiguate escape codes
0b10 (2)Report event types
0b100 (4)Report alternate keys
0b1000 (8)Report all keys as escape codes
0b10000 (16)Report associated text

So \x1b = 11011… So it will turn on all options except “report alternate keys”.

Decimal 13 == Carriage Return

; 5u == modifier flags == “Ctrl” + 1.

shift     0b1         (1)
alt       0b10        (2)
ctrl      0b100       (4)
super     0b1000      (8)
hyper     0b10000     (16)
meta      0b100000    (32)
caps_lock 0b1000000   (64)
num_lock  0b10000000  (128)

Another example…

# In Kitty.conf Example
map ctrl+tab     send_text normal,application \x1b[9;5u
map ctrl+shift+tab send_text normal,application \x1b[9;6u

# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-tab>', ':echom "Hello Tab"<CR>')
vimp.bind('n', '<C-S-tab>', ':echom "Hello Shift Tab"<CR>')

In this example, the relevant parts are:

\x 1b [ 9 ;5u
\x 1b [ 9 ;6u
   ^ 1b = 11011binary == turn on Disambiguate Esc Codes, Report Event Types, Report All Keys as Escaped, Report Associated Text
      [ is just the end of our escape code
         9 is the keycode for Tab
            5u = 1 (constant) + 4 (0100) Control
            6u = 1 (cont) + 5
                   4 = ctrl + 1 shift = 5

Let’s go through a few more examples. Let’s try: CTRL + Super + Tab. Unfortunately, even though Kitty can send this one, until Neovim’s keyboard support is more complete, it doesn’t look like Neovim has any way to receive this keycode.

CTRL     = 4
SUPER    = 8
Constant = 1
Total    =13 
# Remember, Tab = 0x9
\x 1b [ 9; 13u

Sadly, I can’t get Super or Hyper based shortcut keys to work in Neovim or regular Vim yet, but I think it will be here soon…


kitty and Nerd Fonts… bringing Egyptian Hieroglyphs to your terminal…

TL/DR: Just download Nerd Symbols Only Font and Config Kitty

If you haven’t tried the kitty terminal emulator, it’s an awesome one. Even if I was still tortured to use Mac OS X, I think I would use Kitty of iTerm. You’ll find that it’s noticeably faster than gnome-terminal or vscode‘s built in terminal. (konsole is pretty snappy though)

Kitty has tons of features, however the process to discover and configure those features is a bit more complicated than other terminals that I’ve used.

For me, the most confusing aspect of kitty was understanding how to setup fonts. I never wanted to learn how kitty deals with fonts. I just wanted a kitty font config that works. If that’s all you want, skip to kitty config.

First, Kitty requires that fonts have a fixed spacing of 100 defined. To check if a font can be used by Kitty, and the exact name you should enter to configure it, you’ll want to use:

kitty list-fonts

If you’re looking for a good font to use, I suggest you take a look at the open source Fira Code font, since it has nice ligatures for programming…

Fira Code example “ligatures” o visually simplify display of == –> ++ and :=

If you you have a budget to invest in a programming font, I think the MonoLisa font is even more attractive. To get a monospaced font to show up in Kitty if it’s not showing up on it’s own, you’ll need to edit ~/.config/fontconfig/fonts.conf and add a section to “scan” for the font “family” and set “spacing” for that family to 100.

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>

<match target="scan">
    <test name="family">
    <edit name="spacing">


Once you edit fontconfig/fonts.conf you’ll need to rebuild the font cache and restart kitty to see the new font.

sudo fc-cache -fr

If kitty list-fonts still can’t find the font that you’re looking for, several steps possible steps to resolve are discussed on the kitty issue tracker.

Most terminals need you to setup “patched fonts” if you want to use “powerline” symbols… If you’re looking to rice your command line, powerlevel10k, vim-devicons, and vim-airline can will use many of these “patched font” symbols.

vim-airline using patched fonts…
Powerlevel10k Extravagant Style
powerlevel10k zsh theme using patched fonts
vim-devicons overall screenshot
vim-devicons with patched fonts

If you want to use the “fontconfig” method for your whole configuration, you might be able to add something like the following ~/.config/fontconfig/fonts.conf so that symbols from the Symbols Nerd Font will be preferred, but I was not able to get this method to work, so I went with the kitty symbol_map method.

   <family>Symbols Nerd Font</family>
   <family>Noto Color Emoji</family>
   <family>Noto Sans Symbols2</family>

Yes, you list all of them them [symbol_map] and yes you can have multiple ones. However, I recommend against doing that, there is rarely a need for it, since nerd fonts will automatically be used if found in most cases.

kovidgoyal, author of kitty

For me though, “auto detection” of nerd fonts never worked. The most evident example was each time I would run:

vim ~/.vimrc

The stylized “V” from vim-devicons would show a very rarely used Chinese character…

Editing .vimrc, the stylized V is shown for me as: 金 + 弦, which doesn’t exist in most chinese dictionaries
Editing .vimrc, the correct stylized “V” icon from vim-devicons

The “quick and dirty” way to get every symbol you want to load is the use the patched Nerd fonts, for example the patched Nerd Fonts.

However, with Kitty, the preferred way to get each font to show up is actually NOT to use patched fonts, instead to use kitty’s “symbol_map” function.

Map the specified unicode codepoints to a particular font. Useful if you need special rendering for some symbols, such as for Powerline. Avoids the need for patched fonts. Each unicode code point is specified in the form U+<code point in hexadecimal>. You can specify multiple code points, separated by commas and ranges separated by hyphens. symbol_map itself can be specified multiple times. Syntax is:

symbol_map U+E0A0-U+E0A3,U+E0C0-U+E0C7 PowerlineSymbols
symbol_map codepoints Font Family Name

If you run kitty with, you’ll get a list of each character code that’s not found on the terminal you launched kitty from.

kitty --debug-font-fallback

For example, on the Right I’m using xterm to edit ~/.config/kitty/kitty.conf and on the left is my kitty instance… Here you can see pretty clearly the “DejaVu Sans Mono” font is providing the U+2605 (★) and U+21b6 (↶)

I would prefer to get all of the Nerd Fonts from the Nerd Fonts package.

First, make sure you download the “Symbols Nerd Font” (symbols only) font package, you can find it at:

src/glyphs/Symbols-2048-em Nerd Font Complete.ttf

Once you have the Symbols-2048-em Nerd Font Complete.ttf, copy it to ~/.local/share/fonts and then run sudo fc-cache -fr to reload the font cache.

Next, you’ll want to add something like this to your kitty config:

font_family FiraCode
# or #
font_family MonoLisa

# Symbols Nerd Font complete symbol_map
# easily troubleshoot missing/incorrect characters with:
#   kitty --debug-font-fallback

# "Nerd Fonts - Pomicons"
symbol_map  U+E000-U+E00D Symbols Nerd Font

# "Nerd Fonts - Powerline"
symbol_map U+e0a0-U+e0a2,U+e0b0-U+e0b3 Symbols Nerd Font

# "Nerd Fonts - Powerline Extra"
symbol_map U+e0a3-U+e0a3,U+e0b4-U+e0c8,U+e0cc-U+e0d2,U+e0d4-U+e0d4 Symbols Nerd Font

# "Nerd Fonts - Symbols original"
symbol_map U+e5fa-U+e62b Symbols Nerd Font

# "Nerd Fonts - Devicons"
symbol_map U+e700-U+e7c5 Symbols Nerd Font

# "Nerd Fonts - Font awesome"
symbol_map U+f000-U+f2e0 Symbols Nerd Font

# "Nerd Fonts - Font awesome extension"
symbol_map U+e200-U+e2a9 Symbols Nerd Font

# "Nerd Fonts - Octicons"
symbol_map U+f400-U+f4a8,U+2665-U+2665,U+26A1-U+26A1,U+f27c-U+f27c Symbols Nerd Font

# "Nerd Fonts - Font Linux"
symbol_map U+F300-U+F313 Symbols Nerd Font

#  Nerd Fonts - Font Power Symbols"
symbol_map U+23fb-U+23fe,U+2b58-U+2b58 Symbols Nerd Font

#  "Nerd Fonts - Material Design Icons"
symbol_map U+f500-U+fd46 Symbols Nerd Font

# "Nerd Fonts - Weather Icons"
symbol_map U+e300-U+e3eb Symbols Nerd Font

# Misc Code Point Fixes
symbol_map U+21B5,U+25B8,U+2605,U+2630,U+2632,U+2714,U+E0A3,U+E615,U+E62B Symbols Nerd Font

The list of code point ranges above comes directly from the nerd-fonts test script, with the exception of the “Misc Code Point Fixes” that I added myself through trial and error with --debug-font-fallback.

You can test that you’ve got all fonts configured properly by running the “” script provided by nerd-fonts.

Be sure to run kitty with kitty --debug-font-fallback then run this script in your kitty window, and if you’ve set everything up correctly, you should not see any of the symbol missing or symbol fallback notices that were shown in the screenshot above. Once properly configured you’ll get:

Note that there may still be a few messages, but generally all characters will now load correctly.
Small excerpt of the “” script once kitty is properly configured

If you’re experimenting with MonoLisa, here are some settings that I found looked great.

font_family      MonoLisa
font_bold        MonoLisa-Bold
italic_font      MonoLisa-Light
adjust_baseline -4
font_size        14

Without the “adjust_baseline -4”, the MonoLisa font was not aligned when I tried.

One last nugget of wisdom here, the “” site is the best unicode character lookup site I’ve ever seen. You can type the codepoint directly in the URL, and the layout is clean and free of ads.

Did you know there are Unicode codepoints for Egyptian Hieroglyphs?
Now you can put heiroglyphs in your terminal!”

Egyptian Hieroglyphs in markdown document displayed via kitty terminal
# Egyptian Hieroglyphs are Back!

  𓀀 𓀁 𓀂 𓀃 𓀄 𓀅 𓀆 𓀇 𓀈	𓀉 𓀊 𓀋 𓀌 𓀍 𓀎 𓀏 𓀐 𓀑
    𓀒 𓀓 𓀔 𓀕 𓀖 𓀗 𓀘 𓀙 𓀚	𓀛 𓀜 𓀝 𓀞 𓀟 𓀠 𓀡 𓀢 𓀣
    𓀤 𓀥 𓀦 𓀧 𓀨 𓀩 𓀪 𓀫 𓀬	𓀭 𓀮 𓀯 𓀰 𓀱 𓀲 𓀳 𓀴 𓀵
    𓀶 𓀷 𓀸 𓀹 𓀺 𓀻 𓀼 𓀽 𓀾	𓀿 𓁀 𓁁 𓁂 𓁃 𓁄 𓁅 𓁆 𓁇
    𓁈 𓁉 𓁊 𓁋 𓁌 𓁍 𓁎 𓁏 𓁐	𓁑 𓁒 𓁓 𓁔 𓁕 𓁖 𓁗 𓁘 𓁙
    𓁚 𓁛 𓁜 𓁝 𓁞 𓁟 𓁠 𓁡 𓁢	𓁣 𓁤 𓁥 𓁦 𓁧 𓁨 𓁩 𓁪 𓁫
    𓁬 𓁭 𓁮 𓁯 𓁰 𓁱 𓁲 𓁳 𓁴	𓁵 𓁶 𓁷 𓁸 𓁹 𓁺 𓁻 𓁼 𓁽
    𓁾 𓁿 𓂀 𓂁 𓂂 𓂃 𓂄 𓂅 𓂆	𓂇 𓂈 𓂉 𓂊 𓂋 𓂌 𓂍 𓂎 𓂏
    𓂐 𓂑 𓂒 𓂓 𓂔 𓂕 𓂖 𓂗 𓂘	𓂙 𓂚 𓂛 𓂜 𓂝 𓂞 𓂟 𓂠 𓂡
    𓂢 𓂣 𓂤 𓂥 𓂦 𓂧 𓂨 𓂩 𓂪	𓂫 𓂬 𓂭 𓂮 𓂯 𓂰 𓂱 𓂲 𓂳
    𓂴 𓂵 𓂶 𓂷 𓂸 𓂹 𓂺 𓂻 𓂼	𓂽 𓂾 𓂿 𓃀 𓃁 𓃂 𓃃 𓃄 𓃅
    𓃆 𓃇 𓃈 𓃉 𓃊 𓃋 𓃌 𓃍 𓃎	𓃏 𓃐 𓃑 𓃒 𓃓 𓃔 𓃕 𓃖 𓃗
    𓃘 𓃙 𓃚 𓃛 𓃜 𓃝 𓃞 𓃟 𓃠	𓃡 𓃢 𓃣 𓃤 𓃥 𓃦 𓃧 𓃨 𓃩
    𓃪 𓃫 𓃬 𓃭 𓃮 𓃯 𓃰 𓃱 𓃲	𓃳 𓃴 𓃵 𓃶 𓃷 𓃸 𓃹 𓃺 𓃻
    𓃼 𓃽 𓃾 𓃿

Customizing Firefox shortcut keys… Get rid of CTRL + SHIFT + C devtools inspector shortcut

Firefox used to have some plugins that you could use to customize your command keys pretty easily, but as Firefox has evolved recently, many of the XUL generation of shortcut keys has been disabled.

It’s very easy to rebuild Firefox to support the keys that you want though. First, you’ll want to run once to build the initial “” file.

On Linux, the relative file path is:


Open, you’ll find it in:
Then you’ll want to change the default C shortcut key that conflicts with my terminal muscle memory of CTRL+C to interrupt, and Shift + Ctrl + C as Copy…

# LOCALIZATION NOTE (inspector.commandkey):
# Key pressed to open a toolbox with the inspector panel selected

The full workflow is…

# Grab the friendly mozilla build bootstrapper
cd ~/SourceInstall

curl -O

# Run the bootstrap script

# Do a test build to make sure it works
./mach build

./mach run

# edit file you want to change

./mach build

./mach run

# test your shortcut now works...

cd ~/SourceInstall/mozilla-unified/obj-x86_64-pc-linux-gnu/dist/bin

# or link to anywhere else in your path that
# will have higher precedence than /usr/bin

sudo ln -s $HOME/SourceInstall/mozilla-unified/obj-x86_64-pc-linux-gnu/dist/bin/firefox /usr/local/bin/firefox

Enjoy beginning to really customize Firefox… Hopefully for you this is the beginning of more meaningful Firefox hacking than just editing a config file.


Program like it’s 1989… Borland Turbo C++ for DosBox

You can download Borland Turbo C++ from Soft32, and run it inside Dosbox.

If your on Linux, you’ll be unpleasantly surprised to find this MSDOS classic has been wrapped up in a Windows installer that requires DotNet45 (AKA Mono), and tries to cram a bunch of spyware onto your system.

apt install wine winetricks
winetricks dotnet45
wine Turbo\ C++

Once the installer is done, you’ll have “Turbo” in your Wine/Window Downloads folder


$ sha256sum Turbo\
87ccaeb770f61f33ad6693fc8b7e5f8810538cda29fd6a9a388c3f1a984ea0ef  Turbo

$ md5sum Turbo\
3d525e9f65e3b3cf59185fa0887a3e54  Turbo
This .zip file is what you’ve really been after… Extract this to wherever your mapping your dosbox C: drive… For me, that’s: ~/dosbox

You’ll probably want to do some dosbox configuration, so you can give it more of a 1995 feel…

vim $(dosbox -printconf)


@echo off
mount c ~/dosbox

Once that’s all done, just type dosbox to open your new strangely oversized DOS window, then type tc to open the IDE…

You have syntax highlighting and a small set of libraries available…

Borland Turbo C++ 3.0, released 1992

To Install MASM (Microsoft Assembler) you’ll need to virtually mount floppy disks…

mount a: ~/dosbox/masm611/floppy -t floppy

Fortunately, dosbox doesn’t do any validation on the floppy maximum size… In the MASM installer, if you copy each of the disks (DISK1, DISK2… DISK5) into one folder and then mount that folder as one floppy… The MASM installer will install all of the files with no complaint.

Useful 16-bit Software… ls sed perl4 etc…

16-bit Vim… v7.1

Microsoft MASM Assembler for MSDOS – Windows NT

😱 OMG… Rose colored glasses

Actually using a 16-bit system for a few hours and it’s quite amazing how far we’ve come, especially Linux… Version 0.02 of Linux started when MASM for DOS and Borland Turbo C++ were still shipping.
Initially Linux with gcc v1.40 on top of Minix.

Amazingly, Minix still exists if you want to try to run it…

Minix was written for use in teaching operating system concepts. Would be a lot of fun to go through development from a kernel to a booting system.

The Good Old DOS Days we’re really the bad old days in terms of how software and the development environment worked… I would be very surprised if many people used to Linux in 2021 could write anything in Turbo C++ for Dos that was generally useful today… Not impossible to do, but just the very long way around…