Windows – erwin.co

TL/DR: If you run into problems with opensshd permissions on windows, open a PowerShell Administrator prompt and run:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Previously I wrote about Installing OpenSSH on Windows. For my workflow, I actually prefer to dual-boot Linux and Windows even though WSL2 has come a long way.

I use Barrier (open source successor to synergy) to share my mouse (well trackball) and keyboard across my workstation and laptop, regardless of whether Linux or Windows is running - I securely share the same underlying keys, and have the dhcp server assign a fixed IP to each MAC address.

It's actually quite tricky to get your OpenSSH keys from Linux's /etc/ssh/ssh_host_*key to C:\ProgramData\ssh\ssh_host_*key because of ACL details, even though I only edited the files with nvim - I thought that should preserve the icacls status, but it doesn't.

Windows iacls are a bit like selinux or AppArmor. Not a trivial subject, so be prepared if you're going to wade in.

iacls have inheritance, removed with /inheritance:r

For me, the most confusing thing about icacls is that if you break the permissions in certain ways (for example removing inheritance before you've granted some individual permissions to that file), you can no longer use icacls to fix them! You have to use takeown to re-assert ownership, and then you can start using icacls again.

PS C:\ProgramData\ssh> net stop sshd The OpenSSH SSH Server service was stopped successfully.

PS C:\ProgramData\ssh> net start sshd The OpenSSH SSH Server service is starting. The OpenSSH SSH Server service could not be started. A system error has occurred. System error 1067 has occurred. The process terminated unexpectedly.

Because I am not a master of icacls, I completely hosed my entire C:\ProgramData permissions while trying to fix ssh...

When trying to run sshd directly from the command line rather than via the windows service infrastructure, I actually got a bit more detail.

PS C:\WINDOWS\system32> sshd -dd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 158
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 158
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ecdsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

The PowerShell team provides a guide for exactly what ACL permissions are required to for your ssh_host_* files.

https://github.com/PowerShell/Win32-OpenSSH/wiki/Security-protection-of-various-files-in-Win32-OpenSSH

After several rounds of shooting myself in the foot with the not very memorable friendliness of icacls, I finally ran:

icacls "C:\ProgramData\ssh" /setowner system
icacls "C:\ProgramData\ssh" /q /c /t /reset
icacls "C:\ProgramData\ssh\ssh_host_*" /remove erwin

After that, I ran sshd -dd and finally was able to get OpenSSH to start up again on the command line without permissions errors, however running net start sshd still was failing to startup...

Turns out that running sshd -dd as just runs sshd in interactive mode under the currently logged on user (typically as an admin). To simulate the SYSTEM actually running sshd as a service you actually want to run:

psexec -s sshd.exe -ddd

(Note, psexec is part of sysinternals, probably easiest to install with shovel...)

PS C:\WINDOWS\system32> psexec -s sshd.exe -dd PsExec v2.34 - Execute processes remotely Copyright (C) 2001-2021 Mark Russinovich Sysinternals - www.sysinternals.com debug2: load_server_config: filename PROGRAMDATA\ssh/sshd_config debug2: load_server_config: done config len = 158 debug2: parse_server_config: config PROGRAMDATA\ssh/sshd_config len 158 debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2 debug1: get_passwd: LookupAccountName() failed: 1332. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions for 'PROGRAMDATA\ssh/ssh_host_rsa_key' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_rsa_key": bad permissions debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_rsa_key @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions for 'PROGRAMDATA\ssh/ssh_host_ecdsa_key' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ecdsa_key": bad permissions debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ecdsa_key @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions for 'PROGRAMDATA\ssh/ssh_host_ed25519_key' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ed25519_key": bad permissions debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ed25519_key sshd: no hostkeys available -- exiting. sshd.exe exited on XPS with error code 1.

So even though I had fixed the permissions enough for my user to run sshd it was not enough for the system to run sshd, the way that net start sshd works.

I wasn't even able to cd C:\ProgramData\ssh so I started with:

get-acl C: | set-acl C:\ProgramData

Then when I cd into C:\ProgramData\ssh turns out that the permissions are in fact way more open than what Window's SSHD (or Linux sshd for that matter) permit.

PS C:\ProgramData\ssh> icacls ssh_host_*key
ssh_host_dsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)
                 BUILTIN\Administrators:(I)(F)
                 BUILTIN\Users:(I)(RX)

ssh_host_ecdsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)
                   BUILTIN\Administrators:(I)(F)
                   BUILTIN\Users:(I)(RX)

ssh_host_ed25519_key NT AUTHORITY\Authenticated Users:(I)(M)
                     NT AUTHORITY\SYSTEM:(I)(F)
                     BUILTIN\Administrators:(I)(F)
                     BUILTIN\Users:(I)(RX)

ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)
                 BUILTIN\Administrators:(I)(F)
                 BUILTIN\Users:(I)(RX)

Successfully processed 4 files; Failed processing 0 files

So the easy way to do this on Windows is just to focus on one file at a time... We know that sshd complained about ssh_host_rsa_key first, so we'll start there.

# DONT DO THIS... IT WAS NOT EASY TO FIX...
icacls .\ssh_host_rsa_key /inheritance:r

So this removed the inheritance ACL from that file, and it's basically impossible to re-add...

Windows improbable answer for removing inheritance from a single file is to use takeown

takeown /R /F C:\ProgramData\ssh

# Then reset the ACLs to their default values

icacls C:\ProgramData\ssh /T /Q /C /RESET

After takeown runs, you'll be able to fix all the permissions again, but all the permissions will be messed up requiring them to be fixed 😉

Now we'll try again:

icacls.exe .\ssh_host_rsa_key

.\ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)
                   BUILTIN\Administrators:(I)(F)
                   BUILTIN\Users:(I)(RX)

Now I'm first going to explicitly grant Full Control to System.

icacls.exe .\ssh_host_rsa_key /grant SYSTEM:`(F`)
icacls.exe .\ssh_host_rsa_key /grant Administrators:`(F`)

Then I'm going to remove Inheritance

icacls.exe .\ssh_host_rsa_key /inheritance:r

However, unfortunately this still doesn't work... Even though this gives us the exact values that the PowerShell team documents as required:

ssh_host_rsa_key BUILTIN\Administrators:(F)
                 NT AUTHORITY\SYSTEM:(F)

ssh_host_rsa_key.pub NT AUTHORITY\SYSTEM:(F)
                     BUILTIN\Administrators:(F)

When starting up sshd with psexec we'll still get the error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for '__PROGRAMDATA__\\ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "__PROGRAMDATA__\\ssh/ssh_host_rsa_key": bad permissions
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key

After far too long of a detour today, I finally solved it with:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Unlike Linux, the parent directory permissions don't seem to matter.

Now, net start sshd works perfectly 🙂

OpenSSH Server on Windows is very cool, and very weird...

You need to open a PowerShell Prompt as Administrator

If you're going to connect to Windows via OpenSSH, I think you're really best off using PowerShell rather than cmd.exe, as PowerShell can do quite a bit more from the command line that's impossible with cmd.exe.

# Install the OpenSSH Server
Add-WindowsCapability -Online -Name OpenSSH.Server

# Start the SSH server
Start-Service sshd
# or the even shorter and older...
net start sshd

32-bit versions of windows shipped with a ported version of edit.exe from DOS that would work in a command prompt. The 64-bit editor that works in every version of windows and is guaranteed to be installed is notepad.exe. Obviously that's not going to get us very far via ssh so we need a way to edit in the terminal.

The best solution is to install neovim via scoop. If you're not into neovim, regular vim or nano (doesn't use vi keys) would also work.

# Allow PowerShell to run RemoteSigned code
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Install scoop
iwr -useb get.scoop.sh | iex

# Install git - required for scoop to operate
scoop install git

# Install neovim
scoop install neovim
# or `scoop install vim` if Lua is too fast for you
# or `scoop install nano` if hklj are too powerful...

# Install the VC++ runtime (recommended for neovim)
scoop install vcredist2015

Most regular *nix machines put the sshd configuration in /etc/ssh/sshd_config. For Windows, the /etc/ssh directory is:
C:\ProgramData\ssh. Inside of this directory you'll see:

Directory: C:\ProgramData\ssh


logs/
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_ed25519_key
ssh_host_ed25519_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
sshd.pid
sshd_config
administrators_authorized_keys

At the end of the Windows sshd_config you'll see:

Match Group administrators
       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

If you're in a multi-user corporate environment I would strongly recommend changing this to:

AuthorizedKeysFile __PROGRAMDATA__/ssh/%u/authorized_keys

The point is that in a multi-admin environment, you'll realistically want every admin to at least have their own home folder, and not effectively be sharing a login... Using Microsoft's default, the username is set by user set in the ssh user@server command.

If you use any path under ProgramData for your keys, you'll need to icalcls to set exactly these permissions, of course update your file name accordingly.

icacls.exe "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

In my case, I'm just running this as a virtual machine for development, so I just commented out the #match group Administrators Authorized Key File which simplifies permissions and behaves more like you expect of ssh. In a real server environment, I would definitely go with: PROGRAMDATA__/ssh/%u/authorized_keys

#Match Group administrators
#       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

Last, you need to add your ~/.ssh/some_key.pub public key to the authorized_keys file that you've chosen, and then restart sshd.

net stop sshd
net start sshd

Now you should be able to log in via:

ssh myuser@windows-host -p 22 -i ~/.ssh/some_key

Once you've got SSH connected, I would recommend setting ssh to launch powershell as the default shell rather than cmd, so that you can edit the registry via SSH.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

# Check that it's set with:
Get-Item -Path "HKLM:\SOFTWARE\OpenSSH"

You can list the current version of windows that's being used via cmd.exe's 'ver' shell builtin.

cmd.exe /c ver

# Windows "11" is:
Microsoft Windows [Version 10.0.22000.318]

Windows 11 ships with WSLg allowing display of GUI apps, supporting both Wayland and X11 APIs...

wsl --install -d Ubuntu