Categories
Uncategorized

How do variables really work in Dockerfiles?

Whether you're naming a Dockerfile ARG or ENV variable or a regular shell script variable, inside your Dockerfile they're all referenced as simply $var. For Docker Compose commands there's a special $$var syntax for variables that you don't want Docker Compose to interpolate.

Every docker RUN command is a completely separate process/environment, so if you're using a regular shell script variable, setting and getting the variable must all be done within one RUN command.

If you need to share a shell script variable $var across several RUN commands, it's definitely preferable to try and 1) refactor it into a single RUN command or 2) set the value in a build-arg and write a wrapper script. If neither of those work, then you should just read and write your variable value to/from files. For example:

RUN echo 1 > /tmp/__var_1
RUN echo `cat /tmp/__var_1`
RUN rm -f /tmp/__var_1

If your RUN command works in your shell but not via your Dockerfile RUN, it's likely a quoting issue.

  1. Docker RUN commands are hard coded to use /bin/sh -c. On many systems sh will be dash or ash or another very minimal shell with slightly different rules than the shell you typically use.
  2. Use a RUN echo ... command to make sure that any variables your command depends on are in fact set to what you think they're set to.
  3. Try to make the command you're having trouble with the first command in your file so that you can keep running your docker build with --no-cache and still minimize your wait time.
  4. When I was troubleshooting what also ended up being a quoting problem, I should have written my simplified test version like so:
RUN SHELL_PATH=$(head -n 1 /etc/shells) &&\
    useradd --shell $SHELL_PATH --uid 1000 foo

Even if you use zsh as your normal shell, I would recommend using bash to test your RUN commands like:

/bin/sh -c 'SHELL_PATH=$(head -n 1 /etc/shells) &&\
    useradd --shell $SHELL_PATH --uid 1000 foo'
  1. Dockerfile ARG values will overwrite any shell script variables that you set… For example, say we have this Dockerfile
FROM alpine:latest

ARG ArgFoo
ENV EnvFoo="Must be Set"

RUN echo "Value of ArgFoo is $ArgFoo"
RUN echo "Value of EnvFoo is $EnvFoo"
RUN ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
RUN echo "Our \$ShFoo is Gone Again: $ShFoo"

When we run:

$ docker build . --no-cache

Step 4/7 : RUN echo "Value of ArgFoo is $ArgFoo"
Value of ArgFoo is

Step 5/7 : RUN echo "Value of EnvFoo is $EnvFoo"
Value of EnvFoo is Must be Set

Step 6/7 : RUN ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
Value of ShFoo is awesome

Step 7/7 : RUN echo "Our \$ShFoo is Gone Again: $ShFoo"
Our $ShFoo is Gone Again:

If we ran it again, but this time with a --build-arg setting $ShFoo surprisingly, it's still difficult to get ourselves into trouble. First, update the Dockerfile to try and cause some trouble with $ShFoo

FROM alpine:latest

ARG ShFoo
RUN echo "$ShFoo" && ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"

$ docker build . --build-arg ShFoo="Difficult to cause trouble" --no-cache

Step 3/3 : RUN echo "$ShFoo" && ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
Difficult to cause trouble
Value of ShFoo is awesome

So Docker does a surprisingly good job of allowing you to reference any variable as just $var and everything just working.

The main difference when writing RUN commands really has more to do with /bin/sh -c than it does with Docker.

For example, I was working on a Dockerfile that would automatically set the permissions of the running Docker container to match the current user. Ultimately the command that worked was like this:

RUN shell=$(grep -E -m 1 \.\*\\b$USER_SHELL\\b /etc/shells) && \
    echo "DUMP: $shell $USER_ID:$GROUP_ID $USER_NAME:$GROUP_NAME" && \
    groupadd --gid $GROUP_ID $GROUP_NAME && \  
    useradd --shell $shell --uid $USER_ID --gid $GROUP_ID $USER_NAME

The subshell portion is:

grep -E -m 1 \.\*\\b$USER_SHELL\\b /etc/shells

Yet when executing in a normal shell I only need to use:

grep -E -m 1 '.*\bzsh\b' /etc/shells

When you have a fairly complex shell command inside of a docker RUN the format above of assigning a shell variable, then echoing everything so that you can be sure you really have what you think you have, is probably a good way to go if you start running into anything unexpected.

Categories
Uncategorized

Docker, Gemfile, Gemfile.lock, rbenv, rubygems, bundler or why good planning can often beat evolution

Rbenv → specifies a Ruby version

Ruby → loads some gems that are part of the base Ruby distribution (bundler, rubygems, irb, etc)

Rubygems →loads other gems (for the system or user) → GEM_HOME and GEM_PATH

Bundler → loads other gems (for the current project) → BUNDLE_PATH

So basically Ruby ships with a bunch of gems, rubygems manages most of the gems you install, and bundler manages all of the gems for a specific project, and both rubygems and bundler ship with Ruby.

The initial release of Docker was done at PyCon in 2013, but I wonder how much Ruby and the various shades of gem catostrophe helped inspire Docker adoption?

If you're using Docker and thus only running a single set of gems at a time, I think you should simplify the models as much as possible. For a ruby focused docker container, just skip rbenv and similar tools, try to bypass the system gems as much as possible, and try to get bundler / rubygems to force everything into /gems.

You'll want to check what bundler and rubygems detect as the environment via:

bundler env
gems env

Basically, inside of your docker-compose.yml and Dockerfile and perhaps even your dip.yml you want to set

HOME:        /app
BUNDLE_PATH: /gems
GEM_HOME:    /gems
GEM_PATH:    /gems
PATH:        /gems/ruby/<ruby-version>/bin:/gems/bin:/app/bin:$PATH

Also, I found my local docker build environment to be far less annoying when I figured out how to run everything as my local user... Ultimately I need to move to actually using a text template system to process all of the required variables (it's pretty lame that the Docker people make permissions that match the developer impossible without your own wrappers) but if it bothers you enough you could always fork docker.

ARG UID=<your id -u>
ARG GID=<your id -g>
ARG APPUSER=appuser

RUN set -x \
    && addgroup --system --gid $GID $APPUSER || true \
    && adduser  --system --disabled-login --ingroup $APPUSER --no-create-home --home /app --gecos "$APPUSER" --shell /bin/false --uid $UID $APPUSER || true

USER $APPUSER

# Why on earth does Docker's COPY always copy as root instead of the USER: you set? Only the Docker people know...
COPY --chown=$APPUSER: Gemfile* /app/

Then in your dip.yml makle sure to add run_options for your user...

interaction:
  shell:
    description: Open the Bash Shell in Container
    service: web
    command: bash
    compose:
      run_options: ["no-deps", "user=<your id -u>"]
Categories
Rails

Setting ActionText Rich Text values for Polymorphic models in Rails 7 in the Console/IRB

Let's say you have a polymorphic model called Notes that you will be assigning to Projects and Authors, and Notes have a title and content.
Once you've edited the app/models/author.rb and app/models/project.rb to both have:

has_many :notes, as: :notable

And inside of app/models/note.rb you have:

belongs_to :notable, polymorphic: true
has_rich_text :content

(also, make sure you've done bin/rails action_text:install) then in rails console you should be able to run:

n = Project.first.notes.create
n.content.body = "<p>Some body content</p>"
n.title = "Note Title"
n.save

So the not totally obvious part is just using PolymorphicModel.your_actiontext_field.body. My first time around I named the actiontext field body, so then I had to search for body.body which was just too ugly, so I changed it to content.

Categories
Linux Windows

Sharing Windows OpenSSH keys for Linux Dual Boot

TL/DR: If you run into problems with opensshd permissions on windows, open a PowerShell Administrator prompt and run:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Previously I wrote about Installing OpenSSH on Windows. For my workflow, I actually prefer to dual-boot Linux and Windows even though WSL2 has come a long way.

I use Barrier (open source successor to synergy) to share my mouse (well trackball) and keyboard across my workstation and laptop, regardless of whether Linux or Windows is running - I securely share the same underlying keys, and have the dhcp server assign a fixed IP to each MAC address.

It's actually quite tricky to get your OpenSSH keys from Linux's /etc/ssh/ssh_host_*key to C:\ProgramData\ssh\ssh_host_*key because of ACL details, even though I only edited the files with nvim - I thought that should preserve the icacls status, but it doesn't.

Windows iacls are a bit like selinux or AppArmor. Not a trivial subject, so be prepared if you're going to wade in.

iacls have inheritance, removed with /inheritance:r

For me, the most confusing thing about icacls is that if you break the permissions in certain ways (for example removing inheritance before you've granted some individual permissions to that file), you can no longer use icacls to fix them! You have to use takeown to re-assert ownership, and then you can start using icacls again.

PS C:\ProgramData\ssh> net stop sshd
The OpenSSH SSH Server service was stopped successfully.

PS C:\ProgramData\ssh> net start sshd
The OpenSSH SSH Server service is starting.
The OpenSSH SSH Server service could not be started.

A system error has occurred.
System error 1067 has occurred.
The process terminated unexpectedly.

Because I am not a master of icacls, I completely hosed my entire C:\ProgramData permissions while trying to fix ssh...

When trying to run sshd directly from the command line rather than via the windows service infrastructure, I actually got a bit more detail.

PS C:\WINDOWS\system32> sshd -dd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 158
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 158
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ecdsa_key
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

The PowerShell team provides a guide for exactly what ACL permissions are required to for your ssh_host_* files.

https://github.com/PowerShell/Win32-OpenSSH/wiki/Security-protection-of-various-files-in-Win32-OpenSSH

After several rounds of shooting myself in the foot with the not very memorable friendliness of icacls, I finally ran:

icacls "C:\ProgramData\ssh" /setowner system
icacls "C:\ProgramData\ssh" /q /c /t /reset
icacls "C:\ProgramData\ssh\ssh_host_*" /remove erwin

After that, I ran sshd -dd and finally was able to get OpenSSH to start up again on the command line without permissions errors, however running net start sshd still was failing to startup...

Turns out that running sshd -dd as just runs sshd in interactive mode under the currently logged on user (typically as an admin). To simulate the SYSTEM actually running sshd as a service you actually want to run:

psexec -s sshd.exe -ddd

(Note, psexec is part of sysinternals, probably easiest to install with shovel...)

PS C:\WINDOWS\system32> psexec -s sshd.exe -dd

PsExec v2.34 - Execute processes remotely
Copyright (C) 2001-2021 Mark Russinovich
Sysinternals - www.sysinternals.com

debug2: load_server_config: filename PROGRAMDATA\ssh/sshd_config
debug2: load_server_config: done config len = 158
debug2: parse_server_config: config PROGRAMDATA\ssh/sshd_config len 158
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'PROGRAMDATA\ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_rsa_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_rsa_key
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'PROGRAMDATA\ssh/ssh_host_ecdsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ecdsa_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ecdsa_key
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for 'PROGRAMDATA\ssh/ssh_host_ed25519_key' are too open.

It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "PROGRAMDATA\ssh/ssh_host_ed25519_key": bad permissions
debug1: Unable to load host key: PROGRAMDATA\ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.
sshd.exe exited on XPS with error code 1.

So even though I had fixed the permissions enough for my user to run sshd it was not enough for the system to run sshd, the way that net start sshd works.

I wasn't even able to cd C:\ProgramData\ssh so I started with:

get-acl C: | set-acl C:\ProgramData

Then when I cd into C:\ProgramData\ssh turns out that the permissions are in fact way more open than what Window's SSHD (or Linux sshd for that matter) permit.

PS C:\ProgramData\ssh> icacls ssh_host_*key
ssh_host_dsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)
                 BUILTIN\Administrators:(I)(F)
                 BUILTIN\Users:(I)(RX)

ssh_host_ecdsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)
                   BUILTIN\Administrators:(I)(F)
                   BUILTIN\Users:(I)(RX)

ssh_host_ed25519_key NT AUTHORITY\Authenticated Users:(I)(M)
                     NT AUTHORITY\SYSTEM:(I)(F)
                     BUILTIN\Administrators:(I)(F)
                     BUILTIN\Users:(I)(RX)

ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                 NT AUTHORITY\SYSTEM:(I)(F)
                 BUILTIN\Administrators:(I)(F)
                 BUILTIN\Users:(I)(RX)

Successfully processed 4 files; Failed processing 0 files

So the easy way to do this on Windows is just to focus on one file at a time... We know that sshd complained about ssh_host_rsa_key first, so we'll start there.

# DONT DO THIS... IT WAS NOT EASY TO FIX...
icacls .\ssh_host_rsa_key /inheritance:r

So this removed the inheritance ACL from that file, and it's basically impossible to re-add...

Windows improbable answer for removing inheritance from a single file is to use takeown

takeown /R /F C:\ProgramData\ssh

# Then reset the ACLs to their default values

icacls C:\ProgramData\ssh /T /Q /C /RESET

After takeown runs, you'll be able to fix all the permissions again, but all the permissions will be messed up requiring them to be fixed 😉

Now we'll try again:

icacls.exe .\ssh_host_rsa_key

.\ssh_host_rsa_key NT AUTHORITY\Authenticated Users:(I)(M)
                   NT AUTHORITY\SYSTEM:(I)(F)
                   BUILTIN\Administrators:(I)(F)
                   BUILTIN\Users:(I)(RX)

Now I'm first going to explicitly grant Full Control to System.

icacls.exe .\ssh_host_rsa_key /grant SYSTEM:`(F`)
icacls.exe .\ssh_host_rsa_key /grant Administrators:`(F`)

Then I'm going to remove Inheritance

icacls.exe .\ssh_host_rsa_key /inheritance:r

However, unfortunately this still doesn't work... Even though this gives us the exact values that the PowerShell team documents as required:

ssh_host_rsa_key BUILTIN\Administrators:(F)
                 NT AUTHORITY\SYSTEM:(F)

ssh_host_rsa_key.pub NT AUTHORITY\SYSTEM:(F)
                     BUILTIN\Administrators:(F)

When starting up sshd with psexec we'll still get the error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions for '__PROGRAMDATA__\\ssh/ssh_host_rsa_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
debug1: Unable to load host key "__PROGRAMDATA__\\ssh/ssh_host_rsa_key": bad permissions
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key

After far too long of a detour today, I finally solved it with:

cd C:\ProgramData\ssh

takeown /R /F ssh_host*

icacls ssh_host* /T  /Q /C /RESET

icacls ssh_host* /grant SYSTEM:`(F`)

icacls ssh_host* /grant Administrators:`(F`)

icacls ssh_host* /inheritance:r

icacls ssh_host* /setowner system

Unlike Linux, the parent directory permissions don't seem to matter.

Now, net start sshd works perfectly 🙂

Categories
Docker Rails

Docker for Rails Developers in 2022

The pragprog team has a great book called "Docker for Rails Developers" however it's about 3 year old now... So just some minor notes here on how to make everything work right in Ruby 3 / Rails 7 world.

Docker is primary designed for deploying finished images of your software to production. Docker can work great in development, but just keep in mind that it's not really designed for development, so you'll always have to jump through a few extra hoops in exchange. Things like sharing files as your building an image and caching to speed up development builds are extra complicated. If you've ever setup a chroot environment, it's not as simply as just including the stuff you directly want to use --- there's overhead.

As I'm writing, Rails v7.0.3 with Ruby 3.1.2 is current.
Yet this book is based on Rails v5.2.2 and Ruby 2.6.

Docker makes these kind of version changes super easy. I recommend just sticking with exactly what the book uses and explicitly setting:

gem install rails -v 5.2.2

If you keep rails on v5.2.2 then all of the rest of the instructions should work, and when you start the rails server with:

docker run -p 3000:3000 ${my-image-id} bin/rails s -b 0.0.0.0

You should get the Rails 5 welcome page...

Your other also very easy option is to just switch to a Ruby 3.1.2 based image... So in your Dockerfile just use:

From: ruby:3.1.2
# instead of From: ruby:2.6

The examples are so simple from a Ruby perspective that you should be fine either way. Always comforting when your screenshots exactly match the authors though, so I would stick with Rails 5.2.2 and Ruby 2.6 and work though this short book.

Highly recommend this one though. Changed the way I think about Docker, especially all the docker-compose stuff.

Once you get to Chapter 7, Playing Nice with JavaScript, the Ruby 2.6 instructions really start to break down. Node 8 and Node 10 are deprecated, so you'll need to move to Node 12, and the default packages for that aren't compatible with the Debian base image that Ruby 2.6 is built on.

With Rails 7, we've kind of moved on from WebPacker anyway, but if you really want to do that part, I got it to build fine by using this as my Dockerfile.Basically, set ruby to 3.1.2, use setup_12.x for Node JS AND make sure to generate your rails app with Rails 7.

❯ cat Dockerfile        
FROM ruby:3.1.2

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \
    apt-transport-https

RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | \
    tee /etc/apt/sources.list.d/yarn.list

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \
    apt-utils \
    nodejs \
    yarn \
    vim

COPY . /usr/src/app

WORKDIR /usr/src/app
RUN bundle install

Webpacker comes up again in Advanced Gem Management and finally in Chapter 13, A Production-Like Playground.

The Rspec/testing chapter doesn't use Node or webpacker, so you can tear the yarn and nodejs update stuff out of your Dockerfile and get rid of Webpacker in your Gemfile.

FROM ruby:2.6

LABEL maintainer="erwin@fastmail.org"
LABEL foo="bar"

RUN apt-get update -yqq &&\
    apt-get install -yqq --no-install-recommends \
    apt-utils \
    nodejs \
    vim

COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN bundle install

COPY . /usr/src/app/

CMD ["bin/rails", "server", "-b", "0.0.0.0"]

The automated testing with Capybara/Selenium didn't work for me, but I've done some browser automation with Goland and Chrome so I didn't want to spend a lot of time debugging it. If you want to do the RSpec/Capybara test examples in 2022, probably best to reference this post instead:

https://dev.to/amree/ruby-on-rails-and-docker-for-testing-4n8a

The authors tutorial approach of working though each step a piece at a time was extremely helpful for me, even being fairly familiar with Docker, but passively as a developer/user without having been excited enough to deep dive into Docker itself.

Categories
Linux

How to completely remove Snaps? Install voidlinux

Ubuntu Linux is on a dark path these days. I used Debian on a lot of servers in the early 2000s, and like RedHat Enterprise, the Debian Stable stuff is just too old... So having a team "stabilize" Debian's Unstable distribution every six months was a fantastic idea and it worked well for a long time.

Unfortunately there is a big disconnect between who the Ubuntu team wants to serve and it's regular users. Sadly, Wayland seems to have this same disconnect, but that's another rant.

Snaps are just so awful. They break your themes, break your plugins and thus your workflow, block your access to files, clutter up your process list and mount list. Containerization is a great idea AS LONG AS you don't reduce function... So far, containerization on the desktop is not working. I would much rather use a heaver QubesOS like model rather than kind of working but breaking tons of advanced use cases.

voidlinux is an awesome alternative to Ubuntu:

  • Gets rid of Systemd, using very simple runit scripts for services
  • Void is not a FORK of anything. It's a new Linux developed from scratch.
  • It's a "Stable rolling release" distro, while Arch just focuses on Bleeding Edge rolling releases (that occasionally blow up in my face)
  • Easily install any Debian package with xdeb
    • I used it for Zoom, Opera and Jumpapp
  • Enjoy the super simple xbps package system
    • sudo xbps-install -Svu to update the cache the first time
    • sudo xbps-install etckeeper to install etckeeper
    • sudo xbps-install xtools to install xlocate
  • xlocate is a very fast replacement for apt-file search, just type xlocate '/xlocate$' it accepts any regex and will help you find any file inside of any xbps package, even the ones you haven't installed.
  • void is a minimal distro without any bloat

To get the install started, you just download the voidlinux ISO, which will startup a voidlinux live image. Open the terminal and run sudo void-installer - the password is "voidlinux".

That's basically it. Once you're booted into your own voidlinux install, install xtools and run xlocate to find whatever you're missing. Here's my list of daily use tools if you need something to help get started.

# Refresh the XBPS-Cache
sudo xbps-install -Svu

# Always start with etckeeper
sudo xbps-install etckeeper

# Install generic utilities
sudo xbps-install htop neofetch python3 sqlite kitty kitty-terminfo 
sudo xbps-install ddcutil mlocate ntp
sudo xbps-install tree ranger tmux screen direnv mosh fzf fzy
sudo xbps-install bind-utils git curl wget whois netcat socat
sudo xbps-install flameshot zsh xtools syncthing cheat sxhkd
sudo xbps-install rofi aws-cli zathura tdrop mpg123 mp3info 
sudo xbps-install xsel xmodmap xev xrandr wmctrl jump
sudo xbps-install binutils tar curl xbps xz bash-completion
sudo xbps-install base-devel

# KDE Plasma Desktop
sudo xbps-install plasma-desktop
sudo xbps-install kde5-baseapps kde-cli-tools qt5-tools
sudo xbps-install kdegraphics-thumbnailers ffmpegthumbs kwallet-pam
sudo xbps-install ksshaskpass kwalletmanager 
sudo xbps-install pavucontrol-qt pavucontrol

Categories
Linux X11

Adding a `hyper` modifier key to your keyboard, because shift/ctrl/alt/meta/super aren’t enough

TL/DR: Try this in your ~/.Xmodmap log out of X11 and log back in. If you moved to Wayland, hopefully one day we'll get stable APIs for 3rd party window utilities. Until that happens, I recommend switching back to X11 if you can.

clear mod1
clear mod2
clear mod3
clear mod4
clear mod5

keycode 248 = Hyper_L

add Mod1 = Alt_L Alt_R
add Mod2 = Num_Lock
add Mod3 = Hyper_L
add mod4 = Super_L Super_R
add mod5 = ISO_Level3_Shift Mode_switch

Note that you can't blindly copy code 248... You need to find a key that will work for your keyboard.

Soft fn key on keyboard

Use xev -event keyboard to find the keycode for the key that you want to use. While xev is running, focus on the xev window and press the key you want to use. You'll see the KeyRelease event show on your console. Best to use the xev -event keyboard filter so that all mouse events can be ignored.

In my example, this is AFTER I've use Xmodmap to map this key to Hyper_L... You won't see this until your remapping is successful.

You need to make sure Xmodmap will actually start with your widow manager, and for me, it works best with a delay. So I have a file ~/.config/autostart/xmodmap.desktop

[Desktop Entry]
Name=xmodmap
Comment=Xmodmap Keyboard Modification
Exec=delay_xmodmap
Terminal=false
Type=Application

And then delay_xmodmap is a trivial script that just sleeps for a bit before launching.

#!/bin/bash

sleep 10;
xmodmap ~/.Xmodmap

Using a hot key manager like sxhkd you should be able to map shortcuts to the Hyper key now. Put something like this in your ~/.config/sxhkd/sxhkdrc to test that Hyper is working.

hyper + {a-z}
        notify-send "hyper {a-z}"

Every keyboard I've ever seen has a SHIFT key and CONTROL keys. Historical Unix vs Mac vs Windows are a bit different for the other keys:
Alt == Meta (Option in the Mac World)
Windows Key == Super Key == Command Key

So you basically get 3 modifiers on each key, but SHIFT is so common for text it's dangerous for most of us to configure that.

If you're on Linux, and you have a keyboard that generates an actual "keycode" (some fn labeled keys actually run software in the keyboard to send a different keycode to the OS, without the OS knowing that a modifier was involved).

UnixWinMacLinux
mod1AltOptionAlt_L Alt_R / Meta
mod2Num LockNum Lock
mod3HYPER
mod4Windows KeyCommand KeySuper_L Super_R
mod5ISO_Level3_Shift
X11 mod1 keys

Categories
Linux

Git Self Hosted: GitLab vs Gitea vs Gogs in 2022

Github is a double edged sword. Use it when you must, but you should self-host whenever possible.

I started with gitlab as it was the most popular github alternative at the time, and I think for medium-large businesses hosting their own local, dedicated git instance, gitlab is probably a good choice.

For me though, gitlab wad definitely overkill. The upgrades get very complicated, and I'm ultimately just not a big Ruby on Rails fan... I just wanted locally hosted git that I could also browse easily via http, especially to graphically view older commits and branches, because every once in a while I do leave the terminal...

After gitlab my next solution was Gitea, as I read it was a community fork of gogs so I assumed it must be better... In hindsight, I would say Gitea is working hard to compete with GitLab. That's fine, but `GitLab` is actually what I wanted to get away from.

You can find a Gitea feature list here:

https://docs.gitea.io/en-us/comparison/

Here's an old sysadmin rule of thumb: If it involves LDAP, run the other way!

Me, screaming... every time I do something with LDAP...

In all fairness, LDAP is necessary for directory based authorization and authentication. With lots of users, you're kind of stuck with it and so you better know how it works. Everything connected to X.509 is a whole different mindset than the rest of the Linux and Open Source movement. If software had an attitude, ldap software and the zshell would be at opposite edges of the attitude spectrum.

I spent a long time actually getting gitea to work in my particular configuration, and just ended up dedicating way more time to it than I really wanted to. For me, git isn't very much different from samba... I just want it to work so I can focus on problems that I'm more interested in.

I would recommend gitea to anyone looking at gitlab and vice versa. These two are very similar. I think that blindly running either of them in a container just because you can is asking for trouble though. Go through the manual instillation and know how to set things up from scratch. If you can't do that, you shouldn't run it, because you won't be able to fix it when things go wrong. You want a sysadmin that knows how to set these up and how to manage them, back them up, and fix problems along the way.

So I spent far more hours than I'm proud to admit trying to get gitea working in my particular environment, but when I decided to switch over to gogs, I had it all working and my old repos migrated over within 30 minutes.

If you're just looking for a simple, self hosted git server via `https` in 2022, I think that gogs is what you are looking for.

https://github.com/gogs/gogs

Another nice "self hosted" option is cgit but the default interface is very dated, and it's even more minimal than my ideal feature set.

If neither gitlab, gitea, gogs or cgit are what you're looking for, maybe check out the awesome self hosted list, but a lot of these are very niche and not well supported.

Categories
Windows

Configuring OpenSSH-Server (sshd) on Windows 11

OpenSSH Server on Windows is very cool, and very weird...

You need to open a PowerShell Prompt as Administrator

If you're going to connect to Windows via OpenSSH, I think you're really best off using PowerShell rather than cmd.exe, as PowerShell can do quite a bit more from the command line that's impossible with cmd.exe.

# Install the OpenSSH Server
Add-WindowsCapability -Online -Name OpenSSH.Server

# Start the SSH server
Start-Service sshd
# or the even shorter and older...
net start sshd

32-bit versions of windows shipped with a ported version of edit.exe from DOS that would work in a command prompt. The 64-bit editor that works in every version of windows and is guaranteed to be installed is notepad.exe. Obviously that's not going to get us very far via ssh so we need a way to edit in the terminal.

The best solution is to install neovim via scoop. If you're not into neovim, regular vim or nano (doesn't use vi keys) would also work.

# Allow PowerShell to run RemoteSigned code
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Install scoop
iwr -useb get.scoop.sh | iex

# Install git - required for scoop to operate
scoop install git

# Install neovim
scoop install neovim
# or `scoop install vim` if Lua is too fast for you
# or `scoop install nano` if hklj are too powerful...

# Install the VC++ runtime (recommended for neovim)
scoop install vcredist2015

Most regular *nix machines put the sshd configuration in /etc/ssh/sshd_config. For Windows, the /etc/ssh directory is:
C:\ProgramData\ssh. Inside of this directory you'll see:

Directory: C:\ProgramData\ssh


logs/
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_ed25519_key
ssh_host_ed25519_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
sshd.pid
sshd_config
administrators_authorized_keys

At the end of the Windows sshd_config you'll see:

Match Group administrators
       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

If you're in a multi-user corporate environment I would strongly recommend changing this to:

AuthorizedKeysFile __PROGRAMDATA__/ssh/%u/authorized_keys

The point is that in a multi-admin environment, you'll realistically want every admin to at least have their own home folder, and not effectively be sharing a login... Using Microsoft's default, the username is set by user set in the ssh user@server command.

If you use any path under ProgramData for your keys, you'll need to icalcls to set exactly these permissions, of course update your file name accordingly.

icacls.exe "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"

In my case, I'm just running this as a virtual machine for development, so I just commented out the #match group Administrators Authorized Key File which simplifies permissions and behaves more like you expect of ssh. In a real server environment, I would definitely go with: PROGRAMDATA__/ssh/%u/authorized_keys

#Match Group administrators
#       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

Last, you need to add your ~/.ssh/some_key.pub public key to the authorized_keys file that you've chosen, and then restart sshd.

net stop sshd
net start sshd

Now you should be able to log in via:

ssh myuser@windows-host -p 22 -i ~/.ssh/some_key

Once you've got SSH connected, I would recommend setting ssh to launch powershell as the default shell rather than cmd, so that you can edit the registry via SSH.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

# Check that it's set with:
Get-Item -Path "HKLM:\SOFTWARE\OpenSSH"

You can list the current version of windows that's being used via cmd.exe's 'ver' shell builtin.

cmd.exe /c ver

# Windows "11" is:
Microsoft Windows [Version 10.0.22000.318]

Windows 11 ships with WSLg allowing display of GUI apps, supporting both Wayland and X11 APIs...

wsl --install -d Ubuntu

Categories
Uncategorized

Getting CTRL+TAB to work in neovim

If you've been using a terminal for a while, you've surely stumbled across the terminal's legacy keyboard handling. Terminal escape codes were created in the 1970s and haven't been updated much.

CTRL + J == newline
CTRL + I == tab

These two examples are some of the least annoying, but the implications aren't at first obvious. Normal terminal apps can't tell whether you actually typed "ENTER" on your keyboard or "CTRL" + "I". From the perspective of the app, it's the same. Sadly, that's just the tip of the iceberg for these limitations.

CTRL + ... Sequences starting with CTRL+... are basically used up by the way the terminal represents common data (CTRL + D == EOF) with "CTRL CODES", and CTRL+SHIFT+letter is the same as CTRL+letter

ALT + C might be encoded as 0xc3, which collides with UTF-8 for characters like: é.

SHIFT + ... The terminal doesn't know the difference between "Shift + s" and "S"...

Hyper +... Hardly any apps are cool enough to be able to work with the awesome hyper modifier... Without an enhanced way to process keycodes, this isn't possible.

Many terminals rely on timing to distinguish codes like Alt+C from Esc + C... This generates lag and errors when typed too slow or too fast. vim-fixkey has a nice list of workarounds to get full access to keypresses in the terminal, though even in the best case scenario, it would be limited by a timeout, and not work for Cmd, Hyper or various CTRL+x keys...


Fortunately Leonerd, the author of libtermkey (now replaced by libtickit), wrote the "fixterms" specification and the author of Kitty has extended the fixterms spec to cover Super, Hyper and other modifier keys. iTerm 2 is also being extended to cover the new Kitty version of the fixterms spec.

Some of this is "worked around" with the XTerm ModifyOtherKeys option that's also supported in gnome-terminal and Konsole. Unfortunately ModifyOtherKeys is not complete.

There is discussion on the NeoVim github issues about adding fixterms support in neovim, but at this time (Nov 2021) it seems that no support is yet complete. It looks like neovim/src/nvim/keymap.c probably holds most of the code that needs to change.

https://github.com/neovim/neovim/blob/dc4670038e0441dfda7ba11c519b834624a1f6fd/src/nvim/keymap.c#L748-L755

Fix ambigous terminal key strokes? https://github.com/neovim/neovim/issues/176

Someone may have worked on an actual code change for this, tracking at: https://github.com/neovim/neovim/issues/6279


TUI: enable/disable modifyOtherKeys automatically https://github.com/neovim/neovim/issues/15352
TUI: distinguish Tab, CTRL-i (S8C1T mode) https://github.com/neovim/neovim/issues/5916
CTRL-Alt-Space isn't recognized even though terminal sends ^[^@, works in Vim https://github.com/neovim/neovim/issues/14836

Binding <M-S-Tab> https://github.com/neovim/neovim/issues/2379

TUI: S8C1T (8-bit) mode, v:termresponse https://github.com/neovim/neovim/issues/6279

See: "xterm-8-bit" Nvim does not use 8-bit sequence detection, and always uses 7-bit sequences (for now)

One day, hopefully Neovim will support these sequences by default, but in the mean time it's possible to map these sequences manually in Kitty and Neovim, it's possible to manually use a specific mapping by configuring Kitty's map ... send_text

https://github.com/kovidgoyal/kitty/issues/3248

# In Kitty.conf Example
map ctrl+enter send_text normal,application \x1b[13;5u

# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-cr>', ':echom "Hello C + R"<CR>')

Remember, the Kitty chart for progressive enhancement is:

BitMeaning
0b1 (1)Disambiguate escape codes
0b10 (2)Report event types
0b100 (4)Report alternate keys
0b1000 (8)Report all keys as escape codes
0b10000 (16)Report associated text

So \x1b = 11011... So it will turn on all options except "report alternate keys".

Decimal 13 == Carriage Return

; 5u == modifier flags == "Ctrl" + 1.

shift     0b1         (1)
alt       0b10        (2)
ctrl      0b100       (4)
super     0b1000      (8)
hyper     0b10000     (16)
meta      0b100000    (32)
caps_lock 0b1000000   (64)
num_lock  0b10000000  (128)

Another example...

# In Kitty.conf Example
map ctrl+tab     send_text normal,application \x1b[9;5u
map ctrl+shift+tab send_text normal,application \x1b[9;6u


# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-tab>', ':echom "Hello Tab"<CR>')
vimp.bind('n', '<C-S-tab>', ':echom "Hello Shift Tab"<CR>')

In this example, the relevant parts are:

\x 1b [ 9 ;5u
\x 1b [ 9 ;6u
   ^ 1b = 11011binary == turn on Disambiguate Esc Codes, Report Event Types, Report All Keys as Escaped, Report Associated Text
      [ is just the end of our escape code
         9 is the keycode for Tab
            5u = 1 (constant) + 4 (0100) Control
            6u = 1 (cont) + 5
                   4 = ctrl + 1 shift = 5

Let's go through a few more examples. Let's try: CTRL + Super + Tab. Unfortunately, even though Kitty can send this one, until Neovim's keyboard support is more complete, it doesn't look like Neovim has any way to receive this keycode.

CTRL     = 4
SUPER    = 8
Constant = 1
Total    =13 
# Remember, Tab = 0x9
\x 1b [ 9; 13u

Sadly, I can't get Super or Hyper based shortcut keys to work in Neovim or regular Vim yet, but I think it will be here soon...