Categories
Uncategorized

How do variables really work in Dockerfiles?

Whether you're naming a Dockerfile ARG or ENV variable or a regular shell script variable, inside your Dockerfile they're all referenced as simply $var. For Docker Compose commands there's a special $$var syntax for variables that you don't want Docker Compose to interpolate.

Every docker RUN command is a completely separate process/environment, so if you're using a regular shell script variable, setting and getting the variable must all be done within one RUN command.

If you need to share a shell script variable $var across several RUN commands, it's definitely preferable to try and 1) refactor it into a single RUN command or 2) set the value in a build-arg and write a wrapper script. If neither of those work, then you should just read and write your variable value to/from files. For example:

RUN echo 1 > /tmp/__var_1
RUN echo `cat /tmp/__var_1`
RUN rm -f /tmp/__var_1

If your RUN command works in your shell but not via your Dockerfile RUN, it's likely a quoting issue.

  1. Docker RUN commands are hard coded to use /bin/sh -c. On many systems sh will be dash or ash or another very minimal shell with slightly different rules than the shell you typically use.
  2. Use a RUN echo ... command to make sure that any variables your command depends on are in fact set to what you think they're set to.
  3. Try to make the command you're having trouble with the first command in your file so that you can keep running your docker build with --no-cache and still minimize your wait time.
  4. When I was troubleshooting what also ended up being a quoting problem, I should have written my simplified test version like so:
RUN SHELL_PATH=$(head -n 1 /etc/shells) &&\
    useradd --shell $SHELL_PATH --uid 1000 foo

Even if you use zsh as your normal shell, I would recommend using bash to test your RUN commands like:

/bin/sh -c 'SHELL_PATH=$(head -n 1 /etc/shells) &&\
    useradd --shell $SHELL_PATH --uid 1000 foo'
  1. Dockerfile ARG values will overwrite any shell script variables that you set… For example, say we have this Dockerfile
FROM alpine:latest

ARG ArgFoo
ENV EnvFoo="Must be Set"

RUN echo "Value of ArgFoo is $ArgFoo"
RUN echo "Value of EnvFoo is $EnvFoo"
RUN ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
RUN echo "Our \$ShFoo is Gone Again: $ShFoo"

When we run:

$ docker build . --no-cache

Step 4/7 : RUN echo "Value of ArgFoo is $ArgFoo"
Value of ArgFoo is

Step 5/7 : RUN echo "Value of EnvFoo is $EnvFoo"
Value of EnvFoo is Must be Set

Step 6/7 : RUN ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
Value of ShFoo is awesome

Step 7/7 : RUN echo "Our \$ShFoo is Gone Again: $ShFoo"
Our $ShFoo is Gone Again:

If we ran it again, but this time with a --build-arg setting $ShFoo surprisingly, it's still difficult to get ourselves into trouble. First, update the Dockerfile to try and cause some trouble with $ShFoo

FROM alpine:latest

ARG ShFoo
RUN echo "$ShFoo" && ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"

$ docker build . --build-arg ShFoo="Difficult to cause trouble" --no-cache

Step 3/3 : RUN echo "$ShFoo" && ShFoo="awesome" && echo "Value of ShFoo is $ShFoo"
Difficult to cause trouble
Value of ShFoo is awesome

So Docker does a surprisingly good job of allowing you to reference any variable as just $var and everything just working.

The main difference when writing RUN commands really has more to do with /bin/sh -c than it does with Docker.

For example, I was working on a Dockerfile that would automatically set the permissions of the running Docker container to match the current user. Ultimately the command that worked was like this:

RUN shell=$(grep -E -m 1 \.\*\\b$USER_SHELL\\b /etc/shells) && \
    echo "DUMP: $shell $USER_ID:$GROUP_ID $USER_NAME:$GROUP_NAME" && \
    groupadd --gid $GROUP_ID $GROUP_NAME && \  
    useradd --shell $shell --uid $USER_ID --gid $GROUP_ID $USER_NAME

The subshell portion is:

grep -E -m 1 \.\*\\b$USER_SHELL\\b /etc/shells

Yet when executing in a normal shell I only need to use:

grep -E -m 1 '.*\bzsh\b' /etc/shells

When you have a fairly complex shell command inside of a docker RUN the format above of assigning a shell variable, then echoing everything so that you can be sure you really have what you think you have, is probably a good way to go if you start running into anything unexpected.

Categories
Uncategorized

Docker, Gemfile, Gemfile.lock, rbenv, rubygems, bundler or why good planning can often beat evolution

Rbenv → specifies a Ruby version

Ruby → loads some gems that are part of the base Ruby distribution (bundler, rubygems, irb, etc)

Rubygems →loads other gems (for the system or user) → GEM_HOME and GEM_PATH

Bundler → loads other gems (for the current project) → BUNDLE_PATH

So basically Ruby ships with a bunch of gems, rubygems manages most of the gems you install, and bundler manages all of the gems for a specific project, and both rubygems and bundler ship with Ruby.

The initial release of Docker was done at PyCon in 2013, but I wonder how much Ruby and the various shades of gem catostrophe helped inspire Docker adoption?

If you're using Docker and thus only running a single set of gems at a time, I think you should simplify the models as much as possible. For a ruby focused docker container, just skip rbenv and similar tools, try to bypass the system gems as much as possible, and try to get bundler / rubygems to force everything into /gems.

You'll want to check what bundler and rubygems detect as the environment via:

bundler env
gems env

Basically, inside of your docker-compose.yml and Dockerfile and perhaps even your dip.yml you want to set

HOME:        /app
BUNDLE_PATH: /gems
GEM_HOME:    /gems
GEM_PATH:    /gems
PATH:        /gems/ruby/<ruby-version>/bin:/gems/bin:/app/bin:$PATH

Also, I found my local docker build environment to be far less annoying when I figured out how to run everything as my local user... Ultimately I need to move to actually using a text template system to process all of the required variables (it's pretty lame that the Docker people make permissions that match the developer impossible without your own wrappers) but if it bothers you enough you could always fork docker.

ARG UID=<your id -u>
ARG GID=<your id -g>
ARG APPUSER=appuser

RUN set -x \
    && addgroup --system --gid $GID $APPUSER || true \
    && adduser  --system --disabled-login --ingroup $APPUSER --no-create-home --home /app --gecos "$APPUSER" --shell /bin/false --uid $UID $APPUSER || true

USER $APPUSER

# Why on earth does Docker's COPY always copy as root instead of the USER: you set? Only the Docker people know...
COPY --chown=$APPUSER: Gemfile* /app/

Then in your dip.yml makle sure to add run_options for your user...

interaction:
  shell:
    description: Open the Bash Shell in Container
    service: web
    command: bash
    compose:
      run_options: ["no-deps", "user=<your id -u>"]
Categories
Uncategorized

Getting CTRL+TAB to work in neovim

If you've been using a terminal for a while, you've surely stumbled across the terminal's legacy keyboard handling. Terminal escape codes were created in the 1970s and haven't been updated much.

CTRL + J == newline
CTRL + I == tab

These two examples are some of the least annoying, but the implications aren't at first obvious. Normal terminal apps can't tell whether you actually typed "ENTER" on your keyboard or "CTRL" + "I". From the perspective of the app, it's the same. Sadly, that's just the tip of the iceberg for these limitations.

CTRL + ... Sequences starting with CTRL+... are basically used up by the way the terminal represents common data (CTRL + D == EOF) with "CTRL CODES", and CTRL+SHIFT+letter is the same as CTRL+letter

ALT + C might be encoded as 0xc3, which collides with UTF-8 for characters like: é.

SHIFT + ... The terminal doesn't know the difference between "Shift + s" and "S"...

Hyper +... Hardly any apps are cool enough to be able to work with the awesome hyper modifier... Without an enhanced way to process keycodes, this isn't possible.

Many terminals rely on timing to distinguish codes like Alt+C from Esc + C... This generates lag and errors when typed too slow or too fast. vim-fixkey has a nice list of workarounds to get full access to keypresses in the terminal, though even in the best case scenario, it would be limited by a timeout, and not work for Cmd, Hyper or various CTRL+x keys...


Fortunately Leonerd, the author of libtermkey (now replaced by libtickit), wrote the "fixterms" specification and the author of Kitty has extended the fixterms spec to cover Super, Hyper and other modifier keys. iTerm 2 is also being extended to cover the new Kitty version of the fixterms spec.

Some of this is "worked around" with the XTerm ModifyOtherKeys option that's also supported in gnome-terminal and Konsole. Unfortunately ModifyOtherKeys is not complete.

There is discussion on the NeoVim github issues about adding fixterms support in neovim, but at this time (Nov 2021) it seems that no support is yet complete. It looks like neovim/src/nvim/keymap.c probably holds most of the code that needs to change.

https://github.com/neovim/neovim/blob/dc4670038e0441dfda7ba11c519b834624a1f6fd/src/nvim/keymap.c#L748-L755

Fix ambigous terminal key strokes? https://github.com/neovim/neovim/issues/176

Someone may have worked on an actual code change for this, tracking at: https://github.com/neovim/neovim/issues/6279


TUI: enable/disable modifyOtherKeys automatically https://github.com/neovim/neovim/issues/15352
TUI: distinguish Tab, CTRL-i (S8C1T mode) https://github.com/neovim/neovim/issues/5916
CTRL-Alt-Space isn't recognized even though terminal sends ^[^@, works in Vim https://github.com/neovim/neovim/issues/14836

Binding <M-S-Tab> https://github.com/neovim/neovim/issues/2379

TUI: S8C1T (8-bit) mode, v:termresponse https://github.com/neovim/neovim/issues/6279

See: "xterm-8-bit" Nvim does not use 8-bit sequence detection, and always uses 7-bit sequences (for now)

One day, hopefully Neovim will support these sequences by default, but in the mean time it's possible to map these sequences manually in Kitty and Neovim, it's possible to manually use a specific mapping by configuring Kitty's map ... send_text

https://github.com/kovidgoyal/kitty/issues/3248

# In Kitty.conf Example
map ctrl+enter send_text normal,application \x1b[13;5u

# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-cr>', ':echom "Hello C + R"<CR>')

Remember, the Kitty chart for progressive enhancement is:

BitMeaning
0b1 (1)Disambiguate escape codes
0b10 (2)Report event types
0b100 (4)Report alternate keys
0b1000 (8)Report all keys as escape codes
0b10000 (16)Report associated text

So \x1b = 11011... So it will turn on all options except "report alternate keys".

Decimal 13 == Carriage Return

; 5u == modifier flags == "Ctrl" + 1.

shift     0b1         (1)
alt       0b10        (2)
ctrl      0b100       (4)
super     0b1000      (8)
hyper     0b10000     (16)
meta      0b100000    (32)
caps_lock 0b1000000   (64)
num_lock  0b10000000  (128)

Another example...

# In Kitty.conf Example
map ctrl+tab     send_text normal,application \x1b[9;5u
map ctrl+shift+tab send_text normal,application \x1b[9;6u


# In init.lua
vimp = require('vimp')
vimp.bind('n', '<C-tab>', ':echom "Hello Tab"<CR>')
vimp.bind('n', '<C-S-tab>', ':echom "Hello Shift Tab"<CR>')

In this example, the relevant parts are:

\x 1b [ 9 ;5u
\x 1b [ 9 ;6u
   ^ 1b = 11011binary == turn on Disambiguate Esc Codes, Report Event Types, Report All Keys as Escaped, Report Associated Text
      [ is just the end of our escape code
         9 is the keycode for Tab
            5u = 1 (constant) + 4 (0100) Control
            6u = 1 (cont) + 5
                   4 = ctrl + 1 shift = 5

Let's go through a few more examples. Let's try: CTRL + Super + Tab. Unfortunately, even though Kitty can send this one, until Neovim's keyboard support is more complete, it doesn't look like Neovim has any way to receive this keycode.

CTRL     = 4
SUPER    = 8
Constant = 1
Total    =13 
# Remember, Tab = 0x9
\x 1b [ 9; 13u

Sadly, I can't get Super or Hyper based shortcut keys to work in Neovim or regular Vim yet, but I think it will be here soon...

Categories
Uncategorized

Program like it’s 1989… Borland Turbo C++ for DosBox

You can download Borland Turbo C++ from Soft32, and run it inside Dosbox.

If your on Linux, you'll be unpleasantly surprised to find this MSDOS classic has been wrapped up in a Windows installer that requires DotNet45 (AKA Mono), and tries to cram a bunch of spyware onto your system.

apt install wine winetricks
winetricks dotnet45
wine Turbo\ C++ 3.2.2.0.exe

Once the installer is done, you'll have "Turbo C++.zip" in your Wine/Window Downloads folder

Turbo C++.zip

$ sha256sum Turbo\ C++.zip
87ccaeb770f61f33ad6693fc8b7e5f8810538cda29fd6a9a388c3f1a984ea0ef  Turbo C++.zip

$ md5sum Turbo\ C++.zip
3d525e9f65e3b3cf59185fa0887a3e54  Turbo C++.zip
~/.wine/drive_c/users/$USER/Downloads
This .zip file is what you've really been after... Extract this to wherever your mapping your dosbox C: drive... For me, that's: ~/dosbox

You'll probably want to do some dosbox configuration, so you can give it more of a 1995 feel...

vim $(dosbox -printconf)

[sdl]
windowresolution=1280x960
output=opengl

[autoexec]
@echo off
mount c ~/dosbox
PATH %PATH%;C:\TURBOC3\BIN;C:\VIM\VIM70;
C:

Once that's all done, just type dosbox to open your new strangely oversized DOS window, then type tc to open the IDE...

You have syntax highlighting and a small set of libraries available...

Borland Turbo C++ 3.0, released 1992

To Install MASM (Microsoft Assembler) you'll need to virtually mount floppy disks...

mount a: ~/dosbox/masm611/floppy -t floppy

Fortunately, dosbox doesn't do any validation on the floppy maximum size... In the MASM installer, if you copy each of the disks (DISK1, DISK2... DISK5) into one folder and then mount that folder as one floppy... The MASM installer will install all of the files with no complaint.

Useful 16-bit Software... ls sed perl4 etc...

http://reimagery.com/fsfd/unix.htm

16-bit Vim... v7.1

https://www.vim.org/download.php#pc

Microsoft MASM Assembler for MSDOS - Windows NT

https://sourceforge.net/projects/masm611/


😱 OMG... Rose colored glasses

Actually using a 16-bit system for a few hours and it's quite amazing how far we've come, especially Linux... Version 0.02 of Linux started when MASM for DOS and Borland Turbo C++ were still shipping.
Initially Linux with gcc v1.40 on top of Minix.

Amazingly, Minix still exists if you want to try to run it...

Minix was written for use in teaching operating system concepts. Would be a lot of fun to go through development from a kernel to a booting system.

The Good Old DOS Days we're really the bad old days in terms of how software and the development environment worked... I would be very surprised if many people used to Linux in 2021 could write anything in Turbo C++ for Dos that was generally useful today... Not impossible to do, but just the very long way around...

Categories
Uncategorized

Adding a bit of security to your certain SFTP connections

So be sure when you step, Step with care and great tact. And remember that life's A Great Balancing Act.

Theodor Seuss Geisel

Security, like the rest of life, is a great balancing act. A rock on the bottom of the ocean is about the most secure thing in the world, but it's not terribly useful... There are always tradeoffs.

Generally, FTP should be avoided, because with normal FTP your passwords will be sent over the wire in plaintext and vulnerable to replay attacks. All of your data will also be sent over the wire in plaintext, so adiós to any sense of confidentiality for the data you sent that way. There are still times with FTP makes sense to use though, most especially over very lossy, high latency networks, when transferring content that is already public, like static website resources. If you're forced to use FTP, and you will do so repeatedly, make sure to choose a client and server that at least encrypt the authentication.

On any network connection that isn't the equivalent of institutional Grade D beef, you should be using SFTP (SSH File Transfer Protocol) or FTPS (FTP over TLS/SSL).

Sometimes you have a single User Account that you want to share with multiple devices. For example, maybe you want your Tablet to connect to SFTP to your Linux box, but you don't want that key to have full shell access.

Sometimes you should use the "full solution" of creating a separate user account, giving that user a locked down, perhaps chrooted shell, but then you need to figure out the user/group permissions of the files in question, how to maintain those permissions over time, and whether those permissions changes will be compatible with the rest of the required workflow.

There's a little used, not widely understood feature of SSH's ~/.ssh/authorized_keys file, you can prefix each key with:

# authorized_keys "command" example...
command="/usr/local/bin/file-transfer-only" ssh-rsa AAAA...

Then for /usr/local/bin/file-transfer-only you could put something like:

#!/bin/bash

case $SSH_ORIGINAL_COMMAND in
 '/usr/lib/openssh/sftp-server')
    exec /usr/lib/openssh/sftp-server
    ;;
 'scp'*)
    exec $SSH_ORIGINAL_COMMAND
    ;;
 'rsync'*)
    exec $SSH_ORIGINAL_COMMAND
    ;;
 *)
    echo "Access Denied"
    ;;
esac

Most Android "SFTP File Transfer" interfaces like Solid Explorer or the Fx File Manager use /usr/lib/openssh/sftp-server to get the SFTP transfer started, while most command line users will use scp or rsync to move files around.

This can be a nice in-between solution. Far better than sharing an SSH key between multiple devices (because you can easily remove it if any device/user is now gone, accountability, etc) but also a far simpler solution than a whole new account.

It ultimately depends on what you're doing.

You could even experiment with doing a chroot in this script to further tie things down, though keep in mind the chroot environment usually needs several things mounted for kernel access.

If you have any problems getting this setup, just turn on debugging on your SSHD server, and watch the log on your SSHD server, and you should be able to figure it out if you read carefully.

# /etc/ssh/sshd_config
# Change LogLevel
LogLevel DEBUG

# Save and Quit
systemctl reload ssh

On Debian, the SSHD logs will be under:

/var/log/auth.log

On RHEL the's are /var/log/secure

If it's neither of those, you might be stuck using journalctl to access the sshd logs.

If you turned on DEBUG level logging, don't forget to turn it back off.

Enjoy your Linux adventures!

Categories
Uncategorized

Vim one line file headers and footers

For more than 20 years I've been coming across files that have a headers or footers like:

$OpenBSD: sshd_config.5,v 1.45 2005/09/21 23:36:54 djm Exp$

/* vim: set ts=8 sw=4 tw=0 noet : */

Yet I've never found exactly where/how either of these tags are generated... Perhaps it just never got my attention enough. Well, all of that changes now.

The rcs (GNU RCS revision control system) first released in 1991, includes a command called ident. The ident command manual page starts:

NAME
       ident - identify RCS keyword strings in files

SYNOPSIS
       ident [ -q ] [ -V ] [ file ... ]

DESCRIPTION
       ident  searches for all instances of the pattern $keyword: text $ in the named files or, if no files are named,
 the standard input.

Not many people use rcs these days, but it's also possible to do this with other version managers. The key is to know that a "$...$" string is called an "Ident String".

#include <stdio.h>
static char const rcsid[] =
  "$Id: f.c,v 5.4 1993/11/09 17:40:15 eggert Exp $";
int main() { return printf("%s\n", rcsid) == EOF; }

If you want to setup ident strings with git, you can use:

$ echo '*.txt ident' >> .gitattributes
$ echo '$Id$' > test.txt
$ git commit -a -m "test"

$ rm test.txt
$ git checkout -- test.txt
$ cat test.txt

Note that before 2020, the default capability of "$ident$" strings in git was quite limited. Fortunately, "filters" have been extended to provide a lot more information, so you should be replicate ident in git now.

See this helpful note on Stack Overflow summarizing the 2020 git ident related changes.


What about the Vim footers?

Often times you come across a file with something like the following at the top or bottom

// vim: noai:ts=4:sw=4
   -or-
/* vim: noai:ts=4:sw=4
*/
   -or-
/* vim: set noai ts=4 sw=4: */
   -or-
/* vim: set fdm=expr fde=getline(v\:lnum)=~'{'?'>1'\:'1': */

These are called "vim modelines". The modeline can be within the first or last 5 lines in your file.

Best to only trust modelines on files where you trust the authors (like yourself).

Unfortunately modeline has been abused before. There are at least 5 CVE related modeline exploits over the years, so you're probably better off using "EditorConfig" for shared projects.

Categories
Uncategorized

How fast is Assembly? How slow are C and Go?

TL/DC Jump to the Test Results

I started my tech life working with FoxPro and the BASIC language... I wrote in SuperBase, Visual Basic, Perl, Ruby and Shell Scripts. All very high level languages. I always wanted to learn what was truly happening inside of the machine, but I was either too intimidated or too busy to learn low level languages. I moved into management, sales and business, and would code a bit over the years, but I had become a generalist.

A few years back I started getting back into writing software again. I looked at Perl 6. That's interesting, but why? I looked at some of the newer JavaScript and TypeScript, but none of it satisfied my curiosity. I wanted to know what's really going on. Something lower level and faster!

I started writing a fair amount of Go. It's a pretty nice language. Very easy. If I were writing a large app with several developers, I think Go may be a good choice. Everything is simple, compact, and standard. The more I got deeper into Go, the more I would see that I still didn't understand what's happening in the machine. I was reading something like "Advanced Go" that was talking about how to connect Go to C and I though, why am I doing this instead of just writing in C?

I had written a little bit of C in school, but never got very good at it. At that time, the money I could make solving other people's problems in high level languages was more interesting to me than understanding what was actually happening inside the machine.

https://cdnimg.mr-wu.cn/wp-content/uploads/2017/06/%E5%97%A8%E7%BF%BBC%E8%AF%AD%E8%A8%80.jpg
Head First C - Chinese Edition... I actually read the English version, though this one might be more fun

I found a great C book... Head First C. I wish my college had used this as the C textbook instead of the dry, sleep inducing tome the professors chose. I started writing C and enjoying it, but I had a lot of problems solving some of the memory errors, especially at first. You could say that I hit these problems because I didn't have very much experience in C. I would say I hit them because I had finally gotten closer to the processor, but I still didn't know what's going on.

So I thought to myself, a simple C introduction wasn't difficult at all. It was fun and enjoyable. What if there's something similar for Assembler?

http://www.java1234.com/uploads/allimg/200914/1-200914095504405.jpg
Assembly Language Step-By-Step

So I found Assembly Language: Step-By-Step. The author's idea is to teach Assembly Language as a first programming language. It's an interesting idea.

Anyway...

The blog post title... How fast is Assembly, and how slow are Go and C? The thing to remember with ASM is that each line of your assembly code corresponds to object code that you'll feed the processor.

SECTION .data			; init data

	HelloMsg: db "Hello world!",10 ;
	HelloLen: equ $-HelloMsg ;

SECTION .text			; code section

global _start  ; link needs this to find the entry point

_start:
	mov eax, 4        ; sys_write syscall
	mov ebx, 1        ; stdout (file 1)
	mov ecx,HelloMsg  ; pass offset of message
	mov edx,HelloLen  ; pass length of message
	int 80H           ; syscall

	mov eax, 1        ; exit syscall
	mov ebx, 0        ; return 0
	int 80H           ; syscall 

Assuming code is in a file named asm-hello.asm, run the following to build and test (assuming you're on a 64 bit machine)

nasm -f elf64 asm-hello.asm && \
ld -o asm-hello asm-hello.o && \
./asm-hello

If you've never written assembler before, and you're not quite sure what any of that means... eax ebx ecx and edx are registers. You trigger the syscall with interrupt 80H (code 128). If you want a better explanation of how and why it works, read the book linked above.

Performance Testing with perf

The interesting thing about this, is that I don't think you could make a hello world program significantly shorter.

sudo perf stat -r 1000 -d ./asm-hello

For this tiny asm example, you can see it took 0.00029 seconds to execute.

Total execution time: 0.000296 seconds

For the C version, I set -Ofast for maximum optimization. On such a simple program I don't think the optimization made any difference.

C version with -Ofast, Total execution time: 0.000498 seconds
Go version, Total execution time: 0.000984 seconds
Perl version, Total execution time: 0.00135

Of course it's very possible that you could write awful ASM and either never get it to work or get it to work so bad that it's even slower than any of the other options. It's just interesting to see how much different the speed can be. If you do have a part of a Go program that's too slow, it makes a lot of sense to move that part to C. If you have part of a C program that's too slow, it makes a lot of sense to move that part to ASM.

This clarifies for my why old programs on old hardware were so fast! Because RAM and CPU were both extremely expensive, lots of things were written in ASM.

Now that RAM and CPU prices are so low, most RAM segments and CPU cycles are thrown away on things like Electron...