/home/liquidat

[Short Tip] Get combined journal entries for multiple systemd services and Kernel messages

journalctl is a great tool to read and filter logs. Since it is context aware, it is much easier to use compared to just using tail on text files. It can be super helpful for example to filter output given on a specific service. But how do you combine filters with a logical OR?

Today I had the need to filter the output to kernel messages, and NetworkManager. Kernel messages is -k, and NetworkManager is -u NetworkManager in short. But the combination doesn’t reveal anything:

$ journalctl -u NetworkManager -k
-- No entries --

The reason is that the (implied) logical connection here is “AND”. Only output is shown that is both a kernel message and from NetworkManager – which doesn’t exist.

journalctl knows about the + operator to combine output – but that doesn’t help with the short options used above:

$ journalctl -u NetworkManager -k
-- No entries --

Instead, we need to use the long form, call out the systemd unit as a flag explicitly, and the transport for kernel:

$ journalctl _TRANSPORT=kernel +  _SYSTEMD_UNIT=NetworkManager.service --since today
...
Dez 18 21:26:52 russel NetworkManager[1510]: <info>  [1734553612.1558] dhcp4 (wlp9s0): state changed new lease, address=192.168.1.2
Dez 18 21:26:52 russel kernel: wlp9s0: Limiting TX power to 30 (30 - 0) dBm as advertised by 12:34:56:78:90:12

Of course, this can be combined with -f, --since today and other typical journalctl flags. It can even be extended by more services:

$ journalctl _TRANSPORT=kernel + _SYSTEMD_UNIT=NetworkManager.service + _SYSTEMD_UNIT=wpa_supplicant.service -f

[Short Tip] Get all columns in a table in Nushell

When working with larger data structures in Nushell, there are often tables that are wider than the terminal has width, resulting in some columns truncated, indicated by the three dots .... But how can we expand the dots?

❯ ls -la
╭───┬──────────────────┬──────┬────────┬──────────┬─────╮
│ # │       name       │ type │ target │ readonly │ ... │
├───┼──────────────────┼──────┼────────┼──────────┼─────┤
│ 0 │ 213-3123-43432.p │ file │        │ false    │ ... │
│   │ df               │      │        │          │     │
│ 1 │ barcode-picture. │ file │        │ false    │ ... │
│   │ jpg              │      │        │          │     │
│ 2 │ print-me-by-tomo │ file │        │ false    │ ... │
│   │ rrow.pdf         │      │        │          │     │
╰───┴──────────────────┴──────┴────────┴──────────┴─────╯

The answer is simple, but surprisingly, not easily found. The “Working with tables” documentation of Nushell weirdly doesn’t tell, for example. The trick is to use the command columns to get a list of all column names:

❯ ls -la|columns
╭────┬───────────╮
│  0 │ name      │
│  1 │ type      │
│  2 │ target    │
│  3 │ readonly  │
│  4 │ mode      │
│  5 │ num_links │
│  6 │ inode     │
│  7 │ user      │
│  8 │ group     │
│  9 │ size      │
│ 10 │ created   │
│ 11 │ accessed  │
│ 12 │ modified  │
╰────┴───────────╯

How do you get those now? Via the select command:

❯ ls -la|select name size modified
╭────┬──────────────────────────────────────┬──────────┬───────────────╮
│  # │                 name                 │   size   │   modified    │
├────┼──────────────────────────────────────┼──────────┼───────────────┤
│  0 │ .android                             │  4.0 KiB │ 2 years ago   │
│  1 │ .ansible                             │  4.0 KiB │ 2 years ago   │
│  2 │ .arduino15                           │  4.0 KiB │ 3 weeks ago   │
│  3 │ .audacity-data                       │  4.0 KiB │ 2 years ago   │
│  4 │ .aws                                 │  4.0 KiB │ 2 years ago   │
│  5 │ .bash_history                        │ 21.1 KiB │ 3 weeks ago   │
│  6 │ .bash_logout                         │     18 B │ 3 years ago   │

And once you know these commands, you can easily find the corresponding Nushell documentation: nushell.sh/commands/docs/columns.html

[Short Tip] Processing line by line in a loop in Nushell

For a test I recently had to process a plain list of items that was outputted by a program. In Bash, the usual way to do so is:

while read -r line; do COMMAND $line; done

But how could this be done in Nushell? Just using the same command gives an error:

❯ flatpak list|grep system|cut -f 2|while read -r line; do flatpak info $line; done
Error: nu::parser::parse_mismatch

  × Parse mismatch during operation.
   ╭─[entry #2:1:1]
 1 │ flatpak list|grep system|cut -f 2|while read -r line; do flatpak info $line; done
   ·                                              ─┬
   ·                                               ╰── expected operator
   ╰────

❯ flatpak list|grep system|cut -f 2|while read line; do flatpak info $line; done
Error: nu::parser::parse_mismatch

  × Parse mismatch during operation.
   ╭─[entry #3:1:1]
 1 │ flatpak list|grep system|cut -f 2|while read line; do flatpak info $line; done
   ·                                              ──┬─
   ·                                                ╰── expected block, closure or record
   ╰────

Instead, the trick is to tell Nushell to read the input line by line with lines, and then process each and every item with a sub-function:

flatpak list|grep system|cut -f 2| lines|each { |it| flatpak info ($it) }

This worked flawlessly.

[Short Tip] Using a Python virtual environment in Nushell

Nushell is becoming a more and more serious shell every day. One thing missing in the past was the capability to create and use Python virtual environments.

This has changed: Nushell was added as another supported shell in the virtualenv package:

🕙(20:39:55) ~/development
❯ virtualenv ansible
created virtual environment CPython3.11.5.final.0-64 in 190ms
creator CPython3Posix(dest=/home/liquidat/development/ansible, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(extra_search_dir=/usr/share/python-wheels,download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/liquidat/.local/share/virtualenv)
added seed packages: pip==22.3.1, setuptools==65.5.1, wheel==0.38.4
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

However, there is one catch: the source command does not work when you try to use it to switch to the new environment:

🕙(20:40:28) ~ 
❯ source ~/development/ansible/bin/activate.nu
Error: nu::parser::unexpected_keyword

  × Statement used in pipeline.
     ╭─[/home/liquidat/development/ansible/bin/activate.nu:116:1]
 116 │ export alias pydoc = python -m pydoc
 117 │ export alias deactivate = overlay hide activate
     ·                           ───┬───
     ·                              ╰── not allowed in pipeline
     ╰────
  help: 'overlay' keyword is not allowed in pipeline. Use 'overlay' by itself, outside of a pipeline.

Instead, you need to use the overlay command:

🕙(20:40:50) ~ 
❯ overlay use ~/development/ansible/bin/activate.nu
(ansible)

Afterwards, you can continue to operate in the environment like usual:

🕙(20:42:41) ~/development/ansible via 🐍 v3.11.5 (ansible) 
❯ pip install ansible
Collecting ansible
  Downloading ansible-8.4.0-py3-none-any.whl (47.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 MB 19.6 MB/s eta 0:00:00
[...]

[notice] A new release of pip available: 22.3.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
(ansible) 
🕙(20:43:15) ~/development/ansible via 🐍 v3.11.5 (ansible) took 20s 
❯

[Short Tip] Plot live-data in Linux terminal

Recently I realized that one of the disks in my server had died. After the replacement, the RAID sync started – and I quickly had to learn that this was going to take days (!). But I also learned that the time it might take massively jumped up and down.

Thus I thought it would be fun to monitor the progress of this. First, I just crated a command to watch the minutes (calculated into days) every few seconds with watch:

watch 'cat /proc/mdstat |grep recovery|cut -d " " -f 13|cut -d "=" -f 2|cut -d "." -f 1|xargs -n 1 -I {} echo "{}/60/24"|bc'

But since it was jumping so much I was wondering if I could live-plot the data in the terminal (remote server, after all). There are many ways to do that, even gnuplot seems to have an options for that, but I wanted something more simple. Enter: pipeplot

First I tried to use watch together with pipeplot, but it was easier to just write a short for loop around it:

while true;
do
  cat /proc/mdstat |grep recovery|cut -d " " -f 13|cut -d "=" -f 2|cut -d "." -f 1|xargs -n 1 -I {} echo "{}/60/24"|bc;
  sleep 5;
done \
| pipeplot

And the result is rather nice (also shown in the header image):