tech

temporarily speeding up IO performance for KVM VM

by .pQd
i have few VMs running on top of spinning rust. recently i’ve upgraded one of them to not-yet-stable debian bookworm. upgrade would be pretty slow but.. i’ve edited the VM definition and asked KVM to allow host server to cache any IO operations. speedup was great, bullseye->bookworm upgrade finished in less than 5 minutes. for ... Read More

re-parenting a shell command

by .pQd
let’s say you’ve logged in via ssh to some server and started some long running command. minutes or hours later you’ve realized that you did not run it under screen or tmux – so that command will die once your ssh connection is closed. today i’ve learned about reptyr which can help with that – ... Read More

booting KVM VM from an iso image

by .pQd
let’s say you have a vm, maybe with debian. and that vm stopped booting, maybe because of grub corruption. what do you do? add under devices – for pc-i440fx arch: and for pc-q35-3.1: then in the os section change boot from dev=’hd’ to dev=’cdrom’: then also – this article describes how to go back from ... Read More

suddenly in-kernel communication between lxc containers gets erratic

by .pQd
we’re using LXC containers to host multiple workloads on the same physical servers. e.g. few instances of database servers running side-by-side. once in a while we end up with strange situation where tcp connections between containers running on the same physical server get torn down abruptly. in our case – this manifested e.g. by those ... Read More

resizing btrfs on luks on mdadm raid10

by .pQd
one of my KVM VMs keeps data on BTRFS on top of encrypted LUKS drive. that drive is a block device passed from virtualization server to VM… a block device which is mdadm software RAID10. below – steps taken to resize it.

resizing btrfs on top of luks

by .pQd
recently we needed to expand storage space available on one of our servers. originally it was using RAID10 on 4 4TBSSD drives handled by Dell’s PERC h730p controller, we wanted to add 2 more 4TB drives and go from 8 to 12TB array. we’ve decided to be brave and use RAID10 -> RAID10 array expansion. ... Read More

backup via ssh-tunneled connection

by .pQd
in one context i need to pull backup from server C to server A. normally i’d use rsync with direct ssh connection as a transport method. in this particular case C is not directly reachable from A, so i need to use ssh tunneling to connect from A to C via jump-host B: public ssh ... Read More

fighting a false-positive flagging by multiple antivirus vendors

by .pQd
recently i woke up to this: “Hi, some of our employees are using your application. This morning they have received an upgrade notification (in yellow banner) to get the latest version of your app. Our anti-virus/malware has triggered on your module called “somefile.exe” detected at risk being a “Trojan.Gen.MBT “. below few resources that i’ve ... Read More

openvpn – “OpenSSL: error:1408518A:SSL routines:ssl3_ctx_ctrl:dh key too small” after upgrade to Debina Buster

by .pQd
another thing to adjust after upgrade to Buster. on openvpn endpoint that in it’s config – /etc/openvpn/whatever.conf – had: dh dh1024.pem the vpn service did not start. tail -f /var/log/syslog showed: apparently the new version of openssl no longer accepts 1024 Diffie Hellman group. solution: and change in /etc/openvpn/whatever.conf – from dh dh1024.pem to dh ... Read More

bind9 fails to start after an upgrade to Debian Buster

by .pQd
i’ve done a routine upgrade to Buster on some server and all went fine… except bind9 no longer started. every attempt to restart it ended up with long waiting and an error message: logs did not contain anything interesting – actually they showed that bind would start and then after ~ a minute was shut ... Read More

allowing custom file extensions to be uploaded to wordpress

by .pQd
WordPress restricts types of files you can upload. it not only checks the extension of what you’re adding to the media library but also its mime type. so you cannot fool it by uploading .exe renamed to .jpeg – if you try it, you’ll get an error saying: “Sorry, this file type is not permitted ... Read More

“error: internal error: unable to execute QEMU command ‘transaction’: Could not create file: Permission denied” when creating a snapshot under KVM

by .pQd
debian buster brings apparmor. apparmor brings problems – eg it’s too restrictive for libvirt KVM guests and does not allow KVM to create snapshot-related files in VM’s folders. so far i did not find a clean and generic way to address it so i had to disable apparmor for libvirt by adding security_driver = “none” ... Read More

let’s encrypt via proxy server under debian

by .pQd
i prefer to have strict DROP policy for the outgoing traffic from production servers. let’s encrypt API endpoint is behind Akamai’s CDN and IP address to which acme-v02.api.letsencrypt.org resolves changes frequently. i don’t like playing whack-a-mole every 3 months so i’ve: set up a squid-based proxy server that allows for filtering based on domain names: ... Read More

bios upgrade on Dell PowerEdge T20 via AMT

by .pQd
prepare the floppy image file that later can be mounted via AMT, include in it just the bios update file – in my case it was PET20A18.exe from here. once the file is ready use the Manageability Commander Tool > Remote Control > Take control mount such img file as a virtual, oversize floppy. Using ... Read More

bridging lan segments across untrusted links

by .pQd
we’ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it’s possible that we’ll shuffle people and servers between ... Read More

multi-master mysql replication with servers on 3 different continents

by .pQd
at work i’m using mysql replication quite extensively. first it was a straightforward one-way replication that has been rock-solid for us since 2009. in 2012, for another type of data, we’ve started using master-master setup. initially the servers were in different European countries, eventually the secondary site was moved to North America while primary one ... Read More

E5-2643 in r620 stuck at a very low cpu frequency

by .pQd
few weeks ago i’ve done a general firmware upgrade of 2 identical Dell Poweredge r620 servers. a while later i’ve noticed that one of them had really sluggish performances. i’ve noticed that /proc/cpuinfo has shown CPU frequencies of hundreds of MHz rather than thousands. cat /proc/cpuinfo |grep MHz gave me 16 rows of values between ... Read More

lxc broken after upgrade from linux-image-4.12.0-0.bpo.1-amd64 to linux-image-4.13.0-0.bpo.1-amd64

by .pQd
lxc containers no longer start after i’ve upgraded debian stretch server from 4.12 kernel from stretch-backports repository to 4.13 also from backports. symptoms in /var/log/syslog: that was resolved by creating /etc/systemd/network/99-default.link with: via this comment. then there was nothing interesting in syslog but the lxc guest still would not start – lxc-start -F -n serverName ... Read More