.pQd's log – Just another WordPress weblog

manticore search available via ClickHouse

Uncategorizedclickhouse, manticoresearchLeave a Comment

i’m a big fan of ClickHouse; over time it became central hub for any sort of business intelligence analytics. at work we replicate data to it, use dictionaries loaded over http and mysql, recently – we’re also using MySQL table or database functions to be able to run queries with joins across data from various ... Read More

rsync without retransmitting moved files

techrsyncLeave a Comment

i’m using rsync a lot; both at work [ backups, replication of content to various servers, ad-hoc copying ] and privately. it’s smart enough to avoid re-sending the whole file if it has grown a bit [ like logs like to do ] or only few bytes changed in source or destination. out-of-the-box rsync is ... Read More

Ideas: rsync-like tool parallelizing transfer of large files

Uncategorizedproject ideas, rsyncLeave a Comment

i use rsync heavily for backups, batch jobs and one-off tasks. every day it moves around terabytes of data for me. while it’s great – saves time and bandwidth, it could make even better use of modern hardware – multicore, and often with plenty of unused storage IOps: one day, when i’ll have some spare ... Read More

Hong Kong, again

travel, Uncategorizedhong kongLeave a Comment

i got yet another chance to travel to Hong Kong. i’ve spent there 9 days in October this year. what did change since my previous visit in 2018? what did not change: few places that i’ve hiked around for the first time: Lamma Island – i got there with a ferry from Central. concrete path ... Read More

New supermicro server does not ‘see’ some of NVMe drives

UncategorizedLeave a Comment

i got 3 identical new servers from supermicro – SYS-121C-TN10R. all three have the same disk setup – 2+4 NVMe drives, but Debian 12 just see 5 of them instead of 6. when i run dmesg on newly installed system i can see those errors: to solve it i had to change PCIe bifurcation in ... Read More

getting code signing certificate on own Yubikey and then using it to sign windows binaries in unattended way, on KVM VM

UncategorizedLeave a Comment

starting from June 2023 code signing certificates for MS Windows cannot be delivered as a file anymore. files are easy to steal. now certs must reside on security modules which don’t allow private key extraction, at least not for mere mortals. we’re using such a cert to sign exe and msi files on a build ... Read More

using vector to send JSON-formatted nginx logs to ClickHouse

Uncategorizedclickhouse, nginxLeave a Comment

for some particular case i need to log and later analyze POST bodies of HTTP requests sent via nginx reverse proxy. ClickHouse will store the logs and be useful in analysis.

playing with min.io cluster

UncategorizedLeave a Comment

min.io is open source [ although AGPL-3 licensed ] object storage software providing s3-compatible interface and handling clustering for HA & data distribution. below – notes from setting it up on a tiny scale, without using Kubernetes. notes are based on RELEASE.2023-03-24T21-41-23Z i’ve fetched two binaries for the project: my cluster has 4 nodes, looks ... Read More

using hardware tokens to secure SSH, MS365 logins

UncategorizedLeave a Comment

i’ve done a bit of research, below – my understanding of the current state of affairs [ 2023-03 ]. i’m writing this while testing YubiKey 5 NFC, but consider different alternatives. SSH why: i’m considering an attack vector where malicious actor has remote control of my PC – can lift up arbitrary files [ including ... Read More

MySQL / MariaDB login audit

UncategorizedmysqlLeave a Comment

once in a while i need to review which database accounts are used, from what IPs connections arrive. MySQL / MariaDB does not have built-in mechanism [ unless you want to allow full query log ], but there’s quite easy way to get the data.

mysqldump headaches

UncategorizedLeave a Comment

over the years i’ve set up multiple backup workflows. one of them is described here. part of the cycles involves taking database backups. for MySQL i’m mostly using mysqldump. it comes with some headaches: it’s single-threaded by nature – both for backup, and restore [ unless you chop the dump file into pieces and try ... Read More

spooky HTTP requests arriving from the dark corners of the internet

UncategorizedLeave a Comment

i’ve spent most of the last week investigating spooky HTTP request that we’ve found in access logs of few production servers. very likely that traffic was replayed by mail content scanners used by messagelabs and mimecast.

UPC WiFree on Mikrotik and OpenWRT

UncategorizedLeave a Comment

UPC turns your cable modem into WiFi hot-spot available for others. outrageous! and i’ve been using it for quite a while – it generally works. below – how to connect OpenWRT or Mikrotik to such connection.

resizing btrfs on top of luks

techLeave a Comment

recently we needed to expand storage space available on one of our servers. originally it was using RAID10 on 4 4TBSSD drives handled by Dell’s PERC h730p controller, we wanted to add 2 more 4TB drives and go from 8 to 12TB array. we’ve decided to be brave and use RAID10 -> RAID10 array expansion. ... Read More

fighting a false-positive flagging by multiple antivirus vendors

techantivirusLeave a Comment

recently i woke up to this: “Hi, some of our employees are using your application. This morning they have received an upgrade notification (in yellow banner) to get the latest version of your app. Our anti-virus/malware has triggered on your module called “somefile.exe” detected at risk being a “Trojan.Gen.MBT “. below few resources that i’ve ... Read More

raspberry pi as a serial console terminal

techLeave a Comment

i’ve bricked a cheap home router. i’ve installed a beta of dd-wrt firmware which did not work. raspberry pi gave me serial connection to it.

bridging lan segments across untrusted links

techlinux networking, openvpnLeave a Comment

we’ve run out of the office space in one of the locations. in short term it was not possible to find a suitable and large enough place to rent so we had to split and relocate some of the staff to another building few kilometers away. it’s possible that we’ll shuffle people and servers between ... Read More

multi-master mysql replication with servers on 3 different continents

tech, UncategorizedmysqlLeave a Comment

at work i’m using mysql replication quite extensively. first it was a straightforward one-way replication that has been rock-solid for us since 2009. in 2012, for another type of data, we’ve started using master-master setup. initially the servers were in different European countries, eventually the secondary site was moved to North America while primary one ... Read More

MySQL on BTRFS?

Uncategorizedbtrfs, mysql1 Comment

i’ve been running a set of production MySQL databases on BTRFS since April 2016. BTRFS is not exactly known for its stellar performance when hosting databases or images of virtual machines due to its COW nature. why would i do it then? to have data snapshots and be able to ‘go back in time’ quickly ... Read More

A wall of text on bad and good choices

UncategorizedLeave a Comment

In this line of work, you don’t just get to play with shiny toys having plenty of blinking lights. There’s plenty of choices to be done nearly every day. Choices or rather bets: some of the technologies, software stacks, products or services provided internally will eventually be a flop. Decisions made over the years are ... Read More