{"id":3294,"date":"2017-12-30T20:48:34","date_gmt":"2017-12-30T19:48:34","guid":{"rendered":"https:\/\/kudzia.eu\/b\/?p=3294"},"modified":"2022-01-09T20:51:35","modified_gmt":"2022-01-09T19:51:35","slug":"a-wall-of-text-on-bad-and-good-choices","status":"publish","type":"post","link":"https:\/\/kudzia.eu\/b\/2017\/12\/a-wall-of-text-on-bad-and-good-choices\/","title":{"rendered":"A wall of text on bad and good choices"},"content":{"rendered":"<p>In this line of work, you don\u2019t just get to play with shiny toys having plenty of blinking lights. There\u2019s plenty of choices to be done nearly every day. Choices or rather bets: some of the technologies, software stacks, products or services provided internally will eventually be a flop. Decisions made over the years are done mostly based on gut feeling, opinions heard in the conference talks, podcasts, blogs, mailing lists, forums. Below \u2013 incomplete list of worse and better decisions done at my work, focusing on the architecture and sys-admin side.<\/p>\n<p>Bad ones:<\/p>\n<ul>\n<li>Deciding to go for 10GBase-T network standard using the familiar RJ45 connector rather than either fiber or SFP+DA for links between servers and switches. Looks like the days of RJ45 for the datacenter use are counted, power usage and latency are higher compared to the other alternatives, not many new devices support 10gbase-t.<\/li>\n<li>Not starting from the very beginning with \u2018drop by default\u2019 for both incoming and outgoing network traffic on most of the servers. Radical firewall policy changes done after a server is put into the production take much more time and are error-prone.<\/li>\n<li>Running too many services on a single server. This makes potential upgrades, backup restores much more cumbersome.<\/li>\n<li>Trying that one more time hardware from TP-Link, D-Link, Netgear and other cheap brands to see if it got any better. No \u2013 it did not, and the new stable firmware updates are unlikely to come.<\/li>\n<li>Having Active Directory but not utilizing it more. I\u2019m left with trauma from company where AD was a crucial part of setup, yet it was not dependable \u2013 it was too important to remove, too failed to repair. I\u2019m still torn weather we would be better off having no AD at all or using it across all desktops \/ file servers.<\/li>\n<li>Turn-key solutions with open source tools bundled to solve a specific problem. We\u2019ve used ESVA project for spam filtering, it got eventually orphaned by the sole maintainer. Being burned on it we\u2019ve decided to set up Asterisk phone system on a plain Debian server rather than use product like FreePBX. Same goes for using Samba under Debian instead of FreeNAS and many more.<\/li>\n<li>Backup drives connected via ESATA docking stations. And countless disk disconnects. After we\u2019ve moved to less efficient USB3-attached Startech docks all problems were gone.<\/li>\n<li>In the early years \u2013 buying underpowered desktops and laptops.<\/li>\n<li>Having too little documentation on what and why we\u2019ve set up.<\/li>\n<li>Having pet-like servers configured uniquely rather than a herd of identically set up machines. Dev-ops is only easy if server roles are very standardized.<\/li>\n<li>Assuming that \/24 networks \u2013 providing addresses for up to 253 hosts &#8211; are large enough for us.<\/li>\n<li>Not being generous enough with giving separate domain names for different internal services; even if they reside on the same host.<\/li>\n<li>Hardly a technical: saying \u2018yes\u2019 too often which led us to run too many bespoke solutions.<\/li>\n<\/ul>\n<p>Where I have mixed feelings:<\/p>\n<ul>\n<li>Office 365 \u2013 I\u2019m happy with Exchange being in the cloud, on the other hand Skype 4 business is far from problem-free, MS Teams did not help. One Drives and group Share Points are next to impossible to police, backup; Yammer is great, online collaboration of Word and Excel files \u2013 good enough.<\/li>\n<\/ul>\n<p>All doom and gloom, but hey \u2013 it took us where we are today : &#8211; ]<\/p>\n<p>Good ones:<\/p>\n<ul>\n<li>Using Linux and open source solutions where possible \u2013 leaves us with much more funds for hardware \/ service purchases, gives ability to test and scale with less constrains.<\/li>\n<li>Choosing Debian as a preferred distribution. We don\u2019t get the latest versions of the software stack, but stability has been great over the years.<\/li>\n<li>Getting the core network switches from reputable and sensibly priced HP, Dell instead of much more expensive Cisco or Juniper gear.<\/li>\n<li>Using Linux as a router, firewall, VPN end-point, load balancer instead of buying dedicated appliances. This might change over the years if we\u2019ll want to get Intrusion Prevention \/ Detection, Unified Threat Management Systems. Having Linux boxes at the edge of the network at each office gives us tremendous flexibility in introducing new services, monitoring, set up of failover etc.<\/li>\n<li>MySQL as a primary data store; relying on the built-in replication.<\/li>\n<li>[controversial] Java for the backend implementation, PHP for plenty of the internal glue-code and web-apps. .NET is tempting but it\u2019s nowhere near usable under Linux yet.<\/li>\n<li>[controversial] not using the cloud as in AWS, auto-scaling etc. The monthly bills for glorified VPS servers are huge compared with price of renting or owning physical servers.<\/li>\n<li>Don\u2019t letting servers to stagnate, upgrading to the most recent versions of Debian once they are available and reasonably tested.<\/li>\n<li>Moving away from Asus\/D-Link\/Zyxel WiFi devices,<\/li>\n<li>Going with OpenVPN tunnels for both site-to-site and dialup VPNs rather than IPSec that\u2019s to be a mess of incompatible implementations.<\/li>\n<li>Sticking with commodity x86 hardware and not going for gold-plated solutions like Storage Area Networks, PBXes, WiFi controllers. Using Dell servers rather than HP or other brands that prevent use of 3rd party disks, memory modules or extension cards.<\/li>\n<li>Having more spare hardware rather than paying extra for the premium support contracts.<\/li>\n<li>Encrypted, swappable disks for offline backups instead of tape drives or online-only backups.<\/li>\n<li>Labeling things, writing down what do we have on the shelves, taking photos of each of the offices. Those bits of information scattered across wikis, IP address databases, hardware inventories are priceless when we provide remote help.<\/li>\n<li>Moving quite early to Virtualization and Linux Containers.<\/li>\n<li>Buying more bandwidth rather than investing time in QoS and traffic shaping.<\/li>\n<li>Using VoIP rather than traditional PSTN\/ISDN telephony across offices scattered around the globe.<\/li>\n<li>Setting time-outs for tasks, giving up and moving on; there are certain failure scenarios that we\u2019ll not reproduce and even less likely repair. VPN or database replication with added watchdog script that restarts it occasionally is good enough, restarts every few weeks go unnoticed.<\/li>\n<\/ul>\n<p>Someday I should make a similar brain-dump about implementation details \u2013 bad code, bad data structures and why having an audit and verbose logs is always a good idea.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this line of work, you don\u2019t just get to play with shiny toys having plenty of blinking lights. There\u2019s plenty of choices to be done nearly every day. Choices or rather bets: some of the technologies, software stacks, products or services provided internally will eventually be a flop. Decisions made over the years are [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3294","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3294","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/comments?post=3294"}],"version-history":[{"count":3,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3294\/revisions"}],"predecessor-version":[{"id":3298,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/posts\/3294\/revisions\/3298"}],"wp:attachment":[{"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/media?parent=3294"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/categories?post=3294"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kudzia.eu\/b\/wp-json\/wp\/v2\/tags?post=3294"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}