Our server is an i9-9900K at Hetzner. Eight LXC containers are running various Linux distributions on Proxmox-VE 7.3. Everything seems to work great! . . . Except inside one Alpine Linux container. Somehow, IPv6 doesn't work from within this one container. Nevertheless, IPv6 ping to the container somehow seems to work from the Proxmox node. Amazingly, ping also works to the container even from the Wide Area Network (WAN).

How could such strangeness happen? Let's investigate!

A Closer Look At The Problem

The network configuration for all the containers was added individually by hand in the Proxmox GUI. Proxmox numbers LXC containers beginning with the number "100". The containers are IPv6 only. Each container was given a static IPv6 address ending with the container's Proxmox container number.

The container with the IPv6 network problem is number 102. An example of the working containers is number 106. Number 106 was chosen for comparison because both container 102 and container 106 were built from the same Alpine Linux LXC image. The other working containers are Ubuntu and Debian.

Alpine Linux seems less likely to be causing the IPv6 network failure because container 106, also built from the Alpine image, is functioning just fine. So let's look directly at the misbehaving container, number 102.

Container 102 can ping localhost.

root@Proxmox-VE ~ # lxc-attach -n 102
~ # ping6 -c 2 localhost
PING localhost (::1): 56 data bytes
64 bytes from ::1: seq=0 ttl=64 time=0.024 ms
64 bytes from ::1: seq=1 ttl=64 time=0.036 ms

--- localhost ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.024/0.030/0.036 ms
~ # 

Container 102 also responds to ping from external WAN.

[opc@instance-20220717-1620 ~]$ ping -c 2 2a01:4f8:121:24cc::102
PING 2a01:4f8:121:24cc::102(2a01:4f8:121:24cc::102) 56 data bytes
64 bytes from 2a01:4f8:121:24cc::102: icmp_seq=1 ttl=51 time=154 ms
64 bytes from 2a01:4f8:121:24cc::102: icmp_seq=2 ttl=51 time=154 ms

--- 2a01:4f8:121:24cc::102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 153.966/154.042/154.118/0.076 ms
[opc@instance-20220717-1620 ~]$ 

However, from inside container 102, it can't ping its DNS servers, or the WAN.

root@Proxmox-VE ~ # lxc-attach -n 102
~ # cat /etc/resolv.conf
# --- BEGIN PVE ---
search metalvps.com
nameserver 2a01:4ff:ff00::add:2
nameserver 2001:470:20::2
# --- END PVE ---
~ # ping6 -c 2 2a01:4ff:ff00::add:2
PING 2a01:4ff:ff00::add:2 (2a01:4ff:ff00::add:2): 56 data bytes

--- 2a01:4ff:ff00::add:2 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
~ # ping6 -c 2 2001:470:20::2
PING 2001:470:20::2 (2001:470:20::2): 56 data bytes

--- 2001:470:20::2 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
~ # ping6 -c 2 metalvps.com
ping6: bad address 'metalvps.com'
~ # ping6 -c 2 ipv6.google.com
ping6: bad address 'ipv6.google.com'
~ # exit

By contrast, from inside working container 106, we can ping both the DNS servers and the WAN.

root@Proxmox-VE ~ # lxc-attach -n 106
~ # cat /etc/resolv.conf
# --- BEGIN PVE ---
search metalvps.com
nameserver 2a01:4ff:ff00::add:2
nameserver 2001:470:20::2
# --- END PVE ---
~ # ping6 -c 2 2a01:4ff:ff00::add:2
PING 2a01:4ff:ff00::add:2 (2a01:4ff:ff00::add:2): 56 data bytes
64 bytes from 2a01:4ff:ff00::add:2: seq=0 ttl=61 time=0.446 ms
64 bytes from 2a01:4ff:ff00::add:2: seq=1 ttl=61 time=0.274 ms

--- 2a01:4ff:ff00::add:2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.274/0.360/0.446 ms
~ # ping6 -c 2 2001:470:20::2
PING 2001:470:20::2 (2001:470:20::2): 56 data bytes
64 bytes from 2001:470:20::2: seq=0 ttl=57 time=18.516 ms
64 bytes from 2001:470:20::2: seq=1 ttl=57 time=18.552 ms

--- 2001:470:20::2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 18.516/18.534/18.552 ms
~ # ping6 -c 2 metalvps.com
PING metalvps.com (2603:c020:3:a9a9::250): 56 data bytes
64 bytes from 2603:c020:3:a9a9::250: seq=0 ttl=49 time=151.895 ms
64 bytes from 2603:c020:3:a9a9::250: seq=1 ttl=49 time=151.856 ms

--- metalvps.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 151.856/151.875/151.895 ms
~ # ping6 -c 2 ipv6.google.com
PING ipv6.google.com (2a00:1450:4001:830::200e): 56 data bytes
64 bytes from 2a00:1450:4001:830::200e: seq=0 ttl=118 time=5.477 ms
64 bytes from 2a00:1450:4001:830::200e: seq=1 ttl=118 time=5.611 ms

--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 5.477/5.544/5.611 ms
~ # 

Could The Firewall Cause This Problem?

The problem with container 102 probably is not a firewall drop egress issue because there aren't any egress rules in the container's zone and the default egress policy is ACCEPT. Also, the container's internal IPv6 egress problem still happens if container 102's firewall zone is set entirely off.

The ip Command

The ip command from the iproute2 suite provides much information about Linux networking. As the ip(8) man page explains, we can use ip in the following syntax: ip [OPTIONS] OBJECT { COMMAND | help }. In other words, to use ip we need to specify three items: an OPTION, an OBJECT, and a COMMAND.

For IPv4, the usual OPTION is nothing because ip defaults to IPv4. But, if we wish to specify the IPv4 OPTION explicitly, the option could be specified as "-family inet". The IPv4 OPTION can be abbreviated to "ip -f inet" or just "ip -4".

For IPv6, we use the OPTION "ip -family inet6". The IPv6 OPTION can be abbreviated to "ip -f inet6" or just "ip -6".

The OBJECT can be, for example, "link", "address", or "route". A "link" is a hardware or virtualized interface connecting our machine to a network. The "address" is the numerical IP address or addresses we assigned to our link. The "route" is the IP address where the kernel should send network packets initiated by programs running on our machine. "link", "address", and "route" can be abbreviated as "l", "a", and "r", respectively. Longer abbreviations also work. For example, "addr" commonly is used as an abbreviation for "address".

Frequently, when something happens involving networking, it can be very helpful to check the output of the ip command. Let's look with the ip command and see what we can find!

The Error Message

Checking the output of the ip -family inet6 address show command inside container 102 reveals "tentative dadfailed" as follows.

root@Proxmox-VE ~ # lxc-attach -n 102
~ # ip -f inet6 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP qlen 1000
    inet6 2a01:4f8:121:24cc::102/64 scope global tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::d8dd:65ff:fe45:70f2/64 scope link 
       valid_lft forever preferred_lft forever
~ # exit
root@Proxmox-VE ~ # 

Output of the same command from inside container 106 where networking works fine does not include "tentative dadfailed."

root@Proxmox-VE ~ # lxc-attach -n 106
~ # ip -f inet6 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if28: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP qlen 1000
    inet6 2a01:4f8:121:24cc::106/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::64ce:78ff:fe85:15fd/64 scope link 
       valid_lft forever preferred_lft forever
~ # 

Dadfailed

Some quick googling found a page called IPv6 'dadfailed' Problem. The problem there involved reassigning an address previously assigned to another machine which failed without the address being deleted. However, down toward the bottom of the page, we find key information: the "dad" in "dadfailed" is "Duplicate Address Detection."

No Mistake With Container 102's Creation

But, even after reading about Duplicate Address Detection, I still did not immediately find the solution. I was puzzled about how, like the dadfailed page that Google showed me, some machine possibly could be down but with its address still present on my clean, new server with all its containers working fine except for this single itty bitty networking issue inside one container. Nothing was down. All the containers had their own addresses, and all the containers successfully started and continued running.

I kept looking and looking for some mistake I somehow might have made while setting up container 102. I checked and recheckd the screenshots that recorded container 102's creation. But everything looked okay!

Asking For Help

Next I talked with a well known hosting provider about what was happening. I also wrote up a LES post asking for help with the puzzling issue. It was starting to get late by the time I finished drafting the LES post. I decided to sleep. I deferred posting to LES Help until the next morning.

When I woke up, however, the idea was in my mind that possibly I mistakenly also had assigned the 102 address to another container. Such a mistake didn't seem likely, though, because there were only 8 containers, and the container numbers, 100, 101, 102, 103, 104, 105, 106, and 107 seemed understandable. I decided to check all the other containers besides 102 to make sure no additional container inadvertently and duplicatively also had been assigned the 102 address.

Checking All The Containers

• Container 100

First I checked container 100. Container 100 was the test container that had been checked by @yoursunny and found to be "very strong." Container 100 seemed to have been assigned the correct IP address ending in 100.

• Container 101

Uh! Oh! Here it is! Container 101 got the 102 address!!

How Did The Error Happen?

Container 101 was the second container that was created. Obviously it should have an IP address ending with 2, right? Well, not when the first address ended with 0.

What About The Node And WAN Responses To Ping?

How could container 102 respond to a ping from external WAN while seeming unable to initiate a ping to an external WAN destination? Container 101 already had been created with the 102 address. So, the ping responses came from container 101.

How Do We Fix container 102?

All that's needed to fix container 102 is to change its address to any that is not already in use. I tried the 101 address, and that address immediately worked fine.

More Information About Dadfailed

The ip command has a manual page which doesn't mention dadfailed. However, the ip man page refers to the ip-address man page, which does mention dadfailed as an addition to the ip address show command.

   ip address show - look at protocol addresses

        [ . . . ]

       dadfailed
              (IPv6 only) only list addresses which have failed duplicate ad‐
              dress detection.

       -dadfailed
              (IPv6 only) only list addresses which have not failed duplicate
              address detection.

For example, inside the Linux container on my Chromebook:

chronos@penguin:~/log$ ip -6 address show dadfailed
chronos@penguin:~/log$ ip -6 address show -dadfailed
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::216:3eff:fe43:22fb/64 scope link 
       valid_lft forever preferred_lft forever
chronos@penguin:~/log$ 

Dadfailed In The RFCs

Duplicate Address Detection failure is discussed in section 5.4.5 of RFC 4862 which says:

5.4.5. When Duplicate Address Detection Fails

A tentative address that is determined to be a duplicate as described above MUST NOT be assigned to an interface, and the node SHOULD log a system management error.

Further discussion of Duplicate Address Detection occurs in Appendix A of RFC 4862 and in RFC 7527 Enhanced Duplicate Address Detection.

Dadfailed In The Logs

Indeed, the node did log a system management failure just as RFC 4862 said it should:

root@Proxmox-VE /var/log # zcat messages.2.gz | grep "duplicate address"
Jan 10 03:01:27 Proxmox-VE kernel: [97048.519242] IPv6: eth0: IPv6 duplicate address 2a01:4f8:121:24cc::102 used by 92:17:7a:22:c1:e2 detected!
[ . . . ]
Jan 13 06:44:36 Proxmox-VE kernel: [  141.636572] IPv6: eth0: IPv6 duplicate address 2a01:4f8:121:24cc::102 used by 92:17:7a:22:c1:e2 detected!
root@Proxmox-VE /var/log # 

Source Code

Recently I started to look at C source code. I don't know hardly anything about C, so looking at source code always is an adventure.

Above, we previously saw the ip command print the following "dadfailed" output:

~ # ip -f inet6 addr show
  [ . . . ]
2: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP qlen 1000
    inet6 2a01:4f8:121:24cc::102/64 scope global tentative dadfailed 
       valid_lft forever preferred_lft forever
  [ . . . ]

Can we find the source code which printed the dadfailed error? Finding the printing code sounds simple enough. Probably the printing code is nowhere near as complicated as the code which checked for and found the dadfailed error, right?

We saw the dadfailed error notification inside an Alpine container. Possibly that error notification came from iproute2 code running inside Busybox inside the Alpine container. Nevertheless, let's start by imagining how we might get the iproute2 source code that Proxmox itself uses. I'm unsure the method here gets the sources correctly, but we can try. First come the Proxmox sources, which seem to include the Debian sources, and then we get what might be the upstream sources.

root@Proxmox-VE ~ # git clone git://git.proxmox.com/git/iproute2.git
Cloning into 'iproute2'...
remote: Enumerating objects: 187, done.
remote: Total 187 (delta 0), reused 0 (delta 0), pack-reused 187
Receiving objects: 100% (187/187), 11.25 MiB | 88.63 MiB/s, done.
Resolving deltas: 100% (83/83), done.
root@Proxmox-VE ~ # 
cd iproute2/
root@Proxmox-VE ~/iproute2 # ls
debian  iproute2  Makefile  README
root@Proxmox-VE ~/iproute2 # cat README 
We compile our own package to make sure we always have the
latest version and bug fixes.
root@Proxmox-VE ~/iproute2 # ls debian/
changelog  copyright          iproute2-doc.examples  iproute2.links     README.Debian  source
compat     doc                iproute2-doc.install   iproute2.manpages  README.source
control    iproute2-doc.docs  iproute2.install       patches            rules
root@Proxmox-VE ~/iproute2 # ls iproute2/ #The iproute2/iproute2 directory is empty.
root@Proxmox-VE ~/iproute2 # git clone git://git.kernel.org/pub/scm/network/iproute2/iproute2.git
Cloning into 'iproute2'...
remote: Enumerating objects: 313, done.
remote: Counting objects: 100% (313/313), done.
remote: Compressing objects: 100% (289/289), done.
remote: Total 33973 (delta 60), reused 37 (delta 20), pack-reused 33660
Receiving objects: 100% (33973/33973), 8.34 MiB | 42.06 MiB/s, done.
Resolving deltas: 100% (25490/25490), done.
root@Proxmox-VE ~/iproute2 # 

Let's look inside the iproute2/iproute2 directory and then inside the iproute2/iproute2/ip directory.

root@Proxmox-VE ~/iproute2 # cd iproute2/
root@Proxmox-VE ~/iproute2/iproute2 # ls
bash-completion  COPYING  doc       genl     lib       misc   README        tc         vdpa
bridge           dcb      etc       include  Makefile  netem  README.devel  testsuite
configure        devlink  examples  ip       man       rdma   schema        tipc
root@Proxmox-VE ~/iproute2/iproute2 # cd ip
root@Proxmox-VE ~/iproute2/iproute2/ip # ls
ila_common.h           iplink.c            iplink_vlan.c    ipprefix.c          link_vti.c
ip6tunnel.c            iplink_can.c        iplink_vrf.c     iproute.c           link_xfrm.c
ipaddress.c            iplink_dsa.c        iplink_vxcan.c   iproute_lwtunnel.c  Makefile
ipaddrlabel.c          iplink_dummy.c      iplink_vxlan.c   iprule.c            nh_common.h
ip.c                   iplink_geneve.c     iplink_wwan.c    ipseg6.c            routel
ip_common.h            iplink_gtp.c        iplink_xdp.c     ipstats.c           rtm_map.c
ipfou.c                iplink_hsr.c        iplink_xstats.c  iptoken.c           rtmon.c
ipila.c                iplink_ifb.c        ipmacsec.c       iptunnel.c          static-syms.c
ipioam6.c              iplink_ipoib.c      ipmaddr.c        iptuntap.c          tcp_metrics.c
ipl2tp.c               iplink_ipvlan.c     ipmonitor.c      ipvrf.c             tunnel.c
iplink_amt.c           iplink_macvlan.c    ipmptcp.c        ipxfrm.c            tunnel.h
iplink_bareudp.c       iplink_netdevsim.c  ipmroute.c       link_gre6.c         xfrm.h
iplink_batadv.c        iplink_nlmon.c      ipneigh.c        link_gre.c          xfrm_monitor.c
iplink_bond.c          iplink_rmnet.c      ipnetconf.c      link_ip6tnl.c       xfrm_policy.c
iplink_bond_slave.c    iplink_team.c       ipnetns.c        link_iptnl.c        xfrm_state.c
iplink_bridge.c        iplink_vcan.c       ipnexthop.c      link_veth.c
iplink_bridge_slave.c  iplink_virt_wifi.c  ipntable.c       link_vti6.c
root@Proxmox-VE ~/iproute2/iproute2/ip # 

It was the address OBJECT which drew our interest, so let's look at ipaddress.c.

root@Proxmox-VE ~/iproute2/iproute2/ip # cat -n ipaddress.c | grep dadfailed
    68                  "           [-]tentative | [-]deprecated | [-]dadfailed | temporary |\n"
  1411          { .name = "dadfailed",          .mask = IFA_F_DADFAILED,        .readonly = true,      .v6only = true},
root@Proxmox-VE ~/iproute2/iproute2/ip # 

Line 68 seems to be part of the error message which is printed when ip doesn't understand the arguments given to it on the command line.

If we back up from line 1411 to line 1400, we can see the comment reproduced here, just below. Also, a little further down, at lines 1451 and 1452, there's code involving both "print" and "name." I'm guessing that maybe lines 1451 and 1452 might be responsible for printing "tentative" and "dadfailed."

  1400  /* Mapping from argument to address flag mask and attributes */
  1401  static const struct ifa_flag_data_t {
  [ . . . ]
  1411          { .name = "dadfailed",          .mask = IFA_F_DADFAILED,        .readonly = tru
e,       .v6only = true},
  [ . . . ]
  1414          { .name = "tentative",          .mask = IFA_F_TENTATIVE,        .readonly = tru
e,       .v6only = true},
  [ . . . ]
  1451                                  print_string(PRINT_FP, NULL,
  1452                                               "%s ", flag_data->name);

Finally, glancing at the Busybox ip command, the relevant source code seems to be ip.c. The usage message for ip address show is on lines 139 to 149.

139  //usage:#define ipaddr_trivial_usage
140  //usage:       "add|del IFADDR dev IFACE | show|flush [dev IFACE] [to PREFIX]"
141  //usage:#define ipaddr_full_usage "\n\n"
142  //usage:       "ipaddr add|change|replace|delete dev IFACE [CONFFLAG-LIST] IFADDR\n"
143  //usage:       "   IFADDR := PREFIX | ADDR peer PREFIX [broadcast ADDR|+|-]\n"
144  //usage:       "       [anycast ADDR] [label STRING] [scope SCOPE]\n"
145  //usage:       "   PREFIX := ADDR[/MASK]\n"
146  //usage:       "   SCOPE := [host|link|global|NUMBER]\n"
147  //usage:       "   CONFFLAG-LIST := [CONFFLAG-LIST] CONFFLAG\n"
148  //usage:       "   CONFFLAG := [noprefixroute]\n"
149  //usage:       "ipaddr show|flush [dev IFACE] [scope SCOPE] [to PREFIX] [label PATTERN]"

There is more on lines 341 to 347.

341  #if ENABLE_IPADDR
342  int ipaddr_main(int argc, char **argv) MAIN_EXTERNALLY_VISIBLE;
343  int ipaddr_main(int argc UNUSED_PARAM, char **argv)
344  {
345     return ip_do(do_ipaddr, argv);
346  }
347  #endif

ip_do is defined on lines 334 to 338.

334  static int ip_do(ip_func_ptr_t ip_func, char **argv)
335  {
336     argv = ip_parse_common_args(argv + 1);
337     return ip_func(argv);
338  }

Hopefully I soon will be able to understand more about how the iproute2 and Busybox code works to print the dadfailed error. Probably someone here at LES will be kind enough to provide some hints! Thanks very much! It is a lot of fun to learn a little about dadfailed!

If it's the former, what are some of the things sudo protects your server from if say, a sudo account is compromised/hacked?

It all started the day I dreamed of trying lower level dedicated server network and virtual machine configuration by myself instead of having Proxmox do the configuration for me. So, recently, I moved my MetalVPS server from Proxmox to Debian sid.

As part of the move from Proxmox to Debian sid, I got my clueless self into big trouble when I imagined setting up link layer networking inside the server. I imagined that, with link layer networking, virtual machines could use all sixteen of the server's expensive IPv4 addresses. Routing the extra subnet inside the server would have cost three IP addresses from the subnet, leaving only thirteen available for use by virtual machines. I wanted to use all 16 IPs.

Entire days passed by while I kept trying different network configurations. Eventually it seemed like I must have tried every possible configuration except the one configuration that would have worked! I kept reading and rereading the Hetzner tutorials plus several others that I found. Even posting here at LES and getting great responses wasn't enough to push me to victory.

Ultimately there seemed no choice other than to get help. But, from whom? I asked friends for referrals. I looked on Google. I checked Upwork and Fiverr. Most of the search results were about Microsoft, but I wanted Linux.

Eventually, however, I struck pure gold when I searched for "linux server administration -microsoft". There was this guy, Ratko, who seemed listed everywhere on every freelance site. And all the Ratko pages mentioned "networking." It seemed like Ratko was an exact fit for my problem!

But, which freelancing website to use? I've never used any of them, and so, as usual, I was clueless. I wondered if I could make direct contact with Ratko. It took awhile to figure out his last name. And then I found an email address! So I sent off an email.

Wow! Ratko replied almost immediately! Now, after working with Ratko, I know he always replies almost immediately. I don't know how he replies so fast! Well, "immediately" might be overstating the case. Maybe ten or twenty minutes. But that's amazing!

Ratko seemed to think he could fix my server's network configuration so that my plan to use all sixteen IP addresses in the subnet would work.

I made an account on the server for Ratko. Haha! I cluelessly forgot to install the ssh public key that Ratko had sent me at my request. Haha!

After he finally could log in, Ratko suggested changes in my attempted network configuration plus a few changes in Hetzner's or Debian's kernel network configuration and their qemu defaults, plus also a change in my qemu VM launch script.

Haha, well, nothing worked, but that was, as Ratko realized, because I had not left even one VM running.

Next it was time to look inside a running VM. I set one up. Ratko courageously and tirelessly fought through all the trouble of setting up an ssh tunnel! He then dove inside the test VM through qemu's VNC console interface!

Well, you will not be surprised that Ratko made a couple of changes to my VM configuration. And then. . . .

Suddenly, everything worked! It was like magic! Ratko comes in, looks carefully but quickly at what I had been trying for weeks. Then he makes a tweak here, there, everywhere. Suddenly, everything just works!

Now a few days and a few reboots, retests, and IP changes later, Ratko's entire configuration seems stable and indeed does appear to work for all the IPs in the subnet. Plus, no complaints from Hetzner!

Next came time for financial reckoning, Ratko asked me to send him an amount I thought was fair. While thinking about what number might be fair, I happened to notice a three digit number in one of the octets of the IP address of our test VM. Although it was an odd number, it nevertheless jumped out as perfect! I immediately sent Ratko that exact number of dollars.

Ratko told me the amount was generous. I won't say for sure that the funds I sent to Ratko were my best spent money ever, but I do say for sure that they might have been.

Working with Ratko is pure joy! Everything remains calm, but there is speed, focus, intensity, nuance, breadth, depth, vast context, wide experience, patience, courtesy, and great generosity! Plus perfect results!

Here is a link to Ratko's LinkedIn.

Highest thanks to Ratko for a super great job! Five stars! ⭐⭐⭐⭐⭐

Afterword

My biggest problem writing this review is imagining people won't believe what I am saying about Ratko because my review is too good to be true. Well, every language's literature has its classic hero saga genre. So now maybe we have a Beowulf for Linux network and system administration freelancers and their worthy clients.

I am delighted to speak privately with anyone considering hiring Ratko. If you contact me I promise additional superlatives! 🤩

I sent Ratko a few questions in case he wanted to answer them for you guys. I did not include a draft of this review. Ratko's answers are below.

For those interested, a few more tech details are posted here and elsewhere throughout that same thread.

Friendly greetings from New York City and Sonora, MX! 🗽🇺🇸🇲🇽🏜️

Interview with Ratko @PenguinGenius

Hi Tom,

Hey Ratko!
Hope you are doing great!

So far so good

How did you get started with Linux?

25+ years ago With Slackware 1.0 distro.

How did you get started with networking?

About same time, 1996. when I started working in Data Center, and I was
at Junior Network Engineer position first.

How did you get started freelancing?

Couple years after I got my regular job at University, as government
doesn't give high salaries
Friend of mine asked me to start as QA Engineer for his friends company,
which build some shareware applications for Windows.

How do you and your clients find each other?

Freelancer sites at first, LinkedIn, and finally - satisfied clients
recommendation

What has been your most complex and challenging freelance job?

Can't remember, I'm doing this for so many years...

How could your clients make your life easier?

With clear and complete info about problem, access, etc. And timely
response for my additional queries.

What's your schedule on days, nights, and weekends?

8am-3pm - in office at University - sometimes doing some fast freelance
tasks in office when I'm bored.
7pm-10pm - freelance work for clients (or more if needed)
Night - only if world catastrophic failure is in progress
Weekends - only super urgent requests (which of course have super-duper
high hourly rate), or some maintenance tasks I have scheduled over
weekend days.

If you also have a regular job, how do you manage conflicts?

I do have regular job. No conflicts - office work is primary. Then
clients' requests scheduled per urgency.
But in office first rule of system administration helps: "Best sysadmin
is one who is not doing anything!"
Meaning - he did all great, and nothing to do more (until any failure).

How can you respond to clients so quickly?

I know my value, and what I can do
And I'd like any my client to feel special and to know I take care for
him.

What advice would you give a new freelancer?

Don't Learn to Hack... Hack to Learn!
Even something seems hard - try to apply RTFM rule (Read The Fuckin'
Manual) before start to ask around. Try things (and know how to revert).
And don't feel insecure in yourself - otherwise, this is not for you.

• Disable password logins
• Change the default SSH port
• Consolidate servers
• Schedule backups
• Install htop

Any other tips?