3

I have one ubuntu server running 20.04 with 2 NIC – one connected to the internet gateway and the other connected to another ubuntu server running 22.04. I cannot get the computer running 22.04 to see the internet or the 20.04 server. The 20.4 server does see the internet.

The yaml file on the 20.04 server is

# This is the network config written by 'subiquity'
network:


ethernets:
enp3s0:
dhcp4: false
addresses: ['10.0.0.205/24']
gateway4: 10.0.0.1
nameservers:
addresses: [10.0.0.1, 8.8.8.8]
enp2s0:
dhcp4: false
addresses: ['10.0.0.207/24']


version: 2

The lshw -C network command on the 20.04 server is

 sudo lshw -C network
  *-network
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: enp2s0
       version: 05
       serial: 1c:86:0b:22:73:5d
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.15.0-75-generic firmware=rtl8125b-2_0.0.2 07/13/20 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:17 ioport:e000(size=256) memory:df110000-df11ffff memory:df120000-df123fff memory:df100000-df10ffff
  *-network
       description: Ethernet interface
       product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0
       version: 15
       serial: 30:9c:23:0c:90:d9
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.15.0-75-generic duplex=full firmware=rtl8168h-2_0.0.2 02/26/15 ip=10.0.0.205 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:18 ioport:d000(size=256) memory:df004000-df004fff memory:df000000-df003fff

The yaml file for the 22.04 server is

# This is the network config written by 'subiquity'
network:
  ethernets:
        enp3s0:
            addresses: ['10.0.0.206/24']
            gateway4: 10.0.0.1
            nameservers:
                addresses: [8.8.8.8]
            routes:
              - to: default
                via: 10.0.0.207
  version: 2

The lshw -C network command on the 22.04 server is

sudo lshw -C network
  *-network
       description: Wireless interface
       product: Intel Corporation
       vendor: Intel Corporation
       physical id: 14.3
       bus info: pci@0000:00:14.3
       logical name: wlo1
       version: 11
       serial: 98:59:7a:99:6c:2c
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=iwlwifi driverversion=5.19.0-051900-generic firmware=72.a764baac.0 so-a0-gf-a0-72.uc latency=0 link=no multicast=yes wireless=IEEE 802.11
       resources: irq:18 memory:42314000-42317fff
  *-network
       description: Ethernet interface
       product: RTL8125 2.5GbE Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0
       version: 05
       serial: 74:56:3c:2c:f1:bc
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=5.19.0-051900-generic duplex=full firmware=rtl8125b-2_0.0.2 07/13/20 ip=10.0.0.206 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:18 ioport:3000(size=256) memory:42100000-4210ffff memory:42110000-42113fff

Also I cannot ping one server to the other.

@Raffa

Thank you for your quick reply. I have made the changes according to your latest instruction, but I still cannot ping the gateway (10.0.0.1) from server A.

The following is the ipconfig from server A

ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.206  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::7656:3cff:fe2c:f1bc  prefixlen 64  scopeid 0x20<link>
        inet6 2001:16a2:cb96:3c00::5  prefixlen 128  scopeid 0x0<global>
        inet6 2001:16a2:cb96:3c00:7656:3cff:fe2c:f1bc  prefixlen 64  scopeid 0x0<global>
        ether 74:56:3c:2c:f1:bc  txqueuelen 1000  (Ethernet)
        RX packets 306  bytes 26761 (26.7 KB)
        RX errors 0  dropped 60  overruns 0  frame 0
        TX packets 1621  bytes 150679 (150.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 1000  (Local Loopback)
RX packets 3793  bytes 519354 (519.3 KB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 3793  bytes 519354 (519.3 KB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


wlo1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
ether 98:59:7a:99:6c:2c  txqueuelen 1000  (Ethernet)
RX packets 0  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 0  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The following is the ipconfig from server B

ifconfig
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.207  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::1e86:bff:fe22:735d  prefixlen 64  scopeid 0x20<link>
        ether 1c:86:0b:22:73:5d  txqueuelen 1000  (Ethernet)
        RX packets 1428  bytes 99754 (99.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 131  bytes 8666 (8.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 10.0.0.205  netmask 255.255.255.0  broadcast 10.0.0.255
inet6 2001:16a2:cb96:3c00:329c:23ff:fe0c:90d9  prefixlen 64  scopeid 0x0<global>
inet6 fe80::329c:23ff:fe0c:90d9  prefixlen 64  scopeid 0x20<link>
inet6 2001:16a2:cb96:3c00::3  prefixlen 128  scopeid 0x0<global>
ether 30:9c:23:0c:90:d9  txqueuelen 1000  (Ethernet)
RX packets 31323  bytes 33052255 (33.0 MB)
RX errors 0  dropped 1228  overruns 0  frame 0
TX packets 14667  bytes 4587072 (4.5 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 1000  (Local Loopback)
RX packets 54250  bytes 38017605 (38.0 MB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 54250  bytes 38017605 (38.0 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

and the following are the iptables on sever B

 sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
f2b-phpmyadmin-syslog  tcp  --  anywhere             anywhere             multiport dports http,https
f2b-postfix-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,submissions,submission,imap2,imaps,pop3,pop3s
f2b-phpmyadmin-syslog  tcp  --  anywhere             anywhere             multiport dports http,https
f2b-postfix-sasl  tcp  --  anywhere             anywhere             multiport dports smtp,submissions,submission,imap2,imaps,pop3,pop3s
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports 2200
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2200
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:smtp
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imap2
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:submissions
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:submission
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imaps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3s
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:webmin
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:3478
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere


Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere


Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere


Chain f2b-phpmyadmin-syslog (2 references)
target     prot opt source               destination
REJECT     all  --  103.175.198.129      anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere


Chain f2b-postfix-sasl (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere


Chain f2b-sshd (1 references)
target     prot opt source               destination
REJECT     all  --  211.36.142.65        anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

Let me know if I can provide you with more details.

@Raffa,

Ok I have deleted all iptables rules and here is the iptables

/etc/iptables# cat rules.v4
# Generated by iptables-save v1.8.4 on Sat Aug 19 16:54:30 2023
*nat
:PREROUTING ACCEPT [8:428]
:INPUT ACCEPT [7:364]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o enp3s0 -j MASQUERADE
COMMIT
# Completed on Sat Aug 19 16:54:30 2023
# Generated by iptables-save v1.8.4 on Sat Aug 19 16:54:30 2023
*filter
:INPUT ACCEPT [37731:25530374]
:FORWARD ACCEPT [1311:95992]
:OUTPUT ACCEPT [38754:27316299]
-A FORWARD -i enp2s0 -o enp3s0 -j ACCEPT
-A FORWARD -i enp3s0 -o enp2s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Aug 19 16:54:30 2023

I still cannot ping the gateway from the 10.0.0.206 server Any ideas ?

Thanks

Following your last comment find the following :

sudo ethtool enp2s0 from server B

sudo ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes

And here is sudo ethtool enp3s0 from server A

Settings for enp3s0:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Auto-negotiation: on
        master-slave cfg: preferred slave
        master-slave status: slave
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes
Hisham Dawalibi
  • 31
  • Two interfaces on the same network on 20.04 server? (10.0.0.205/24 and 10.0.0.207/24) That's asking for trouble... Usually, you should NOT configure two interfaces on the same network on the same system, because you don't know via which interface the packet will be sent out, and on which interface will the response be received (all interfaces reply to ARP for each other), and newer kernels by default block asymmetric routing. Rethink your network design. – raj Aug 14 '23 at 18:56
  • @raj asymetric-routing(where outgoing and incoming packets for the same connection (or flow) use different interfaces) might happen with multiple networks, gateways, or exit "public" IPs i.e. the opposite situation of the OP ... See for example https://utcc.utoronto.ca/~cks/space/blog/tech/SymmetricAndAsymmetricIPRouting – Raffa Aug 16 '23 at 06:59
  • @Raffa packet going out via 10.0.0.205 and reply returning via 10.0.0.207 is asymmetric routing as well. – raj Aug 17 '23 at 13:45
  • @raj Agreed that's indeed asymmetric ,,, But, I guess you aren't addressing the situation in the question, right? ... As I don't see how would a packet leaving via 10.0.0.205 find it's way back in via 10.0.0.207 that is connected directly (not through a switch) to 10.0.0.206 ... Or why would the external internet router/NAT do that ... Or how would that happen even internally on the same sub-net level with the kernel's ARP tables in-place ... Or even why would communication between two hosts on the same sub-net be asymmetric at all ... That's a bit hard for me to imagine AFAIK. – Raffa Aug 17 '23 at 14:21
  • If something on the net queries ARP for 10.0.0.205, either 10.0.0.205 or 10.0.0.207 may answer, because that's how Linux TCP/IP stack works. So packet destined for 10.0.0.205 may actually arrive at 10.0.0.207. Two interfaces on the same network are strongly discouraged. – raj Aug 17 '23 at 14:32
  • @raj That would be just a hop in the route and not asymmetric routing per-se ... The ARP request will get echoed from the same interface that received the ping as I understand it ... And have actually previously traced it in fact out of boredom :-). – Raffa Aug 17 '23 at 14:56
  • To be able to ping 10.0.0.1 add a route "to 10.0.0.0/24 via 0.0.0.0" – Raffa Aug 18 '23 at 19:17
  • @Raffa, I am unable to ping 10.0.0.1 from 10.0.0.206. So what do you mean by your message above about adding a route to 10.0.0.0/24. Where should this be added ? – Hisham Dawalibi Aug 19 '23 at 19:24
  • I thought you meant to ping the gateway from server B at 10.0.0.205 as 10.0.0.1 is it's gateway as you need the route "to 10.0.0.0/24 via 0.0.0.0" which should be automatically there ... For server A at 10.0.0.206 it's gateway is 10.0.0.207 and it should be pingable ... 10.0.0.1 means nothing to server A which you have deliberately isolated from the rest of it's subnet members behind server B, isn't that what you want? ... That is how it works ... Is everything else working though? – Raffa Aug 20 '23 at 14:44
  • @Raffa. The problem has not been resolved. I still cannot ping from 10.0.0.206 to either 10.0.0.207 or 10.0.0.1. I spent so many hours on this and still cannot figure it out. – Hisham Dawalibi Aug 21 '23 at 08:30
  • In the least, you should be able to ping Server A at 10.0.0.207 from Server B at 10.0.0.206. If not, then perhaps the cable is faulty. Otherwise, how old is your hardware? If it’s old and you’re using a straight-through cable, then that could be a problem. Your hardware might not support auto MDI-X. In this case, a crossover cable is needed. Finally, have you attempted using a switch or hub between the servers? – mpboden Aug 21 '23 at 16:14
  • Thanks for your reply. The NICs are 1Gb/s and fairly new. I have tried using a crossover cable and it did not work. However when I connect the 2 servers through a switch, then I am able to ping in both directions. I have done that with the same cables I used for the direct connect between the 2 servers, so I guess it cannot be a faulty cable !!!. – Hisham Dawalibi Aug 22 '23 at 00:29
  • Sounds like the addition of a switch is the solution. Meanwhile, can you post output of sudo ethtool enp3s0 from server A and sudo ethtool enp2s0 from Server B? You may need to install ethtool if it’s not already. This output will show you if you have a link established when the cables are plugged in. It’ll also tell you if Auto MDI-X is enabled. – mpboden Aug 22 '23 at 03:23
  • I have added the posts that you have asked for, with the a link between thee 2 servers (not going through the switch) – Hisham Dawalibi Aug 22 '23 at 22:23
  • Since you’re able to ping with a switch inserted between the two hosts, and both nics indicate MDI-X as unknown, my presumption is that MDI-X is not auto-enabling. If so, this could explain why you’re having issues. You may want to force MDI-X by running sudo ethtool -s enp3s0 mdix on for Server A and sudo ethtool -s enp2s0 mdix on for Server B. – mpboden Aug 23 '23 at 00:17

2 Answers2

0

On server A

The minimal net-plan configuration file needed is:

network:
  ethernets:
    enp3s0:
      dhcp4: false
      addresses: [10.0.0.206/24]
      nameservers:
        addresses: [10.0.0.1, 8.8.8.8]
      routes:
        - to: default
          via: 10.0.0.207
  version: 2

On server B

The minimal net-plan configuration file needed is:

network:
  ethernets:
    enp3s0:
      dhcp4: false
      addresses: [10.0.0.205/24]
      nameservers:
        addresses: [10.0.0.1, 8.8.8.8]
      routes:
        - to: default
          via: 10.0.0.1
    enp2s0:
      dhcp4: false
      addresses: [10.0.0.207/24]
  version: 2

The minimal system configuration

IP forwarding(routing)
sudo sysctl -w "net.ipv4.ip_forward=1"
Packet forwarding (both outbound and inbound) between the two interfaces

outbound:

sudo iptables -A FORWARD -i enp2s0 -o enp3s0 -j ACCEPT

inbound:

sudo iptables -A FORWARD -i  enp3s0 -o enp2s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
NATing
sudo iptables -t nat -A POSTROUTING -o enp3s0 -j MASQUERADE
Raffa
  • 35,113
0

I solved it by installing a bridge on the 2 NIC's on Server B and doing the following command on the iptables on Server B

sudo iptables -t nat -A POSTROUTING -o enp3s0 -j MASQUERADE

Here is the yaml file on Server B

network:
  version: 2
#  renderer: networkd
  ethernets:
       enp3s0: {}
       enp2s0: {}


bridges:
br0:
interfaces: [enp2s0, enp3s0]
dhcp4: no
addresses: [10.0.0.205/24]
gateway4: 10.0.0.1
nameservers:
addresses: [10.0.0.1, 8.8.8.8]
parameters:
stp: true
forward-delay: 0

and here is the yaml file for Server A

network:
  ethernets:
        enp3s0:
            dhcp4: false
            addresses: ['10.0.0.206/24']
            gateway4: 10.0.0.1
            nameservers:
                addresses: [10.0.0.1, 8.8.8.8]

Thank you @Raffa for guiding me through the maze of IP networking.

Hisham Dawalibi
  • 31