Kolla-Ansible OVN with SR-IOV 설치

개요 : OVN 환경에서 SR-IOV 설치하는 kolla ansible 환경 구성

 

1. openstack 구성

• OS : ubuntu 22.04
• OpenStack Version : Bobcat
• Deployment Tool : Kolla-Ansible
• Node : deploy, controller, network, compute
• Network(ML2) : OVN

 

2. network 구성

• external Network : External Network
• internal Network : Internal API
• tenant Network : Tenant Network 

 

3. SR-IOV 설정

• SR-IOV 설정 할 대상이 bond 되어 있을 경우 사용이 불가능하니 bond 해제 후 사용
• [compute] sr-iov enable

  # Bios Booting device setting → sr iov enable

• [compute] sr-iov vf 설정

# cat /sys/class/net/enp23s0f0/device/sriov_totalvfs 128 # 최대 설정 가능한 vf 개수수 # cat /sys/class/net/enp23s0f1/device/sriov_numvfs 0 # 현재 설정된 사용 가능한 vf 개수 # vi /sys/class/net/enp23s0f1/device/sriov_numvfs 16 # 16개의 vf를 생성

 

4. Netplan 설정

• Compute

# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      dhcp4: true
    eno2:
      dhcp4: true
    eno3:
      dhcp4: true
    eno4:
      dhcp4: true
    enp23s0f0:
      dhcp4: true
    enp23s0f1:
      dhcp4: true
    enp24s0f0:
      dhcp4: true
    enp24s0f1:
      dhcp4: true
    enp59s0f0:
      dhcp4: true
    enp59s0f1:
      dhcp4: true
    enp94s0f0:
      dhcp4: true
    enp94s0f1:
      dhcp4: true
    enp95s0f0:
      dhcp4: true
    enp95s0f1:
      dhcp4: true
  version: 2
  bonds:
    bond2:
      dhcp4: false
      dhcp6: false
      interfaces: ['enp23s0f0', 'enp24s0f1']
      mtu: 9000
      parameters:
        mii-monitor-interval: "100"
        mode: active-backup
  vlans:
    internal:
      addresses:
      - 172.19.244.12/24
      dhcp4: false
      dhcp6: false
      id: 220
      link: eno2
      mtu: 9000
    tenant:
      addresses:
      - 172.19.240.12/24
      dhcp4: true
      dhcp6: true
      id: 224
      link: bond2
      mtu: 9000
  bridges:
    external:
      interfaces: ['enp23s0f1', 'enp24s0f0']
      mtu: 9000
      addresses: [172.19.217.12/24]

 

5. OpenStack 배포

• [deploy] kolla-ansible globals.yml 수정

vi /etc/kolla/globals.yml

workaround_ansible_issue_8743: yes
kolla_base_distro: "ubuntu"
openstack_release: "2023.2"
node_custom_config: "{{ node_config }}/config"
kolla_internal_vip_address: "<public ip>"
kolla_external_vip_address: "192.168.10.111"
network_interface: "internal"
kolla_external_vip_interface: "eno1"
api_interface: "internal"
tunnel_interface: "tenant"

neutron_plugin_agent: "ovn"
enable_neutron_packet_logging: "yes"

enable_openstack_core: "yes"
enable_haproxy: "yes"
enable_horizon: "{{ enable_openstack_core | bool }}"

enable_neutron_sriov: "yes"
openvswitch_hw_offload: "yes"

neutron_sriov_physnet_mappings: {
  sriovnet1: "enp23s0f1",
  sriovnet2: "enp24s0f0"
}

enable_neutron_dvr: "yes"
enable_neutron_agent_ha: "yes"
enable_neutron_provider_networks: "yes"

nova_compute_virt_type: "kvm"
nova_console: "novnc”

 

• [deploy] compute node sriov_agent.ini 수정

vi /etc/kolla/config/neutron/compute01/sriov_agent.ini

[sriov_nic]
physical_device_mappings = sriovnet1:enp23s0f1,sriovnet2:enp24s0f0

 

• [deploy] nova conf 수정

vi /etc/kolla/config/nova.conf

[filter_scheduler]
enabled_filters = PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

 

• group_vars

vi /home/archiadmin/group_vars/control.yml

neutron_external_interface: "external"
neutron_bridge_name: "br-ex"

vi /home/archiadmin/group_vars/network.yml

neutron_external_interface: "external"
neutron_bridge_name: "br-ex"

vi /home/archiadmin/group_vars/compute.yml

neutron_external_interface: "enp23s0f1"
neutron_bridge_name: "br-ex"

 

[deploy] openstack 배포

• kolla-ansible -i ./multinode deploy

 

6. OpenStack 배포 후 작업

• [control] ml2 수정

vi /etc/kolla/neutron-server/ml2_conf.ini

[ml2]
type_drivers = flat,vlan,vxlan,geneve
tenant_network_types = geneve
mechanism_drivers = ovn
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges = sriovnet1,sriovnet2

[ml2_type_flat]
flat_networks = sriovnet1,sriovnet2

[ml2_type_vxlan]
vni_ranges = 1:1000

[ml2_type_geneve]
vni_ranges = 1001:2000
max_header_size = 38

[ovn]
ovn_nb_connection = tcp:<public ip>:6641
ovn_sb_connection = tcp:<public ip>:6642
ovn_metadata_enabled = True
enable_distributed_floating_ip = False
ovn_emit_need_to_frag = True

docker restart neutron_server

docker restart neutron_sriov_agent

 

7. 확인

• [seed] VM 생성

openstack network create --provider-network-type=flat --provider-physical-network=sriovnet1 sriovnet1

openstack network create --provider-network-type=vlan --provider-segment 217 --provider-physical-network=sriovnet1 sriovnet1-vlan217

openstack subnet create --network sriovnet1 --subnet-range=172.19.216.0/24 --allocation-pool start=172.19.216.100,end=172.19.216.200 sriovnet1-sub

openstack port create --network sriovnet1 --vnic-type=direct sriovnet1-port1

openstack server create --flavor m1.small --image init-ubuntu --nic port-id=`openstack port list | grep sriovnet1-port1 | awk '{print $2}'` sriov-vm

• 생성 후 lspci로 정상적으로 vf가 할당되었는지 확인
• vm이 위치한 compute node의 lspci로 정상적으로 vf가 할당 되었는지 확인