Cisco MDS 9000 Family Configuration Guide, Release 1.2(2a) - Configuring IP Storage& [Cisco MDS 9000 NX-OS and SAN-OS Software] - Cisco Systems
Download the complete book
(PDF&-&7&MB)
Configuring IP Storage
Cisco MDS 9000 Family IP storage (IPS) services modules extend the reach of Fibre Channel SANs by using open-standard, IP-based technology. The switch connects separated SAN islands using Fibre Channel over IP (FCIP), and allows IP hosts to access Fibre Channel storage using iSCSI protocol.
This chapter includes the following sections:
Note FCIP and iSCSI features are specific to the IPS module and can be implemented in Cisco MDS 9216 switches or Cisco MDS 9500 Directors running Cisco MDS SAN-OS Release 1.1(x) or above.
The IPS services module (IPS module) allows you to use FCIP and iSCSI features. It integrates seamlessly into the Cisco MDS 9000 Family, and supports the full range of features available on other switching modules, including VSANs, security, and traffic management.
The IPS module can be used in any Cisco MDS 9000 Family switch and has eight Gigabit Ethernet ports. Each port can run FCIP and iSCSI protocols simultaneously.
•FCIP—FCIP transports Fibre Channel frames transparently over an IP network between two Cisco MDS 9000 Family switches or other FCIP standards-compliant devices.
depicts the FCIP scenarios in which the IPS module is used.
Figure 18-1 FCIP Scenarios
•iSCSI—The IPS module provides IP hosts access to Fibre Channel storage devices. The IP host sends SCSI commands encapsulated in iSCSI protocol data units (PDUs) to a MDS 9000 IPS port over a TCP/IP connection. At this point, the commands are routed from an IP network into a Fibre Channel network and forwarded to the intended target.
depicts the iSCSI scenarios in which the IPS module is used.
Figure 18-2 iSCSI Scenarios
After inserting the module, verify the status of the module using the show module command:
switch# show module
Module-Type
------------------------------- ------------------ ------------
1/2 Gbps FC Module
IP Storage Module
DS-X9308-SMIP
ok &----------------IPS module
Supervisor/Fabric-1
DS-X9530-SF1-K9
Supervisor/Fabric-1
DS-X9530-SF1-K9
ha-standby
World-Wide-Name(s) (WWN)
-----------
--------------------------------------------------
20:41:00:05:30:00:86:5e to 20:50:00:05:30:00:86:5e
20:c1:00:05:30:00:86:5e to 20:c8:00:05:30:00:86:5e
 0.602
 0.602
MAC-Address(es)
Serial-Num
--------------------------------------
----------
00-05-30-00-9f-62 to 00-05-30-00-9f-66
JAB064505YV
00-05-30-00-a1-ae to 00-05-30-00-a1-ba
JAB0649059h
00-05-30-00-9f-f6 to 00-05-30-00-9f-fa
JAB06350B1M
00-05-30-00-9f-f2 to 00-05-30-00-9f-f6
JAB06350B1F
* this terminal session
This section includes the following topics:
Both FCIP and iSCSI rely on TCP/IP for network connectivity. On the IPS module, connectivity is provided in the form of Gigabit Ethernet interfaces that are appropriately configured. This section covers the steps required to configure IP for subsequent use by FCIP and iSCSI.
A new port mode, called IPS, is defined for Gigabit Ethernet ports on the IPS module. IP storage ports are implicitly set to IPS mode, so it can only be used to perform iSCSI and FCIP storage functions. IP storage ports do not bridge Ethernet frames or route other IP packets.
Tip Gigabit Ethernet ports on the IPS module should not be configured in the same Ethernet broadcast domain as the management Ethernet port—they should be configured in a different broadcast domain, either by using separate standalone hubs or switches or by using separate VLANs.
depicts a basic Gigabit Ethernet configuration.
Figure 18-3 Gigabit Ethernet Configuration
To configure the Gigabit Ethernet interface for the scenario in , follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# interface gigabitethernet 2/2
switch(config-if)#
Enters the interface configuration mode on the Gigabit Ethernet interface (slot2, port 2).
Step 3 
switch(config-if)# ip address 10.1.1.100
255.255.255.0
Enters the IP address (10.1.1.100) and subnet mask (255.255.255.0) for the Gigabit Ethernet interface.
Step 4 
switch(config-if)# no shutdown
Enables the interface.
You can configure the switch to receive and transfer large (or jumbo) frames on a port. The default IP MTU frame size is 1500 bytes for all Ethernet ports. By configuring jumbo frames on a port, the MTU size can be increased to 9000 bytes. In this example, the size was set to 3000 bytes. Independent of the MTU size, the IPS module does not pack multiple IP frames (converted to FCIP or to iSCSI).
Note The minimum MTU size for a port running iSCSI is 620 bytes.
To configure MTU frame size, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# interface gigabitethernet 2/2
switch(config-if)#
Enters the interface configuration mode on the Gigabit Ethernet interface (slot2, port 2).
Step 3 
switch(config-if)# switchport mtu 3000
Changes the IP maximum transmission unit (MTU) to 3000. The default is 1500.
Virtual LANs (VLANs) create multiple virtual Layer 2 networks over a physical LAN network. VLANs provide traffic isolation, security, and broadcast control.
IPS Ethernet ports recognize the IEEE 802.1Q standard for VLAN encapsulation.
Note If the IPS module is connected to a Cisco Ethernet switch, verify the following requirements on the Ethernet switch:- the Ethernet switch port connected to the IPS module is configured as a trunking port, and - the encapsulation is set to 802.1Q and not ISL, which is the default.
To configure a VLAN subinterface (the VLAN ID), follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# interface gigabitethernet 2/2.100
switch(config-if)#
Specifies the subinterface on which 802.1Q is used (slot2, port 2, VLAN ID 100).
Note The subinterface number, 100 in this example, is the VLAN ID. The VLAN ID ranges from 1 to 4093.
Step 3 
switch(config-if)# ip address 10.1.1.100
255.255.255.0
Enters the IP address (10.1.1.100) and IP mask (255.255.255.0) for the Gigabit Ethernet interface.
Step 4 
switch(config-if)# no shutdown
Enables the interface.
Gigabit Ethernet interface (major), subinterfaces (VLAN tags) and management interfaces (mgmt 0) can be configured in the same or different subnet depending on the configuration (see ).
Table 18-1 Subnet Requirements for Interfaces
Gigabit Ethernet 1/1
Gigabit Ethernet 1/2
Two major interfaces can be configured in the same or different subnets.
Gigabit Ethernet 1/1.100
Gigabit Ethernet 1/2.100
Two subinterfaces with the same VLAN tag can be configured in the same or different subnets.
Gigabit Ethernet 1/1.100
Gigabit Ethernet 1/2.200
Two subinterfaces with different VLAN tags cannot be configured in the same subnet.
Gigabit Ethernet 1/1
Gigabit Ethernet 1/1.100
A VLAN tag cannot be configured on the same subnet as the major interface.
Gigabit Ethernet 1/1.100
The mgmt0 interface cannot be configured in the same subnet as the Gigabit Ethernet interfaces or subinterfaces.
Gigabit Ethernet 1/1
Note The configuration requirements in
also applies to Ethernet PortChannels.
To configure static IP routing through the Gigabit Ethernet interface, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# ip route
10.100.1.0 255.255.255.0 10.1.1.1
switch(config-if)#
Enters the IP subnet (10.100.1.0 255.255.255.0) of the IP host and configures the next hop 10.1.1.1, which is the IP address of the router connected to the Gigabit Ethernet interface.
The show ips ip route interface ethernet command takes the ethernet interface as a parameter and returns the route table for the interface. See .
Example 18-1 Displays the Route Table
switch# show ips ip route interface gig 8/1
Codes: C - connected, S - static
No default gateway
C 10.1.3.0/24 is directly connected, GigabitEthernet8/1
Connected (C) identifies the subnet in which the interface is configured (directly connected to the interface). Static (S) identifies the static routes that go through the router.
The ping command sends echo request packets out to a remote device at an IP address that you specify (see the ).
Once the Gigabit Ethernet interfaces are connected with valid IP addresses, verify the interface connectivity on each switch using the ping command. Ping the IP host using the IP address of the host to verify that the static IP route is configured correctly. See .
Example 18-2 Verifying Gigabit Ethernet Connectivity
switch# ping 10.100.1.25
PING 10.100.1.25 (10.100.1.25): 56 data bytes
64 bytes from 10.100.1.25: icmp_seq=0 ttl=255 time=0.1 ms
64 bytes from 10.100.1.25: icmp_seq=1 ttl=255 time=0.1 ms
64 bytes from 10.100.1.25: icmp_seq=2 ttl=255 time=0.1 ms
--- 10.100.1.25 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.1 ms
Note If the connection fails, verify the following, and repeat the ping command:- the IP address for the destination (IP host) is correctly configured,- the host is active (powered on), - the IP route is configured correctly, - the IP host has a route to get to the Gigabit Ethernet interface subnet, and - the Gigabit Ethernet interface is in the up state (use the show interface gigabitethernet command).
Use the show ips arp interface gigabitethernet command to display the ARP cache on the Gigabit Ethernet interfaces. This command takes the Ethernet interface as a parameter and returns the ARP cache for that interface. See .
Example 18-3 Displays ARP Caches
switch# show ips arp interface gigabitethernet 7/1
Hardware Addr
GigabitEthernet7/1
0004.76eb.2ff5
GigabitEthernet7/1
0003.47ad.21c4
GigabitEthernet7/1
GigabitEthernet7/1
GigabitEthernet7/1
GigabitEthernet7/1
GigabitEthernet7/1
GigabitEthernet7/1
The ARP cache can be cleared in two ways: clearing just one entry or clearing all entries in the ARP cache. See Examples
Example 18-4 Clearing One ARP Cache Entry
switch# clear ips arp address 10.2.2.2 interface gigabitethernet 8/7
arp clear successful
Example 18-5 Clearing All ARP Cache Entries
switch# clear ips arp interface gigabitethernet 8/7
arp clear successful
Note Use the physical interface, not the subinterface, to display TCP/IP statistics.
This section provides examples to verify Gigabit Ethernet and TCP/IP statistics on the IP storage ports.
Use the show interface Gigabit Ethernet command on each switch to verify that the interfaces are up and functioning as desired. See .
Example 18-6 Displays the Gigabit Ethernet Interface
switch# show interface gigabitethernet 8/1
GigabitEthernet8/1 is up
&-----------The interface is in the up state.
Hardware is GigabitEthernet, address is .a98e
Internet address is 10.1.3.1/24
MTU 1500 bytes, BW 1000000 Kbit
Port mode is IPS
Speed is 1 Gbps
Beacon is turned off
5 minutes input rate 744 bits/sec, 93 bytes/sec, 1 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
3343 packets input, 406582 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
8 packets output, 336 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors
Example 18-7 Displays the Gigabit Ethernet's Subinterface
switch# show interface gigabitethernet 4/2.100
GigabitEthernet4/2.100 is up
Hardware is GigabitEthernet, address is .abcb
Internet address is 10.1.2.100/24
MTU 1500 bytes
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 packets input, 0 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
1 packets output, 46 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors
The show ips stats mac interface gigabitethernet command takes the main Gigabit Ethernet interface as a parameter and returns Ethernet statistics for that interface. See .
Example 18-8 Displays Ethernet MAC Statistics
switch# show ips stats mac interface gigabitethernet 8/1
Ethernet MAC statistics for port GigabitEthernet8/1
Hardware Transmit Counters
237 frame 43564 bytes
0 collisions, 0 late collisions, 0 excess collisions
0 bad frames, 0 FCS error, 0 abort, 0 runt, 0 oversize
Hardware Receive Counters
427916 bytes, 3464 frames, 0 multicasts, 3275 broadcasts
0 bad, 0 runt, 0 CRC error, 0 length error
0 code error, 0 align error, 0 oversize error
Software Counters
3429 received frames, 237 transmit frames
0 frames soft queued, 0 current queue, 0 max queue
0 dropped, 0 low memory
You can display direct memory access (DMA) device statistics using the show ips stats dma-bridge interface gigabitethernet command. This command takes the main Gigabit Ethernet interface as a parameter and returns Ethernet statistics for that interface. See .
Example 18-9 Displays DMA-Bridge Statistics
switch# show ips stats dma-bridge interface gigabitethernet 7/1
Dma-bridge ASIC Statistics for port GigabitEthernet7/1
Hardware Egress Counters
231117 Good, 0 bad protocol, 0 bad header cksum, 0 bad FC CRC
Hardware Ingress Counters
218255 Good, 0 protocol error, 0 header checksum error
0 FC CRC error, 0 iSCSI CRC error, 0 parity error
Software Egress Counters
231117 good frames, 0 bad header cksum, 0 bad FIFO SOP
0 parity error, 0 FC CRC error, 0 timestamp expired error
0 unregistered port index, 0 unknown internal type
0 RDL ok, 0 RDL drop (too big), 0 RDL ttl_1
idle poll count, 0 loopback, 0 FCC PQ, 0 FCC EQ
Flow Control: 0 [0], 0 [1], 0 [2], 0 [3]
Software Ingress Counters
218255 Good frames, 0 header cksum error, 0 FC CRC error
0 iSCSI CRC error, 0 descriptor SOP error, 0 parity error
0 frames soft queued, 0 current Q, 0 max Q, 0 low memory
0 out of memory drop, 0 queue full drop
0 RDL ok, 0 RDL drop (too big)
Flow Control: 0 [0], 0 [1], 0 [2], 0 [3]
This output shows all Fibre Channel frames that ingress or egress from the Gigabit Ethernet port.
Note Use the physical interface, not the subinterface, to display TCP/IP statistics.
Use the show ips stats ip interface gigabitethernet to display and verify IP statistics. This command takes the main Gigabit Ethernet interface as a parameter and returns IP statistics for that interface. See .
Example 18-10 Displays IP Statistics
switch# show ips stats ip interface gigabitethernet 4/1
Internet Protocol Statistics for port GigabitEthernet4/1
168 total received, 168 good, 0 error
0 reassembly required, 0 reassembled ok, 0 dropped after timeout
371 packets sent, 0 outgoing dropped, 0 dropped no route
0 fragments created, 0 cannot fragment
Use the show ips stats tcp interface gigabitethernet to display and verify TCP statistics. This command takes the main ethernet interface as a parameter, and shows TCP stats along with the connection list and TCP state. The detail option shows all information maintained by the interface. See Examples
Example 18-11 Displays TCP Statistics
switch# show ips stats tcp interface gigabitethernet 4/1
TCP Statistics for port GigabitEthernet4/1
Connection Stats
0 active openings, 3 accepts
0 failed attempts, 12 reset received, 3 established
Segment stats
163 received, 355 sent, 0 retransmitted
0 bad segments received, 0 reset sent
TCP Active Connections
Remote Address
0.0.0.0:3260
Example 18-12 Displays Detailed TCP Statistics
switch# show ips stats tcp interface gigabitethernet 4/1 detail
TCP Statistics for port GigabitEthernet4/1
TCP send stats
355 segments, 37760 bytes
222 data, 130 ack only packets
3 control (SYN/FIN/RST), 0 probes, 0 window updates
0 segments retransmitted, 0 bytes
0 retransmitted while on ethernet send queue, 0 packets split
0 delayed acks sent
TCP receive stats
163 segments, 114 data packets in sequence, 6512 bytes in sequence
0 predicted ack, 10 predicted data
0 bad checksum, 0 multi/broadcast, 0 bad offset
0 no memory drops, 0 short segments
0 duplicate bytes, 0 duplicate packets
0 partial duplicate bytes, 0 partial duplicate packets
0 out-of-order bytes, 1 out-of-order packets
0 packet after window, 0 bytes after window
0 packets after close
121 acks, 37764 ack bytes, 0 ack toomuch, 4 duplicate acks
0 ack packets left of snd_una, 0 non-4 byte aligned packets
8 window updates, 0 window probe
30 pcb hash miss, 0 no port, 0 bad SYN, 0 paws drops
TCP Connection Stats
0 attempts, 3 accepts, 3 established
3 closed, 2 drops, 0 conn drops
0 drop in retransmit timeout, 1 drop in keepalive timeout
0 drop in persist drops, 0 connections drained
TCP Miscellaneous Stats
115 segments timed, 121 rtt updated
0 retransmit timeout, 0 persist timeout
12 keepalive timeout, 11 keepalive probes
TCP SACK Stats
0 recovery episodes, 0 data packets, 0 data bytes
0 data packets retransmitted, 0 data bytes retransmitted
0 connections closed, 0 retransmit timeouts
TCP SYN Cache Stats
15 entries, 3 connections completed, 0 entries timed out
0 dropped due to overflow, 12 dropped due to RST
0 dropped due to ICMP unreach, 0 dropped due to bucket overflow
0 abort due to no memory, 0 duplicate SYN, 0 no-route SYN drop
0 hash collisions, 0 retransmitted
TCP Active Connections
Remote Address
0.0.0.0:3260
Use the show ips stats icmp interface gigabitethernet to display and verify IP statistics. This command takes the main ethernet interface as a parameter and returns the ICMP statistics for that interface. See .
Example 18-13 Displays ICMP Statistics
switch# show ips stats icmp interface gigabitethernet 4/1
ICMP Statistics for port GigabitEthernet4/1
5 ICMP messages received
0 ICMP messages dropped due to errors
ICMP input histogram
5 echo request
ICMP output histogram
5 echo reply
Virtual Router Redundancy Protocol (VRRP) and Ethernet PortChannels are two Gigabit Ethernet features that provide high availability for iSCSI and FCIP services.
VRRP provides a redundant alternate path to the Gigabit Ethernet port for iSCSI and FCIP services (see the ).
VRRP provides IP address fail over protection to an alternate Gigabit Ethernet interface so the IP address is always available (see ).
Figure 18-4 VRRP Scenario
In , all members of the VRRP group must be IP storage Gigabit Ethernet ports. VRRP group members can be one or more of the following interfaces:
•One or more interfaces in the same IPS module
•Interfaces across IPS modules in one switch
•Interfaces across IPS modules in different switches
•Gigabit Ethernet subinterfaces
•Ethernet PortChannels
•Subinterfaces
To configure VRRP for Gigabit Ethernet interfaces, follow these steps:
Step 1 
switch1# config terminal
switch1(config)#
Enters configuration mode.
Step 2 
switch(config)# interface
gigabitethernet 2/2
switch(config-if)#
Enters the interface configuration mode on the Gigabit Ethernet interface (slot2, port 2).
Step 3 
switch(config-if)# ip address
10.1.1.10 255.255.255.0
Enters the IP address (10.1.1.10) and IP mask (255.255.255.0) for the Gigabit Ethernet interface.
Step 4 
switch(config-if)# no shutdown
Enables the selected interface.
Step 5 
switch(config-if)# vrrp 100
switch(config-if-vrrp)
Creates a VR ID 100.
Step 6 
switch(config-if-vrrp)# address
10.1.1.100
Configures the virtual IP address (10.1.1.100) for the selected VRRP group (identified by the VR ID).
Note The virtual IP address must be in the same subnet as the IP address of the Gigabit Ethernet interface. All members of the VRRP group must configure the same virtual IP address.
Step 7 
switch(config-if-vrrp)# priority 10
Configures the priority for the selected interface within this VRRP group.
Note The interface with the highest priority is selected as the master.
Step 8 
switch(config-if-vrrp)# no shutdown
Enables the VRRP protocol on the selected interface.
Note The VRRP preempt option is not supported on IP storage Gigabit Ethernet interfaces. However, if the virtual IP address is also the IP address for the interface, then preemption is implicitly applied.
Ethernet PortChannels refer to the aggregation of multiple physical Gigabit Ethernet interfaces into one logical Ethernet interface to provide link redundancy and, in some cases, higher aggregated bandwidth and load balancing.
The data traffic from one TCP connection always travels on the same physical links. An Ethernet switch connecting to the MDS Gigabit Ethernet port can implement load balancing based on its IP address, its source-destination MAC address, or its IP and port. If Ethernet-based load balancing cannot be implemented for iSCSI scenarios based on the IP and port, multiple iSCSI initiators are required to take advantage of the Ethernet PortChannel feature.
Note The Cisco Ethernet switch's PortChannel should be configured as a static PortChannel, and not the default 802.3aa protocol.
Ethernet PortChannels can only aggregate two physical interfaces that are adjacent to each other on a given IPS module (see ).
Note PortChannel members must be one of these combinations: ports 1-2, ports 3-4, ports 5-6, or ports 7-8.
Figure 18-5 Ethernet PortChannel Scenario
In , Gigabit Ethernet ports 3 and 4 in slot 9 are aggregated into an Ethernet PortChannel.
Note All FCIP data traffic for one FCIP link is carried on one TCP connection. Consequently, the aggregated bandwidth will be one Gbps for that FCIP link.
PortChannel configuration specified in
also apply to Ethernet PortChannel configurations.
PortChannel interfaces provide configuration options for both Gigabit Ethernet and Fibre Channel. However, based on the PortChannel membership, only Gigabit Ethernet parameters or Fibre Channel parameters are applicable.
To configure Ethernet PortChannels, follow these steps:
Step 1 
switch1# config terminal
switch1(config)#
Enters configuration mode.
Step 2 
switch(config)# interface port-channel 10
switch(config-if)#
Configures the specified PortChannel (10).
Step 3 
switch(config-if)# ip address 10.1.1.1
255.255.255.0
Enters the IP address (10.1.1.1) and IP mask (255.255.255.0) for the PortChannel.
Note A PortChannel does not have any members when first created.
Step 4 
switch(config-if)# no shutdown
Enables the interface.
Step 5 
switch(config)# interface gigabitethernet 9/3
switch(config-if)#
Configures the specified Gigabit Ethernet interface (slot 9, port 3).
Step 6 
switch(config-if)# channel-group 10
gigabitethernet 9/3 added to port-channel 10
and disabled
please do the same operation on the switch at
the other end of the port-channel, then do &no
shutdown& at both ends to bring them up
switch(config-if)#
Adds Gigabit Ethernet interfaces 9/3 to channel group 10. If channel group 10 does not exist, it is created. The port is shut down.
Step 7 
switch(config-if)# no shutdown
Enables the selected interface.
Step 8 
switch(config)# interface gigabitethernet 9/4
switch(config-if)#
Configures the specified Gigabit Ethernet interface (slot 9, port 4).
Step 9 
switch(config-if)# channel-group 10
gigabitethernet 9/4 added to port-channel 10
and disabled
please do the same operation on the switch at
the other end of the port-channel, then do &no
shutdown& at both ends to bring them up
Adds Gigabit Ethernet interfaces 9/4 to channel group 10. The port is shut down.
Step 10 
switch(config-if)# no shutdown
Enables the selected interface.
Note Gigabit Ethernet interfaces cannot be added to a PortChannel if one of the following cases apply:- if the interface already has an IP address assigned, or - if subinterfaces are configured on that interface.
The Cisco Discovery Protocol (CDP) is supported on the management Ethernet interface on the supervisor module and the Gigabit Ethernet interface on the IPS module. See the .
IPS core dumps are different from the system's kernel core dumps for other modules. When the IPS module's operating system (OS) unexpectedly resets, it is sometimes useful to obtain a full copy of the memory image (called a IPS core dump) to identify the cause of the reset. Under that condition, the IPS module sends the core dump to the supervisor module for storage. Core dumps take up significant space and hence the level of what gets stored can be configured using one of the two options:
•Partial core dumps (default)—Each partial core dump consists of four parts (four files).
•Full core dumps—Each full core dump consists of 75 parts (75 files). This dump cannot be saved on the supervisor module due to its large space requirement. If you choose this option, then you must configure an external TFTP server using the system cores tftp: command (see ).
This section includes the following topics:
The Fibre Channel over IP Protocol (FCIP) is a tunneling protocol that connects geographically distributed Fibre Channel storage area networks (SAN islands) transparently over IP local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). See .
Figure 18-6 Fibre Channel SANs Connected by FCIP
FCIP uses Transmission Control Protocol (TCP) as a network layer transport.
Note For more information about FCIP protocols, refer to the IETF standards for IP storage at . Also refer to Fibre Channel standards for switch backbone connection at
(see FC-BB-2).
To configure the IPS module for FCIP, you should have a basic understanding of the following concepts:
provides the internal model of FCIP with respect to Fibre Channel inter switch links (ISLs) and Cisco's enhanced ISLs (EISLs). See the .
FCIP defines virtual E (VE) ports, which behave exactly like standard Fibre Channel E ports, except that the transport in this case is FCIP instead of Fibre Channel. The only requirement is for the other end of the VE port to be another VE port.
A virtual ISL is established over a FCIP link and transports Fibre Channel traffic. Each associated virtual ISL looks like a Fibre Channel ISL with either an E port or a TE port at each end (see ).
Figure 18-7 FCIP Links and Virtual ISLs
FCIP links consist of one or more TCP connections between two FCIP link end points. Each link carries encapsulated Fibre Channel frames.
When the FCIP link comes up, the VE ports at both ends of the FCIP link create a virtual Fibre Channel (E)ISL and initiate the E port protocol to bring up the (E)ISL.
By default, the FCIP feature on any Cisco MDS 9000 Family switch creates two TCP connections for each FCIP link.
•One connection is used for data frames.
•The second connection is used only for Fibre Channel control frames, i.e. switch-to-switch protocol frames (all Class F) frames. This arrangement is used to provide low latency for all control frames.
To enable FCIP on the IPS module, a FCIP profile and FCIP interface (interface FCIP) must be configured.
The FCIP link is established between two peers, the VE port initialization behavior is identical to a normal E port. This behavior is independent of the link being FCIP or pure Fibre Channel, and is based on the E port discovery process (ELP, ESC).
Once the FCIP link is established, the VE port behavior is identical to E port behavior for all inter-switch communication (including domain management, zones, and VSANs). At the Fibre Channel layer, all VE and E port operations are identical.
The FCIP profile contains information about local IP address and TCP parameters. The profile defines the following information:
•the local connection points (IP address and TCP port number)
•the behavior of the underlying TCP connections for all FCIP links that use this profile
The FCIP profile's local IP address determines the Gigabit Ethernet port where the FCIP links terminates (see ).
Figure 18-8 FCIP Profile and FCIP Links
The FCIP interface is the local end point of the FCIP link and a VE port interface. All the FCIP and E port parameters are configured in context to the FCIP interface.
The FCIP parameters consist of the following:
•The FCIP profile determines which Gigabit Ethernet port terminates FCIP links and defines the TCP connection behavior.
•Peer information.
•Number of TCP connections for the FCIP link.
•E port parameters—trunking mode and trunk allowed VSAN list.
To configure a FCIP link, perform this procedure on both switches.
Step 1 Configure the Gigabit Ethernet interface.
Step 2 Create a FCIP profile, assign the Gigabit Ethernet interface's IP address to the profile. See the .
Step 3 Create a FCIP interface, assign the profile to the interface. See the .
Step 4 Configure the peer IP address for the FCIP interface. See the .
Step 5 Enable the interface. See the .
To create a FCIP profile, you must assign a local IP address of a Gigabit Ethernet interface or subinterface to the FCIP profile (see ).
Figure 18-9 Assigning Profiles to Each Gigabit Ethernet Interface
To create a FCIP profile in switch 1, follow these steps:
Step 1 
switch1# config terminal
switch1(config)#
Enters configuration mode.
Step 2 
switch1(config)# fcip profile 10
switch1(config-profile)#
Creates a profile for the FCIP connection. The valid range is from 1 to 255.
Step 3 
switch1(config-profile)# ip address
10.100.1.25
Associates the profile (10) with the local IP address of the Gigabit Ethernet interface (3/1).
To assign FCIP profile in switch 2, follow these steps:
Step 1 
switch2# config terminal
switch2(config)#
Enters configuration mode.
Step 2 
switch2(config)# fcip profile 20
switch2(config-profile)#
Creates a profile for the FCIP connection.
Step 3 
switch2(config-profile)# ip address 10.1.1.1
Associates the profile (20) with the local IP address of the Gigabit Ethernet interface.
When two FCIP link end points are created, a FCIP link is established between the two IPS modules.To create a FCIP link, assign a profile to the FCIP interface and configure the peer information. The peer IP switch information initiates (creates) a FCIP link to that peer switch (see ).
Figure 18-10 Assigning Profiles to Each Gigabit Ethernet Interface
To create a FCIP link end point in switch 1, follow these steps:
Step 1 
switch1# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch1(config)# interface fcip 51
switch1(config-if)#
Creates a FCIP interface (51).
Step 3 
switch1(config-if)# use-profile 10
Assigns the profile (10) to the FCIP interface.
Step 4 
switch1(config-if)# peer-info ipaddr 10.1.1.1
Assigns the peer IP address information (10.1.1.1 for switch 2) to the FCIP interface
Step 5 
switch1(config-if)# no shutdown
Enables the interface.
To create a FCIP link end point in switch 2, follow these steps:
Step 1 
switch2# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch2(config)# interface fcip 52
switch2(config-if)#
Creates a FCIP interface (52).
Step 3 
switch2(config-if)# use-profile 20
Binds the profile (20) to the FCIP interface.
Step 4 
switch2(config-if)# peer-info ip
address 10.100.1.25
Assigns the peer IP address information (10.100.1.25 for switch 1) to the FCIP interface
Step 5 
switch1(config-if)# no shutdown
Enables the interface.
A basic FCIP configuration uses the local IP address to configure the FCIP profile. In addition to the local IP address and the local port, you can specify other TCP parameters as part of the FCIP profile configuration.
FCIP configuration options can be accessed from the switch(config-profile)# submode prompt.
To enter the switch(config-profile)# prompt, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# fcip profile 20
switch(config-profile)#
Creates the profile (if it does not already exist). The valid range is from 1 to 255.
The default TCP port for FCIP is 3225. You can change this port using the port command.
To change the default FCIP port number (3225), follow these steps:
Step 1 
switch(config-profile)# port 5000
Associates the profile with the local port number (5000).
switch(config-profile)# no port
Reverts to the default 3225 port.
This section provides details on the TCP parameters that can be configured to control TCP behavior in a switch. The following TCP parameters can be configured.
The tcp minimum-retransmit-time option controls the minimum amount of time TCP waits before retransmitting. By default, this value is 300 milliseconds.
To configure the minimum retransmit time, follow these steps:
Step 1 
switch(config-profile)# tcp
min-retransmit-time 500
Specifies the minimum TCP retransmit time for the TCP connection in milliseconds (500). The default is 300 milliseconds and the range is from 250 to 5000 milliseconds.
switch(config-profile)# no tcp
min-retransmit-time 500
Reverts the minimum TCP retransmit time to the factory default of 300 milliseconds.
The tcp keepalive-timeout option enables you to configure the interval between which the TCP connection verifies if the FCIP link is functioning. This ensures that a FCIP link failure is detected quickly even when there is no traffic.
If the TCP connection is idle for more than the specified transmission time, then keepalive timeout packets are sent to ensure that the connection is active. This command can be used to detect FCIP link failures.
The first interval during which the connection is idle is 60 seconds (default). When the connection is idle for 60 seconds, 8 keepalive probes are sent at 1-second intervals. If no response is received for these 8 probes and the connection remains idle throughout, that FCIP link is automatically closed.
Note Only the first interval (during which the connection is idle) can be changed from the default of 60 seconds. This interval is identified using the keepalive-timeout option. The valid range is from 1 to 7200 seconds.
To configure the keep alive timeout, follow these steps:
Step 1 
switch(config-profile)# tcp
keepalive-timeout 120
Specifies the keepalive timeout interval for the TCP connection in seconds (120). The default is 60 seconds. The range is from 1 to 7200 seconds.
switch(config-profile)# no tcp
keepalive-timeout 120
Reverts the keepalive-timeout to 60 seconds.
The tcp max-retransmissions option specifies the maximum number of times a packet is retransmitted before TCP decides to close the connection.
To configure maximum retransmissions, follow these steps:
Step 1 
switch(config-profile)# tcp
max-retransmissions 6
Specifies the maximum number of retransmissions (6). The default is 4 and the range is from 1 to 8 retransmissions.
switch(config-profile)# no tcp
max-retransmissions 6
Reverts to the default of 4 retransmissions.
Path MTU (PMTU) is the minimum MTU on the IP network between the two end points of the FCIP link. PMTU discovery is a mechanism by which TCP learns of the PMTU dynamically and adjusts the maximum TCP segment accordingly (RFC 1191).
By default, PMTU discovery is enabled on all switches with a default timeout of 3600 seconds. If TCP reduces the size of the max segment because of PMTU change, the reset-timeout specifies the time after which TCP tries the original MTU.
To configure PMTU, follow these steps:
Step 1 
switch(config-profile)# no tcp pmtu-enable
Disables PMTU discovery.
switch(config-profile)# tcp pmtu-enable
Enables (default) PMTU discovery with the default value of 3600 seconds.
switch(config-profile)# tcp pmtu-enable
reset-timeout 90
Specifies the PMTU reset timeout to 90 seconds. The default is 3600 seconds and the range is from 60 to 3600 seconds.
switch(config-profile)# no tcp pmtu-enable
reset-timeout 600
Leaves the PMTU in an enabled state but changes the timeout to the default of 3600 seconds.
TCP may experience poor performance when multiple packets are lost within one window. With the limited information available from cumulative acknowledgments, a TCP sender can only learn about a single lost packet per round trip time. A selective acknowledgment (SACK) mechanism helps overcome the limitations of multiple lost packets during a TCP transmission.
The receiving TCP sends back SACK advertisements to the sender. The sender can then retransmit only the missing data segments. By default, SACK is enabled on Cisco MDS 9000 Family switches.
To configure SACK, follow these steps:
Step 1 
switch(config-profile)# no tcp sack-enable
Disables SACK.
switch(config-profile)# tcp sack-enable
Enables SACK (default).
Window Management
The optimal TCP window size is computed using three options.
•The maximum-bandwidth option configures the maximum available end-to-end bandwidth in the path (900 Mbps in the configuration example).
•The minimum-available-bandwidth option configures the minimum slow start threshold.
•The round-trip-time option is the estimated round trip time across the IP network to reach the FCIP peer end point (10 milliseconds in the configuration example). If the round-trip-time value is under-estimated, the TCP window size will be too small to reach the maximum available bandwidth. If the round-trip-time is overestimated, the TCP window size will be too big. If the maximum available bandwidth is correct, this will cause increase in latency and potential packet drop in the network but will not affect the speed.
The maximum-bandwidth option and the round-trip-time option together determine the window size.
The minimum-available-bandwidth option and the round-trip-time option together determine the threshold below which TCP aggressively increases its size. After it reaches the threshold the software uses standard TCP rules to reach the maximum available bandwidth. The defaults are max-bandwidth = 1G, min-available-bandwidth = 2 Mbps, and round-trip-time is 10ms
To configure window management, follow these steps:
Step 1 
switch(config-profile)# tcp max-bandwidth-mbps
900 min-available-bandwidth-mbps 300
round-trip-time-ms 10
Configures the maximum available bandwidth at 900 Mbps, the minimum slow start threshold as 300 Mbps, and the round trip time as 10 milliseconds.
switch(config-profile)# no tcp max-bandwidth-mbps
900 min-available-bandwidth-mbps 300
round-trip-time-ms 10
Reverts to the factory defaults. The defaults are max-bandwidth = 1G, min-available-bandwidth = 2 Mbps and round-trip-time is 10ms.
switch(config-profile)# tcp max-bandwidth-kbps
2000 min-available-bandwidth-kbps 2000
round-trip-time-us 200
Configures the maximum available bandwidth at 2000 Kbps, the minimum slow start threshold as 2000 Kbps, and the round trip time as 200 microseconds.
The send-buffer-size option defines the required additional buffering—beyond the normal send window size —that TCP allows before flow controlling the switch's egress path for the FCIP interface. The default buffer size is 0 KB.
To set the buffer size, follow these steps:
Step 1 
switch(config-profile)# tcp
send-buffer-size 5000
Configure the advertised buffer size to 5000 KB. The valid range is from 0 to 8192 KB.
switch(config-profile)# no tcp
send-buffer-size 5000
Reverts the switch to its factory default (0 KB).
The qos control option specifies the differentiated services code point (DSCP) value to mark all IP packets (type of service—TOS field in the IP header).
•The control DSCP value applies to all FCIP frames in the control TCP connection.
•The data DSCP value applies to all FCIP frames in the data connection.
If the FCIP link has only one TCP connection, that data DSCP value is applied to all packets in that connection.
To set the control values, follow these steps:
Step 1 
switch(config-profile)# tcp qos control
Configures the control TCP connection and data connection to mark all packets on that DSCP value.
switch(config-profile)# no tcp qos
control 3 data 5
Reverts the switch to its factory default (no packets).
By configuring the congestion window monitoring (CWM) option, you can influence the rate at which TCP ramps up the transmitted bandwidth after an idle period as listed below:
•If the traffic is transmitted in burst sizes that are smaller than the configured CWM value, you can send the whole traffic burst immediately, provided no drops occurred.
•If the traffic burst is larger than the configured CWM value, some traffic will not be sent immediately.
•If the end-to-end path between the two Cisco MDS 9000 Family switches is 1G, you can set the maximum burst size.
•If the router connecting to the IPS port does not have sufficient buffering, you can use the smallest available value to decrease the burst size.
By default the tcp cwm option is enabled and the default burst size is 10KB.
Tip We recommend that this feature remain enabled to realize optimal performance.
To change the CWM defaults, follow these steps:
Step 1 
switch(config-profile)# no tcp cwm
Disables congestion monitoring.
switch(config-profile)# tcp cwm
Enables congestion monitoring and sets the defaults burst size at 10 KB.
switch(config-profile)# tcp cwm burstsize 30
Changes the burst size to 30 KB. The valid range is from 10 to 100 KB.
switch(config-profile)# no cp cwm burstsize 25
Leaves the CWM feature in an enabled state but changes the burst size to the default of 10 KB.
You can establish connection to a peer by configuring one or more of the following options for the FCIP interface. To do so, you must first create the interface and enter the config-if submode.
To enter the config-if submode, follow these steps:
Step 1 
switch# config terminal
Enters configuration mode.
Step 2 
switch(config)# interface fcip 100
Creates a FCIP interface (100).
To establish a FCIP link with the peer, you can use one of two options:
•—used to configure both ends of the FCIP link. Optionally, you can also use the peer TCP port along with the IP address.
•—used to configure one end of the FCIP link when security gateways are present in the IP network. Optionally, you can also use the port and profile ID along with the IP address.
The basic FCIP configuration uses the peer's IP address to configure the peer information. You can also specify the peer's port number to configure the peer information. If you do not specify a port, the default 3225 port number is used to establish connection.
To assign the peer information based on the IP address, port number, or a profile ID, follow these steps:
Step 1 
switch(config-if)# peer-info ipaddr 10.1.1.1
Assigns an IP address to configure the peer information. Since no port is specified, the default port number, 3225, is used.
switch(config-if)# no peer-info ipaddr
Deletes the assigned peer port information.
Step 2 
switch(config-if)# peer-info ipaddr 10.1.1.1
Assigns the IP address and sets the peer TCP port to 3000. The valid port number range is from 0 to 65535.
switch(config-if)# no peer-info ipaddr
10.1.1.1 port 2000#
Deletes the assigned peer port information.
Step 3 
switch(config-if)# no shutdown
Enables the interface.
You can alternatively establish a FCIP link with a peer using an optional protocol called special frames. You can enable or disable the special-frame option. On the peer side, the special-frame option must be enabled in order to establish the FCIP link. When the special-frame option is enabled, the peer IP address (and optionally the port or the profile ID) only needs to be configured on one end of the link. Once the connection is established, a special frame is exchanged to discover and authenticate the link.
By default, the special frame feature is disabled.
Note Refer to the Fibre Channel IP standards for further information on special frames.
To enable special frames, follow these steps:
Step 1 
switch(config-if)# special-frame peer-wwn
12:12:34:45:ab:bc:cd:00
Enables special frames and sets the peer WWN as specified.
Note The peer WWN is the WWN of the peer switch. Use the show wwn switch command to obtain the peer WWN .
switch(config-if)# no special-frame peer-wwn
12:12:34:45:ab:bc:cd:00
Disables special frames (default).
Step 2 
switch(config-if)# special-frame peer-wwn
12:12:34:45:ab:bc:cd:00 peer profile-id 155
Enables special frames and sets the peer WWN as specified by the profile ID (155).
switch(config-if)# no special-frame peer-wwn
12:12:34:45:ab:bc:cd:00 peer profile-id 155
Disables special frames (default).
Step 3 
switch(config-if)# no shutdown
Enables the interface.
Use the passive-mode option to configure the required mode for initiating an IP connection. By default, active mode is enabled to actively attempt an IP connection.
If you enable the passive mode, the switch does not initiate a TCP connection and merely waits for the peer to connect to it.
Note Ensure that both ends of the FCIP link are not configured as passive mode. If both ends are configured as passive, the connection will not be initiated.
To enable the passive mode, follow these steps:
Step 1 
switch(config-if)# passive-mode
Enable passive mode while attempting a TCP connection.
switch(config-if)# no passive-mode
Reverts to the factory set default of using the active mode while attempting the TCP connection.
Step 2 
switch(config-if)# no shutdown
Enables the interface.
Use the tcp-connection option to specify the number of TCP connections from a FCIP link. By default, the switch tries two (2) TCP connections for each FCIP link. You can configure 1 or 2 TCP connections.
For example, the Cisco PA-FC-1G Fibre Channel port adapter which has only 1 (one) TCP connection interoperates with any switch in the Cisco MDS 9000 Family. One TCP connection is within the specified limit and you can change the configuration on the switch using the tcp-connection 1 command. If the peer initiates one TCP connection, and your MDS switch is configured for two TCP connections, the software handles it gracefully and moves on with just one connection.
To specify the TCP connection attempts, follow these steps:
Step 1 
switch(config-if)# tcp-connection 1
Specifies the number of TCP connections. Two (2) is the default and the maximum number of TCP connection attempts.
switch(config-if)# no tcp-connection 1
Reverts to the factory set default of two attempts.
Step 2 
switch(config-if)# no shutdown
Enables the interface.
Use the time-stamp option to enable or disable FCIP time stamps on a packet. The time stamp option instructs the switch to discard packets that are outside the specified time. By default, the time-stamp option is disabled.
The acceptable-diff option specifies the time range within which packets can be accepted. If the packet arrived within the range specified by this option, the packet is accepted. Otherwise, it is dropped. By default if a packet arrives within a 1000 millisecond interval (+ or -1000 milliseconds), that packet is accepted.
Note If the time-stamp option is enabled, be sure to configure NTP on both switches (see the ).
To enable or disable the time-stamp option, follow these steps:
Step 1 
switch(config-if)# time-stamp
Please enable NTP with a common time
source on both MDS Switches that are
on either side of the FCIP link
Enables time stamp checking for received packets with a default acceptable time difference of 1000 milliseconds.
switch(config-if)# no time-stamp
Disables (default) time stamps.
Step 2 
switch(config-if)# time-stamp
acceptable-diff 4000
Configures the acceptable time within which a packet is accepted. The default difference is a 1000 millisecond interval from the network time. The valid range is from 1 to 60,000 milliseconds.
switch(config-if)# no time-stamp
acceptable-diff 500
Deletes the configured time difference and reverts the difference to factory defaults.
Step 3 
switch(config-if)# no shutdown
Enables the interface.
While E ports typically interconnect Fibre Channel switches, some SAN extender devices, such as Cisco's PA-FC-1G Fibre Channel port adapter and the SN 5428-2 storage router, implement a bridge port model to connect geographically dispersed fabrics. This model uses B port as described in the T11 Standard FC-BB-2.
depicts a typical SAN extension over an IP network.
Figure 18-11 FCIP B Port and Fibre Channel E Port
B ports bridge Fibre Channel traffic from one E port to a remote E port without participating in fabric-related activities such as principal switch election, Domain ID assignment, and Fibre Channel routing (FSPF). For example, Class F traffic entering a SAN extender does not interact with the B port. The traffic is transparently propagated (bridged) over a WAN interface before exiting the remote B port. This bridge results in both E ports exchanging Class F information which ultimately leads to normal ISL behavior such as fabric merging and routing.
FCIP links between B port SAN extenders do not exchange the same information as FCIP links between E ports, and are therefore incompatible. This is reflected by the terminology used in FC-BB-2: while VE ports establish a virtual ISL over a FCIP link, B ports use a B access ISL.
The IPS module supports FCIP links that originate from a B port SAN extender device by implementing the B access ISL protocol on a Gigabit Ethernet interface. Internally, the corresponding virtual B port connects to an virtual E port which completes the end-to-end E port connectivity requirement (see ).
Figure 18-12 FCIP Link Terminating in a B Port Mode
The B port feature in the IPS module allows remote B port SAN extenders to communicate directly with a Cisco MDS 9000 Family switch, therefore eliminating the need for local bridge devices.
When a FCIP peer is a SAN extender device that only supports Fibre Channel B ports, you need to enable the B port mode for the FCIP link. When a B port is enabled, the E port functionality is also enabled and they coexist. If the B port is disabled, the E port functionality remains enabled.
To enable B port mode, follow these steps:
Step 1 
switch(config-if)# bport
Enables B port mode on the FCIP interface.
switch(config-if)# no bport
Reverts to E port mode on the FCIP interface (default).
Step 2 
switch(config-if)# bport-keepalive
Enables the reception of keepalive responses sent by a remote peer.
Step 3 
switch(config-if)# no bport-keepalive
Disables the reception of keepalive responses sent by a remote peer (default).
•VSANs (see )
–FCIP interfaces can be a member of any VSAN.
•Trunk mode (see )
–Trunk mode can be configured.
–Trunk allowed VSANs can be configured
•PortChannels (see )
–Multiple FCIP links can be bundled into a Fibre Channel PortChannel.
–FCIP links and Fibre Channel links cannot be combined in one PortChannel.
•FSPF (see )
•Fibre Channel domains (fcdomains—see )
•Zone merge (see )
–Importing the zone database from the adjacent switch.
–Exporting the zone database from the adjacent switch.
Use the show interface commands to view the summary, counter, description, and status of the FCIP link. Use the output of these commands to verify the administration mode, the interface status, the operational mode, the related VSAN ID, and the profile used. See Examples
Example 18-14 Displays the FCIP Interface
switch# show interface fcip 3
fcip3 is trunking
Hardware is GigabitEthernet
Port WWN is 20:ca:00:05:30:00:07:1e
Peer port WWN is 20:ca:00:00:53:00:18:1e
Admin port mode is auto, trunk mode is on
Port mode is TE
Trunk vsans (allowed active) (1,10)
Trunk vsans (operational)
Trunk vsans (up)
Trunk vsans (isolated)
Trunk vsans (initializing)
Using Profile id 3 (interface GigabitEthernet4/3)
Peer Information
Peer Internet address is 43.1.1.1 and port is 3225
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
B-port mode disabled
TCP Connection Information
2 Active TCP connections
Control connection: Local 43.1.1.2:3225, Remote 43.1.1.1:65532
Data connection: Local 43.1.1.2:3225, Remote 43.1.1.1:65534
30 Attempts for active connections, 0 close of connections
TCP Parameters
Path MTU 1500 bytes
Current retransmission timeout is 300 ms
Round trip time: Smoothed 10 ms, Variance: 5
Advertised window: Current: 122 KB, Maximum: 122 KB, Scale: 1
Peer receive window: Current: 114 KB, Maximum: 114 KB, Scale: 1
Congestion window: Current: 2 KB, Slow start threshold: 1048560 KB
5 minutes input rate 64 bits/sec, 8 bytes/sec, 0 frames/sec
5 minutes output rate 64 bits/sec, 8 bytes/sec, 0 frames/sec
808 frames input, 75268 bytes
808 Class F frames input, 75268 bytes
0 Class 2/3 frames input, 0 bytes
0 Error frames timestamp error 0
806 frames output, 74712 bytes
806 Class F frames output, 74712 bytes
0 Class 2/3 frames output, 0 bytes
0 Error frames 0 reass frames
Example 18-15 Displays Detailed FCIP Interface Counter Information
switch# show interface fcip 3 counters
TCP Connection Information
2 Active TCP connections
Control connection: Local 43.1.1.2:3225, Remote 43.1.1.1:65532
Data connection: Local 43.1.1.2:3225, Remote 43.1.1.1:65534
30 Attempts for active connections, 0 close of connections
TCP Parameters
Path MTU 1500 bytes
Current retransmission timeout is 300 ms
Round trip time: Smoothed 10 ms, Variance: 5
Advertised window: Current: 122 KB, Maximum: 122 KB, Scale: 1
Peer receive window: Current: 114 KB, Maximum: 114 KB, Scale: 1
Congestion window: Current: 2 KB, Slow start threshold: 1048560 KB
5 minutes input rate 64 bits/sec, 8 bytes/sec, 0 frames/sec
5 minutes output rate 64 bits/sec, 8 bytes/sec, 0 frames/sec
814 frames input, 75820 bytes
814 Class F frames input, 75820 bytes
0 Class 2/3 frames input, 0 bytes
0 Error frames timestamp error 0
812 frames output, 75264 bytes
812 Class F frames output, 75264 bytes
0 Class 2/3 frames output, 0 bytes
0 Error frames 0 reass frames
Example 18-16 Displays Brief FCIP Interface Counter Information
switch# show interface fcip 3 counters brief
-------------------------------------------------------------------------------
Input (rate is 5 min avg)
Output (rate is 5 min avg)
-----------------------------
-----------------------------
-------------------------------------------------------------------------------
Example 18-17 Displays the FCIP Interface Description
switch# show interface fcip 51 description
Sample FCIP interface
Example 18-18 Displays FCIP Profiles
switch# show fcip profile
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
10.10.100.150
10.10.100.150
Example 18-19 Displays the Specified FCIP Profile Information
switch# show fcip profile 7
FCIP Profile 7
Internet Address is 47.1.1.2 (interface GigabitEthernet4/7)
Listen Port is 3225
TCP parameters
SACK is disabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 300 ms
Maximum number of re-transmissions is 4
Send buffer size is 0 KB
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 15000 kbps
Estimated round trip time is 1000 usec
The following high availability solutions are available for FCIP configurations:
provides an example of a PortChannel-based load balancing configuration. To perform this configuration, you need two IP addresses on each SAN island. This solution addresses link failures.
Figure 18-13 PortChannel Based Load Balancing
The following characteristics set Fibre Channel PortChannel solutions apart from other solutions:
•The entire bundle is one logical (E)ISL link.
•All FCIP links in the PortChannel should be across the same two switches.
•The Fibre Channel traffic is load balanced across the FCIP links in the PortChannel.
displays a FPSF-based load balancing configuration example. This configuration requires two IP addresses on each SAN island, and addresses IP and FCIP link failures.
Figure 18-14 FSPF-Based Load Balancing
The following characteristics set FSPF solutions apart from other solutions:
•Each FCIP link is a separate (E)ISL.
•The FCIP links can connect to different switches across two SAN islands.
•The Fibre Channel traffic is load balanced across the FCIP link.
displays a VRRP-based high availability FCIP configuration example. This configuration, requires at least two physical Gigabit Ethernet ports connected to the Ethernet switch on the island where you need to implement high availability using VRRP.
Figure 18-15 VRRP-Based High Availability
The following characteristics set VRRP solutions apart from other solutions:
•If the active VRRP port fails, the standby VRRP port takes over the VRRP IP address.
•When the VRRP switchover happens, the FCIP link automatically disconnects and reconnects.
•This configuration has only one FCIP (E)ISL link.
displays a Ethernet PortChannel-based high availability FCIP example. This solution addresses the problem caused by individual Gigabit Ethernet link failures.
Figure 18-16 Ethernet PortChannel-Based High Availability
The following characteristics set Ethernet PortChannel solutions apart from other solutions:
•The Gigabit Ethernet link level redundancy ensures a transparent failover if one of the Gigabit Ethernet links fails.
•Two Gigabit Ethernet ports in one Ethernet PortChannel appears like one logical Gigabit Ethernet link.
•The FCIP link stays up during the failover.
Ethernet PortChannels offer Ethernet-level redundancy, Fibre Channel PortChannels offer (E)ISL-level redundancy. FCIP is unaware of any Ethernet PortChannels or Fibre Channel PortChannels. Fibre Channel PortChannels are unaware of any Ethernet PortChannels, and there is no mapping between the two (see ).
Figure 18-17 PortChannels at the Fibre Channel and Ethernet Levels
To configure Fibre Channel PortChannels, see
To configure Ethernet PortChannels, refer to the .
This section includes the following topics:
The IPS module provides transparent SCSI routing. IP hosts using iSCSI protocol can transparently access iSCSI targets on the Fibre Channel network.
provides an example of a typical configuration of iSCSI hosts with access to a Fibre Channel SAN.
Figure 18-18 Typical IP to Fibre Channel SAN Configuration
The IPS module enables you to create virtual iSCSI targets and maps them to physical Fibre Channel targets available in the Fibre Channel SAN. It presents the Fibre Channel targets to IP hosts as if the physical targets were attached to the IP network (see ).
Figure 18-19 iSCSI View
In conjunction with presenting Fibre Channel targets to iSCSI hosts, the iSCSI feature presents each iSCSI host as a Fibre Channel host, i.e. Host Bus Adaptor (HBA) to the Fibre Channel storage device. The storage device responds to each IP host as if it were a Fibre Channel host connected to the Fibre Channel network (see ).
Figure 18-20 Fibre Channel SAN View
Note Refer to the IETF standards for IP storage at , for information on the iSCSI protocol.
The iSCSI feature consists of routing iSCSI requests and responses between hosts in an IP network and Fibre Channel storage devices in the Fibre Channel SAN that are accessible from any Fibre Channel interface of the Cisco MDS 9000 Family switch (see ).
Figure 18-21 Routing iSCSI Requests and Responses for Transparent iSCSI Routing
Each iSCSI host that requires access to storage via the IPS module needs to have a compatible iSCSI driver installed. (The CCO website at provides a list of compatible drivers). Using iSCSI protocol, the iSCSI driver allows an iSCSI host to transport SCSI requests and responses over an IP network. From the host operating system perspective, the iSCSI driver appears to be a SCSI transport driver similar to a Fibre Channel driver for a peripheral channel in the host. From the storage device perspective, each IP host appears as a Fibre Channel host.
Routing SCSI from the IP host to the Fibre Channel storage device consists of the following main actions (see ):
•Transporting iSCSI requests and responses over an IP network between hosts and the IPS module.
•Routing SCSI requests and responses between hosts on an IP network and the Fibre Channel storage device (converting iSCSI to FCP and vice versa). This routing is performed by the IPS module.
•Transporting FCP requests or responses between the IPS module and Fibre Channel storage devices.
Figure 18-22 Transparent SCSI Routing Actions
Note FCP (the Fibre Channel equivalent of iSCSI) carries SCSI commands over a Fibre Channel SAN.
The IPS module presents physical Fibre Channel targets as iSCSI targets allowing them to be accessed by iSCSI hosts. It does this in one of two ways:
•—used if all logical units (LUs) in all Fibre Channel storage targets are made available to iSCSI hosts (subject to VSAN and zoning).
•—used if iSCSI hosts are restricted to subsets of LUs in the Fibre Channel targets and additional iSCSI access control is needed (see the ). Also, static import allows automatic failover if the Fibre Channel targets' LU is reached by redundant Fibre Channel ports (see the ).
Note The IPS module does not import Fibre Channel targets to iSCSI by default. Either dynamic or static mapping must be configured before the IPS module makes Fibre Channel targets available to iSCSI initiators. When both are configured, statically mapped Fibre Channel targets have a configured name. Un mapped targets are advertised with the name created by the conventions explained in this section.
To enable dynamic importing of Fibre Channel targets into iSCSI, use the iscsi import target fc command.
The IPS module maps each physical Fibre Channel target port as one iSCSI target. That is, all LU accessible via the physical storage target port are available as iSCSI LUs with the same LU number (LUN) as in the storage target.
For example, if an iSCSI target was created for Fibre Channel target port with pWWN 31:00:11:22:33:44:55:66 and that pWWN contains LUN 0 through 2, those LUNs would become available to an IP host as LUNs 0 through 2 as well.
The iSCSI target node name is created automatically using the iSCSI I qualified name (IQN) format. The IPS module creates an IQN formatted iSCSI node name using the following conventions:
•IPS ports that are not part of a VRRP group use this format:
iqn..cisco:05.&mgmt-ip-address&.&slot#&-&port#&-&sub-intf#&.&Target-pWWN&
•IPS ports that are part of a VRRP group use this format:
iqn..cisco:05.vrrp-&vrrp-ID#&-&vrrp-IP-addr&.&Target-pWWN&
•Ports that are part of a PortChannel use this format:
iqn..cisco:05.PC-&port-ch-intf#&-&port-ch-sub-intf#&.&Target-pWWN&
Note In this format, each IPS port in a Cisco MDS 9000 Family switch creates a different iSCSI target node name for the same Fibre Channel target.
To dynamically import Fibre Channel targets, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# iscsi import target fc
IPS modules dynamically map each Fibre Channel target in the Fibre Channel SAN to the IP network. The automatically-created iSCSI target node names use the IQN format.
Note Each iSCSI initiator may not have access to all targets depending on the configured access control mechanisms.
You can manually (statically) create an iSCSI target and assign a node name to it. A statically-mapped iSCSI target can either contain the whole FC target port, or it can contain one or more LUs from a Fibre Channel target port.
To create a static iSCSI virtual target for the entire Fibre Channel target port, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# iscsi virtual-target name
switch(config-(iscsi-tgt))#
Creates the iSCSI target name iqn.abc.
Step 3 
switch(config-(iscsi-tgt))# pWWN
26:00:01:02:03:04:05:06
Maps a virtual target node to a Fibre Channel target. One iSCSI target cannot contain more than one Fibre Channel target.
Don't specify the LUN if you wish to map the whole Fibre Channel target to an iSCSI target. All Fibre Channel target LUNs are exposed to iSCSI.
Use the LUN option to map different Fibre Channel LUNs to different iSCSI virtual targets. If you have already mapped the whole Fibre Channel target, you will not be able to use this option.
switch(config-(iscsi-tgt))# pWWN
26:00:00:00:00:11:00:11 fc-lun 1 iscsi-lun 1
Maps the whole target using LUN mapping options.
To create a static iSCSI target for the entire Fibre Channel target port, follow these steps:
Step 1 
switch# config terminal
switch(config)#
Enters configuration mode.
Step 2 
switch(config)# iscsi virtual-target
name iqn.abc
switch(config-(iscsi-tgt))#
Creates the iSCSI target name iqn.abc.
You can limit the Gigabit Ethernet interfaces over which static iSCSI targets are advertised. By default iSCSI targets are advertised on all Gigabit Ethernet interfaces, subinterface