Cisco 9200 stacking

Cisco 9200 stacking DEFAULT

Contents

Overview.. 3

Introduction to StackWise-160/80. 3

Stacking components. 4

Stack ports. 5

Stack connectors. 6

Stacking architecture. 6

Ring architecture. 6

Stack discovery. 8

ACTIVE election. 8

Stackwise-160/80 architecture. 12

Distributed forwarding architecture. 12

SSO architecture. 13

Implementing StackWise-160/80 SSO.. 14

Stack operations. 14

Adding a switch. 14

Removing a switch. 16

Stack split17

Stack merge. 17

Auto-upgrade. 17

Licensing. 19

Smart Licensing. 19

Stack design. 19

Simplified multilayer network design. 19

Scalable StackWise-160/80 cross-stack EtherChannel design. 20

Optimal StackWise-160/80 cross-stack forwarding EtherChannel design. 21

Reliable StackWise-160/80 cross-stack forwarding Ether Channel design. 23

Summary. 24


 

Overview

The network access model in the enterprise campus has evolved significantly from basic user connectivity to an intelligent, powerful, and high-speed building block. Security, cloud, mobility, and the Internet of Things (IoT) in enterprise networks have been driving the network toward major innovations. Cisco® Catalyst® 9000 software and hardware have been designed to address these current and future demands.

Wireless technology is the industry’s new megatrend, and it presents a multidimensional challenge to enterprise IT organizations. This swiftly growing technology, with exponentially increasing numbers of mobile devices with high-performance demands, is rapidly changing the landscape of network infrastructure and its reliability. IT requires a reassessment of traditional network models and assembly of network designs that can respond with a broad set of evolutionary architectures. Stacking provides an opportunity to use a pay-as-you-grow model to meet these demands.

As the wired and wireless Gigabit evolution begins at the access layer, user and application demands will increase to high-speed, low-latency data switching for optimal performance. Cisco has built a system architecture to support such innovations. The new Cisco Catalyst 9000 switching family is the next generation in the legendary family of enterprise LAN access, aggregation, and core switches. Cisco Catalyst 9200 Series Switches extend the power of intent-based networking and hardware and software innovation to a broader set of deployments. Compared to the scale and feature richness of the Cisco Catalyst 9300 Series Switches, the 9200 Series switches focus on offering features for the midmarket and simple branch deployments. With its family pedigree, the Cisco Catalyst 9200 Series offers simplicity without compromise. It is secure, is always on, and provides IT simplicity. This white paper talks in detail about the benefits and architecture of Cisco StackWise®-160/80, which provides the always-on experience.

Introduction to StackWise-160/80

The StackWise-160/80 architecture allows stacking of up to eight switches in a ring topology to achieve a high density of stack bandwidth. The stacking architecture expands the switches’ form factor, switching capacity, port density, and redundancy as well as providing a single control plane. It also provides resiliency, scalability, and central management. The latest Cisco Catalyst 9200 Series Switches support StackWise-160, whereas the Cisco Catalyst 9200L models support StackWise-80. This technology is flexible, modular, and evolutionary, and it delivers Cisco IOS® XE feature capabilities with hardware acceleration to every port in the stack.

Cisco Catalyst 9200 Series Switches come in data, Power over Ethernet (PoE), and Multigigabit versions. Each version has 24-port and 48-port variants. The hardware design of each model is cost-effective to support different network capacity load and switching performance.

Up to a maximum of eight switches can be stacked together physically in a ring topology to form a single, unified, virtual stack system. A Cisco Catalyst 9200 Series Switch, when deployed in StackWise-160/80 mode, has been designed to deliver deterministic and nonblocking switching performance to a maximum port density of 448 ports with a distributed data plane, single control plane, and management plane. The switching performance delivers hardware-accelerated, integrated borderless network services such as PoE, PoE+, Quality of Service (QoS), Access Control Lists (ACLs), Flexible NetFlow, Next-Generation Network-Based Application Recognition (NBAR2), streaming telemetry, and many more services on every port.

Depending on the requirements of each switch in the stack, the Cisco Catalyst 9200 Series Switches provide the flexibility for mixed-mode support between different models in a single stack ring. You can mix switches with different model variants (PoE, data, Multigigabit) and different network modules in the stack; however, it is not possible to mix the9200 and 9200L models due to different stacking bandwidth.

Figure 1 shows the StackWise-160/80 technology when four switches are part of the stack. Figure 2 shows simplified physical and logical views of a stack.

Figure 1.      Cisco Catalyst 9200 Series StackWise-160 technology

nb-06-stackwise-architecture-cte-en_0.jpg

Figure 2.      Simplified Cisco Catalyst 9200 Series physical and logical views

nb-06-stackwise-architecture-cte-en_1.jpg

Stacking components

Stacking cables are mandatory for stacking architecture. Depending on the physical setup of the infrastructure, different lengths of stacking cable may be needed. Each Cisco Catalyst switch supports a maximum of two stack cables for data stacking. Table 1 lists the stacking cables available. Stacking cables are not backward compatible with the predecessor Cisco Catalyst 2960-X Series Switches. The stacking kit is not included by default when a switch is ordered and should be added separately.

Table 1.       Different orderable stack cables

Product ID

Description

C9200-STACK-KIT

Stack kit for C9200 SKUs only: Two data stack adapters and one data stack cable

C9200L-STACK-KIT

Stack kit for C9200L SKUs only: Two data stack adapters and one data stack cable

STACK-T4-50CM

50CM Type 3 Stacking Cable

STACK-T4-1M

1M Type 3 Stacking Cable

STACK-T4-3M

3M Type 3 Stacking Cable

Stack ports

Each Cisco Catalyst 9200 Series Switch comes with two stack ports located on the back panel of the switch to support the StackWise-160/80 architecture. Figure 3 shows the stack port location.

Figure 3.      Stack cable and stack cable slot on a 9200L model

nb-06-stackwise-architecture-cte-en_2.jpg

Stack connectors

Figure 4 shows the stack connector. All stack ports are identical on all models of the Cisco Catalyst 9200 Series. Any side of the stack cable can connect to any stack port. Make sure the screws are completely tightened and the connection is secure.

Figure 4.      Cisco Catalyst 9200 Series stacking kit

nb-06-stackwise-architecture-cte-en_3.jpg

Stacking architecture

Ring architecture

When the stack is operating in a full ring, it can deliver high-speed throughput of up to 160 Gbps on each Cisco Catalyst 9200 Series Switch and throughput of 80 Gbps on each Cisco Catalyst 9200L switch. This multifold performance improvement is possible by combining the new internal Cisco Unified Access® Data Plane Application-Specific Integrated Circuit (UADP ASIC) and dual stack ports.

The high-speed backplane of the Cisco Catalyst 9200 Series stack-ring fabric is constructed by daisy-chaining the stack-member switches with Cisco proprietary cables that connect rear-side stack ports. The stack fabric consists of six unidirectional data transmission rings.

Figure 5 illustrates the 9200 Series StackWise-160 internal forwarding architecture.

Figure 5.      Cisco Catalyst 9200 Series StackWise-160 internal forwarding architecture

nb-06-stackwise-architecture-cte-en_4.jpg

Figure 6 illustrates the StackWise-80 internal forwarding architecture for the 9200L models.

Figure 6.      Cisco Catalyst 9200L StackWise-80 internal forwarding architecture

nb-06-stackwise-architecture-cte-en_5.jpg

The aggregated throughput that a stack of switches supports is a combination of two primary factors:

●   Total transmission rings: Each stack connector bundles multiple individual cables that carry data across the stack ring. This cabling structure creates four internal stack rings. This hardware design significantly improves the data transmission performance of each stack port of a Cisco Catalyst 9200 Series Switch.

●   Maximum throughput per ring: Each stack ring can transmit data at up to 40 Gbps on StackWise-160 and up to 20 Gbps on StackWise-80. These four internal stack rings enable an aggregated throughput of up to 80 Gbps per switch (up to 160 Gbps of unicast with Spatial Reuse Protocol [SRP]).

Table 2 lists the major details of the stacking architecture.

Table 2.       Details of the Cisco StackWise architecture

 

Cisco Catalyst 9200 StackWise-160

Cisco Catalyst 9200L StackWise-80

Total number of rings

4

4

Throughput per ring

40 Gbps

20 Gbps

Throughput per stack (full ring)

80 Gbps

40 Gbps

Throughput per stack (full ring) with SRP

160 Gbps

80 Gbps

Stack discovery

Once all switches are powered on and the stack interfaces are up, the Stack Discovery Protocol (SDP) discovers the stack topology using broadcasts. Neighbor information is shared with all other switches in the stack. In a full ring, discovery exits after all the members are found. In a half ring, the system waits for two minutes. Once all switches are discovered, switch numbers are determined. After switch number conflicts are resolved, the information is stored in a flash variable block for future use. Active election begins after discovery exits.

The following command can be used to check the status of the stack cable and to identify the neighbor device of the stack cable.

nb-06-stackwise-architecture-cte-en_6.jpg

ACTIVE election

To determine the single ACTIVE and STANDBY switch role during the complete stack reboot process or during the initial boot-up, all switches are required to go through an election process. All member switches participate in the election of an ACTIVE stack switch if they all boot up within the election window (120 seconds).

The following parameters are taken into account in the order listed below for active switch election:

●   Highest priority

●   Lowest MAC address

The STANDBY is elected by the ACTIVE switch after two minutes to reduce the stress of high-availability sync on the stack.

By default, the priority is 1 for all switches; hence it falls back to the MAC address to determine the ACTIVE switch if a priority is not defined explicitly. The switch with the lowest MAC address will take the role of the ACTIVE switch. The remaining switches join the stack as member switches. Once all the switches in the stack are discovered as members, the ACTIVE switch elects the STANDBY switch. Any switch that you add to the stack after the election process completes is considered a straggler and will not participate in the ACTIVE stack switch election.

To determine which switch is elected as the ACTIVE switch, adjust the switch priority for all stack switches. The switch with the highest priority becomes the ACTIVE switch. This configuration is a one-step process and is typically done at the time of the initial switch boot-up. The switch priorities are set in the ROMMON configuration of each individual switch in the stack ring. The switch priority configuration from ROMMON is parsed during the boot cycle instead of being read from the startup configuration stored in NVRAM. Thus, the switch-priority configuration cannot be verified from startup or the running configuration, since it is programmed into a different configuration component.

The priority of a Cisco Catalyst 9200 Series Switch can be modified using Exec mode commands. A reload is required for the change to take effect. The following commands show how to configure the switch priority and change switch numbers.

Cisco Catalyst 9200 Series switch priority

9200>enable

9200#switch <number> priority 15

!Set priority 15 to elect switch in ACTIVE role

9200#switch <number> priority 14

!Set priority 14 to elect switch in STANDBY role

9200#switch <number> priority 13

!Set priority 13 to elect switch in next STANDBY role

9200#switch <number> priority 12

!Set priority 12 to elect switch in next STANDBY role

9200>enable

9200#switch <number> renumber <number>

!Statically renumber switch in stack-ring

There are two ways to influence a particular switch in the stack to take over the role of the ACTIVE switch:

●   Configure the preferred switch with the highest priority (15) so that it assumes the role of the ACTIVE switch.

●   Power the preferred switch on first to have it take on the role of the ACTIVE switch.

Some conditions apply for switches when deployed in StackWise-160/80:

●   Valid switch numbers are 1 through 8 for Cisco Catalyst 9200 Series Switches. The port numbers of the member switches begin with the switch number, such as Gig1/0/1, Te1/1/1, or TwentyFiveGigE1/1/1, depending on whether the ports are Gigabit Ethernet (GE), 10 GE, or 25 GE.

Example: Switch 3 would have G3/0/1, Te3/1/1, TwentyFiveGigE3/1/1.

●   Switch numbers are persistent, meaning that each switch keeps the same switch number after it reboots as a stack member, even if it is no longer part of the stack.

●   The stack ACTIVE switch resolves any switch number conflicts and renumbers the switch.

●   Switch numbering does not reflect the physical location of the switch. However, the numbering can be changed to match the physical location using the command “switch current-stack-member-number renumber new-stack-member-number” from the enable mode.

Example: “switch 1 renumber 2” renames switch 1 to 2 and changes the port numbers from G1/1/1 and Te1/1/1 to G2/1/1 and Te2/1/1. A reload is required for this to take effect.

When you disconnect a stack member, the switch numbers and port numbers of the remaining stack members do not change, and the stack does not reload.

The system roles in the resilient StackWise-160 architecture can be verified with the commands shown below.

C92-Stack#show switch

Switch/Stack Mac Address : dc8c.3777.5b00 - Local Mac Address

Mac persistency wait time: Indefinite

                                             H/W   Current

Switch#   Role    Mac Address     Priority Version  State

-------------------------------------------------------------------------------------

*1       Active   dc8c.3777.5b00     10     V01     Ready               

 2       Standby  dc8c.3777.6c00     1      V01     Ready               

 3       Member   dc8c.377f.af00     1      V01     Ready               

 4       Member   dc8c.37a3.d100     1      V01     Ready               

 

C92-Stack#show redundancy

Redundant System Information :

------------------------------

       Available system uptime = 4 days, 21 hours, 27 minutes

Switchovers system experienced = 0

              Standby failures = 0

        Last switchover reason = none

 

                 Hardware Mode = Duplex

    Configured Redundancy Mode = sso

     Operating Redundancy Mode = sso

              Maintenance Mode = Disabled

                Communications = Up

 

Current Processor Information :

-------------------------------

               Active Location = slot 1

        Current Software state = ACTIVE

       Uptime in current state = 4 days, 21 hours, 27 minutes

                 Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 16.11.1, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Thu 28-Mar-19 07:19 by mcpre

                          BOOT = flash:packages.conf;

        Configuration register = 0x102

 

Peer Processor Information :

----------------------------

              Standby Location = slot 2

        Current Software state = STANDBY HOT

       Uptime in current state = 4 days, 21 hours, 24 minutes

                 Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 16.11.1, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Thu 28-Mar-19 07:19 by mcpre

                          BOOT = flash:packages.conf;

        Configuration register = 0x102

The STANDBY switch, which is in HOT-STANDBY mode, will transition into the ACTIVE role upon detecting failure of the primary ACTIVE switch. A new STANDBY switch will be elected from the available member switches, and it will transition to HOT-STANDBY.

Figure 7 shows the roles and operation of switches in a StackWise-160/80 architecture.

Figure 7.      Roles and operation of StackWise-160/80

nb-06-stackwise-architecture-cte-en_7.jpg

Stackwise-160/80 architecture

The forwarding architecture in the Cisco Catalyst 9200 Series Switches has been designed to provide up to 160G of stack bandwidth. The software architecture uses the credit-based token algorithm. To optimally forward the traffic within the stack ring, the packet-stripping function is performed on the destination switch instead of on the source or ingress switch.

This mechanism is known as spatial-reuse forwarding. It allows multiple flows to coexist to enable a parallel forwarding design. The spatial-reuse capability significantly boosts data-plane switching performance in the stack-ring switching architecture. The broadcast and multicast packets are still required to do source stripping, since the location of the destination device is known and there could be multiple multicast listener devices within the stack ring.

Distributed forwarding architecture

The forwarding architecture is designed to provide distributed switching across all member switches in the stack, as implemented in distributed, modular Cisco platforms. To optimize data-plane performance by using hardware resources from each Cisco Catalyst 9200 Series stack member switch, network services such as QoS, security ACLs, and others are fully distributed and programmed to be enforced locally on network ports. This distributed hardware resource utilization process delivers wire-speed switching performance that increases overall system resource capacity, prevents centralized overload processing on the ACTIVE switch, and optimizes stack-ring bandwidth capacity.

Since Cisco StackWise-160/80 technology is designed to offer modular-class system redundancy in stack design, It requires centralized control and a management plane with a distributed forwarding architecture. To enable the stacked switches to logically appear as a single virtual switch, the Cisco IOS® daemon (IOSd) process on the ACTIVE switch centrally manages all management-plane and network-control-plane operations with Layer 2 and Layer 3 protocols, including Spanning Tree Protocol (STP), IP routing, Cisco Express Forwarding, Policy-Based Routing (PBR), and others.

Depending on the implemented network protocols, the ACTIVE switch communicates with the rest of the multilayer or routed access infrastructure to dynamically develop the forwarding tables. The ACTIVE switch also updates all member switches for forwarding information. The distributed forwarding capability provides local switching lookup for the switching decision process. All ingress and egress wired data-plane traffic is fully distributed in a StackWise-160/80-based system design.

Independent of the IOSd operational state, the hardware forwarding information base (FIB) is actively programmed in ASICs across all stack-member switches in a stack ring. Figure 8 shows the Cisco Catalyst 9200 Series system architecture with centralized processing for control and management functions and distributed forwarding.

Figure 8.      Cisco Catalyst 9200 Series and centralized processing

nb-06-stackwise-architecture-cte-en_8.jpg

SSO architecture

The highly resilient stateful switchover (SSO) technology is a widely deployed solution in mission-critical campus and branch network designs. The key advantage of SSO is that it constantly delivers network availability without compromising performance and scalability during planned or unplanned network outages. The StackWise-160/80 architecture takes advantage of the same technology to maintain state machines and gracefully recover during an ACTIVE switch failure.

StackWise-160 SSO technology expands route processor redundancy (RPR) capabilities to provide transparent failover of several high-availability-aware Layer 2 and 3 protocols and Cisco IOS Software applications when the ACTIVE switchover occurs.

The state machines of non-high-availability-aware protocols and applications are not synchronized from ACTIVE to STANDBY, something the Cisco Catalyst 9200 Series Switches require to rebuild adjacencies and forwarding entries during an ACTIVE switch failure.

Implementing StackWise-160/80 SSO

To increase availability, the SSO capability is enabled by default when Cisco Catalyst 9200 Series Switches are deployed in StackWise-160/80 mode. No additional user intervention is required to enable this capability. The user can verify that SSO is configured and that the operational state is using a consistent CLI as a modular Cisco Catalyst system. The following example shows sample output of SSO redundancy states in the StackWise-160-based network design.

C92-Stack#show redundancy states

       my state = 13 -ACTIVE

     peer state = 8  -STANDBY HOT

           Mode = Duplex

           Unit = Primary

        Unit ID = 1

 

Redundancy Mode (Operational) = sso

Redundancy Mode (Configured)  = sso

Redundancy State              = sso

     Maintenance Mode = Disabled

    Manual Swact = enabled

 Communications = Up

 

   client count = 86

 client_notification_TMR = 30000 milliseconds

           RF debug mask = 0x0  

Stack operations

Adding a switch

When adding a new switch to the stack, stack cables have to be connected appropriately before the switch is powered on. The stack will operate at half bandwidth until the newly added switch has been powered on and has been discovered as a member. Once the switch had been discovered and is part of the stack, the stack ring transitions to the “Full” state.

The following output shows that the stack is in the “Half” ring state

9200#show switch stack-ring speed

 

Stack Ring Speed        : 80G

Stack Ring Configuration: Half

Stack Ring Protocol     : StackWise

Figure 9 shows the stack cabling when three switches are operating in StackWise-160/80.

Figure 9.      Stack with three switches

nb-06-stackwise-architecture-cte-en_9.jpg

Figure 10 shows the stacking cable structure when a fourth switch has been added to the above stack.

Figure 10.    StackWise-160/80 with four switches

nb-06-stackwise-architecture-cte-en_10.jpg

The following logs describe the addition of a new switch to the stack.

Newly added member switch logs

Initializing Hardware...

 

<SNIP>

 

Current ROMMON image : Primary

Last reset cause : PowerOn

C9200-24P platform with 8388608 Kbytes of main memory

boot: attempting to boot from [flash:packages.conf]

boot: reading file packages.conf

< SNIP >

 

Waiting for 120 seconds for other switches to boot

 

The switch number is 4

All switches in the stack have been discovered. Accelerating discovery

 

The stack discovers this new switch and assigns a switch number of 4:

9200-STACK#

*Jun 20 18:44:51.427: %STACKMGR-6-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 18:44:51.428: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 18:44:51.430: %STACKMGR-6-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 4 has been added to the stack.

 

<SNIP>

 

*Jun 20 18:44:57.034: %STACKMGR-6-SWITCH_ADDED: Switch 4 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 18:44:57.034: %STACKMGR-6-SWITCH_ADDED: Switch 4 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 18:44:57.249: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 4: EMP_RELAY: Channel UP!

We recommend having the stack cables connected to the newly added switch before it is powered on. If the stack cables are connected after the switch has been powered on, it will result in a reload of the newly added switch. A message similar to the following will be seen during this process on the new switch:

Chassis 1 reloading, reason - stack merge

Removing a switch

To remove a switch from the stack, the switch has be powered down and the stack cables have to be disconnected from the switch. The stack will operate at half bandwidth during this state until the stack cables are reconnected to form a full ring.

Figures 11 and 12 show stack cable connections before and after the removal of SW-4 from the stack.

Figure 11.    Stack cable connections with four switches in the stack

nb-06-stackwise-architecture-cte-en_11.jpg

SW-4 has been removed and the stack cable connections have been moved to bring the ring to the “Full” state.

Figure 12.    Stack cable connections after SW-4 has been removed from the stack

nb-06-stackwise-architecture-cte-en_12.jpg

Stack split

Stack split can occur when the backplane connectivity via the stack cables is lost due to unforeseen reasons, the member switches reload as they lose connectivity to both the ACTIVE and STANDBY switches.

Chassis 3 reloading, reason - lost both active and standby                                     

Chassis 4 reloading, reason - lost both active and standby

In this scenario above, SW-3 and SW-4 discover each other during the reload process and the ACTIVE switch is elected. The ACTIVE switch elects the STANDBY switch. The stack ring will continue to operate at half bandwidth until the link has been restored.

Stack merge

Once the connectivity is restored on a split stack, stack merge occurs. The stack pair that has the lowest priority on the ACTIVE switch will reload and join the stack pair that has the highest priority. If the priority is the same, the ACTIVE switch in the stack pair with the highest MAC address reloads.

Auto-upgrade

One of the options that gives the StackWise-160/80 architecture greater flexibility is the ability to increase the number of ports in the wiring closet without needing to configure the management plane or upgrade the newly installed switch manually. The newly added switch automatically upgrades to the software that the stack is currently running, without any manual intervention. The hardware and software architecture of the Cisco Catalyst 9200 Series continues to provide backward compatibility to dynamically insert a new 9200 Series switch into the stack ring without a major network disruption. The system and management operation, network configuration, and topologies remain transparent for network upgrades.

As a best practice, the newly joined switch can automatically receive consistent software versions from an ACTIVE switch and bring the system online without any user intervention. To automatically download consistent software versions to newly joined switches, you can use the following command from the global configuration mode. The auto-upgrade feature is not supported in bundled mode.

9200-STACK#conf t

Enter configuration commands, one per line. End with CNTL/Z.

9200-STACK(config)#software auto-upgrade enable

With the aforementioned command enabled, any member switch that is added to the stack will automatically upgrade to the current stack software.

Stack logs

9200-STACK#

*Jun 20 22:56:20.696: %STACKMGR-6-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 22:56:20.697: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 22:56:20.740: %STACKMGR-6-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 22:56:21.172: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Switch 1 R0/0: issu_stack: Incompatible software detected. Details: Chassis 4 is detected INCOMPATIBLE with software version of Active: FAILED: Version '16.10.01' mismatch with Active's running version '16.11.01' for package: 'guestshell'

*Jun 20 22:56:21.298: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Switch 1 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.

*Jun 20 22:56:24.452: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'chasfs' has taken 3168 msec (runtime: 0 msec) to process a 'stack chasfs fd' message

*Jun 20 22:56:25.476: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Switch 1 R0/0: auto_upgrade_client: Auto upgrade initiated for switch 4.

*Jun 20 22:56:25.502: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Switch 1 R0/0: auto_upgrade_client: Searching stack for software to upgrade switch 4.

*Jun 20 22:56:25.518: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Switch 1 R0/0: auto_upgrade_client: Found donor switch 1 to auto upgrade switch 4.

*Jun 20 22:56:25.534: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Switch 1 R0/0: auto_upgrade_client: Upgrading switch 4 with software from switch 1.

*Jun 20 22:57:05.536: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: Switch 1 R0/0: auto_upgrade_client: Initiating SMU autoupgrade for switch 4

*Jun 20 22:57:05.904: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: Switch 1 R0/0: auto_upgrade_client: Finished installing software on switch 4. à upgrade complete

 

*Jun 20 22:57:09.625: %AUTO_UPGRADE-5-AUTO_UPGRADE_RELOAD: Switch 1 R0/0: auto_upgrade_client: Reloading switch 4 to complete the auto upgrade.  à reload after upgrade

 

<SNIP >

 

*Jun 20 23:00:07.066: %STACKMGR-6-SWITCH_ADDED: Switch 4 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 23:00:07.066: %STACKMGR-6-SWITCH_ADDED: Switch 4 R0/0: stack_mgr: Switch 4 has been added to the stack.

*Jun 20 23:00:07.736: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 4: EMP_RELAY: Channel UP!

Licensing

Smart Licensing

With Release 16.9.1, for which a Smart License is mandatory, the stack automatically enables the EVAL license on a newly added switch should a license mismatch be present on it. Data programming will occur once the switch joins the stack.

License usage can be verified using the following command.

9200-STACK#sh license usage

License Authorization:

  Status: AUTHORIZED on May 25 22:53:33 2019 UTC

 

C9300 Network Advantage (C9200-24 Network Advantage):

  Description: C9200-24P Network Advantage

  Count: 4 à Number of switches in stack

  Version: 1.0

  Status: AUTHORIZED

C9200 DNA Advantage (C9200-24 DNA Advantage):

  Description: C9200-24P DNA Advantage

  Count: 4

  Version: 1.0

  Status: AUTHORIZED

Stack design

Simplified multilayer network design

The enterprise customer relies on real-time business applications such as voice, video, and others. System reliability and network availability become core, integrated service requirements to deliver nonstop communication in the network.

When the access layer environment becomes highly dense, StackWise-160/80 pools up to eight physical chassis into a single logical system from a network design perspective. As the access layer network infrastructure expands, the device-pooling capability of the StackWise-160/80 technology significantly simplifies operations and the network architecture itself.

Cisco recommends designing and deploying the multilayer distribution block with four major elements in the overall architecture: reduced fault domain, increased network security, deterministic forwarding paths, and optimal resiliency. Designing and developing a wiring closet with this architecture requires isolated broadcast domains or VLANs for each workgroup category, device, and application type. This network design needs to be applied consistently throughout the wiring closet network. It provides solid network security, stability, and reliability, but depending on the access layer network size, it may increase operational and troubleshooting complexity due to the increased number of VLANs, subnets, neighbor counts, and more.

The Cisco Catalyst 9200 Series StackWise-160/80 device-pooling design retains Cisco’s multilayer design principles. It also simplifies operational challenges with a reduced number of VLANs, STP instances, subnets, neighbor counts, etc. at the access and distribution layers. Figure 13 shows the simplified network design and operational data points of 9200 Series switches deployed in standalone mode compared to StackWise-160/80 mode.

Figure 13.    Comparison between Cisco Catalyst 9200 Series Switches in standalone mode and a StackWise

nb-06-stackwise-architecture-cte-en_13.jpg

Scalable StackWise-160/80 cross-stack EtherChannel design

StackWise-160/80 can make it possible to build a single-uplink EtherChannel interface by bundling up to eight parallel physical links, which could then be distributed evenly across all stack switches. Multiple uplinks from a mission-critical access layer switch are a base requirement for reliable networks to provide high-speed data load sharing and to deliver 1+1 path redundancy upon failure.

However, from a forwarding perspective the Layer 2 network becomes suboptimal when multiple parallel interfaces are deployed between two Layer 2 Ethernet switches.

The cross-stack EtherChannel interface simplifies STP topology over the logical interface and builds loop-free forwarding paths across all bundled uplink ports. If the Cisco Catalyst 9200 Series Switch is deployed in routed access mode, the cross-stack EtherChannel can also be deployed as a Layer 3 uplink interface to simplify unicast or multicast neighbor adjacencies as well as to simplify network topology with one forwarding table. Regardless of the 9200 Series switch’s deployed mode (multilayer or routed access), during a member switch or member link failure the EtherChannel does not disrupt network topology tables and delivers deterministic, hardware-driven, subsecond network recovery processes.

Cisco Catalyst 9200 Series Switches support multiple physical uplink ports to connect distribution-layer switches. Typically, up to two physical uplink interfaces are deployed from access layer switches for optimal load balancing and redundancy in the wiring closet.

When these switches are deployed in stack configuration mode, we recommend maintaining the same uplink connection design principle as for a dual stack-member system. The recommendation is to use uplinks on the member switches with multichassis EtherChannel (MEC), which helps with spanning multiple uplinks over the member switches. For example, eight Cisco Catalyst 9200 Series Switches deployed in a stack ring would have multiple diversified uplink ports from member switches. The remaining switches, where uplinks are not connected, would forward the data toward the core using a high-speed stack backplane.

This recommended uplink port design offers various benefits, from enhanced application performance to optimal user experience. Some key benefits include:

●   Improved application performance by increasing aggregated stack switching capacity with multiple distributed, high-speed 10-Gbps/25-Gbps uplinks between stack-member switches

●   Enhanced bidirectional traffic engineering with intelligent network data load sharing within the stack ring and across all distributed uplink physical ports

●   Improved system and application performance by using the distributed forwarding architecture advantage of hardware resources: buffers, queues, Ternary Content-Addressable Memory (TCAM), and others

●   Protection of the stack and network-level redundancy and reduction in congestion between distributed aggregation systems during a major outage at the access or distribution layer

Figure 14 illustrates a sample uplink network design for a Cisco Catalyst 9200 Series StackWise-160/80 distribution layer system.

Figure 14.    Cisco Catalyst 9200 Series StackWise-160/80 uplink design best practices

nb-06-stackwise-architecture-cte-en_14.jpg

Optimal StackWise-160/80 cross-stack forwarding EtherChannel design

The egress data load forwarding from StackWise-160/80 is determined based on how the upstream network is designed. The loop-free forwarding topology uses all available paths to switch data traffic based on precomputed Cisco Express Forwarding or EtherChannel hash results. In a distributed forwarding architecture, the Cisco Catalyst 9200 Series stack switches verify the Layer 2 to Layer 3 data variables from incoming traffic to determine the best physical uplink ports before forwarding traffic to the upstream system.

The Cisco Catalyst 9200 Series Switches require more variables in packet tuples to perform granular switching decisions. In a large-scale design, the source MAC address-based EtherChannel load-sharing mode may deliver adequate results to enable the use of all upstream member links. However, in mid- to low-scale networks, the switches may not have enough variable points to compute the best egress uplink path. In this case, to optimize the switching performance with granular packet forwarding decisions across all available cross-stack uplink paths, the default EtherChannel hash computation can be tuned to include Layer 2 to Layer 3 address variables.

The next-generation Cisco Catalyst 9200 Series is designed to support a large number of EtherChannel hash variables to deliver optimal upstream egress forwarding decisions. Table 3 outlines the supported Layer 2 to Layer 4 EtherChannel hash algorithm.

Table 3.       Cisco Catalyst 9200 Series EtherChannel hash algorithm

Layer

EtherChannel hash

Non-IP

src-mac (default)

Layer 2

src-mac (default)

dst-mac

src-dst-mac

Layer 3

src-ip

dst-ip

src-dst-ip

Layer 4

src-port

dst-port

src-dst-port

Layer 3 plus Layer 4

src-mixed-ip-port

dst-mixed-ip-port

src-dst-mixed-ip-port (Recommended)

The network administrator can adjust the default EtherChannel hash algorithm from a global configuration mode, as demonstrated in the sample code that follows.

Verification:

9200-STACK#show etherchannel load-balance

EtherChannel Load-Balancing Configuration:

        src-mac

 

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source MAC address

  IPv4: Source MAC address

  IPv6: Source MAC address

Command to change the default load-balance method:

9200-STACK#conf t

9200-STACK(config)#port-channel load-balance src-dst-mixed-ip-port

Verification:

9200-STACK#sh etherchannel load-balance

EtherChannel Load-Balancing Configuration:

        src-dst-mixed-ip-port

 

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address and TCP/UDP (layer-4) port number

IPv6: Source XOR Destination IP address and TCP/UDP (layer-4) port number

Reliable StackWise-160/80 cross-stack forwarding Ether Channel design

The link aggregation protocols build stateful, consistent, and reliable EtherChannel communication between two systems. To successfully establish a logical EtherChannel interface between two systems, the link aggregation protocol performs several link parameter checks to assure that each member link is equipped to deliver consistent switching performance and network service in the event of failure. During the EtherChannel startup process, each end of the system verifies the capabilities of each local and remote member link, including attributes such as speed, duplex, protocol dependencies, QoS capabilities, and more.

Cisco recommends bundling the cross-stack EtherChannel interface using link aggregation protocols such as Cisco Port Aggregation Protocol Plus (PAgP+) or Link Aggregation Control Protocol (LACP). Cisco Catalyst 9200 Series Switches deployed in StackWise-160/80 support both of these link aggregation protocols. 

Figure 15.    Cisco Catalyst 9200 Series cross-stack EtherChannel design

nb-06-stackwise-architecture-cte-en_15.jpg

Summary

Next-generation Cisco Catalyst 9200 Series Switches have been designed to meet future demands in wiring closet networks. Stackwise-160/80 provides maximum port density at the access layer, along with platform, software, and network resiliency at the access layer. As more technologies are integrated into the system, the Cisco Catalyst 9200 Series offers operational simplicity, scalability, performance, and adaptability for future protocols. The software architecture of Cisco StackWise-160/80 technology delivers superior performance and best-in-class resiliency along with the flexibility of the UADP ASIC. This document has focused primarily on the StackWise architecture for the Cisco Catalyst 9200 Series Switches.

 

 

 

Sours: https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9200-series-switches/nb-06-stackwise-architecture-cte-en.html

 

Extend intent-based networking everywhere

Cisco® Catalyst® 9200 Series switches extend the power of intent-based networking and Catalyst 9000 hardware and software innovation to a broader set of deployments. With its family pedigree, Catalyst 9200 Series switches offer simplicity without compromise – it is secure, always on, and IT simplified.

As foundational building blocks for the Cisco Digital Network Architecture, Catalyst 9200 Series switches help customers simplify complexity, optimize IT, and reduce operational costs by leveraging intelligence, automation and human expertise that no other vendor can deliver regardless of where you are in the intent-based networking journey.

Catalyst 9200 Series switches provide security features that protect the integrity of the hardware as well as the software and all data that flows through the switch. It provides resiliency that keeps your business up and running seamlessly. Combine that with open APIs of Cisco IOS XE and programmability of the UADP ASIC technology, Catalyst 9200 Series switches give you what you need now with investment protection on future innovations.

With full PoE+ capability, power and fan redundancy, stacking bandwidth up to 160 Gbps, modular uplinks, Layer 3 feature support, and cold patching, Catalyst 9200 Series switches are the industry’s unparalleled solution with differentiated resiliency and progressive architecture for cost-effective branch-office access.

Product overview

Product highlights

●    Up to 48 ports of full Power over Ethernet Plus (PoE+) capability

●    Resiliency with Field-Replaceable Units (FRU) and redundant power supply, fans, and modular uplinks

●    Flexible downlink options with data, PoE+ or mGig

●    Operational efficiency with optional backplane stacking, supporting stacking bandwidth up to 160 Gbps

●    UADP 2.0 Mini with integrated CPU offers customers optimized scale with better cost structure

●    Enhanced security with AES-128 MACsec encryption, policy-based segmentation, and trustworthy systems

●    Layer 3 capabilities, including OSPF, EIGRP, ISIS, RIP, and routed access

●    Advanced network monitoring using Full Flexible NetFlow

●    Cisco Software-Defined Access (SD-Access):

◦    Simplified operations and deployment with policy-based automation from edge to cloud managed with Cisco Identity Services Engine (ISE)

◦    Network assurance and improved resolution time through Cisco DNA Center

●    Plug and Play (PnP) enabled: A simple, secure, unified, and integrated offering to ease new branch or campus device rollouts or updates to an existing network

●    Cisco IOS XE: A Common Licensing based operating system for the enterprise Cisco Catalyst 9000 product family with support for model-driven programmability and streaming telemetry

●    ASIC with programmable pipeline and micro-engine capabilities, along with template-based, configurable allocation of Layer 2 and Layer 3 forwarding, Access Control Lists (ACLs), and Quality of Service (QoS) entries

Features and benefits

Platform details

Switch models and configurations

Models

FRU Power Supply

FRU Fans

Modular Uplinks

Stacking Bandwidth Support

SD-Access Support 1

Modular uplink models (C9200 Enhanced VN SKUs)

160 Gbps

Yes (32 Virtual Networks)2

Modular uplink models (C9200 SKUs)

160 Gbps

Yes (4 Virtual Networks)

Fixed uplink Models (C9200L SKUs)

80 Gbps

Limited (1 Virtual Network)

1 Catalyst 9200 standalone and stack can support 25 Access Tunnels (for fabric enabled APs). Note: Over the top fabric deployments eventually migrating to fabric wireless architecture should consider this limit during design/deployment

2 Catalyst 9200-24PB-A, C9200-48PB-A PIDs supports 32 Virtual Networks. These skus cannot be stacked with C9200 SKUs with 4VNs

The Cisco Catalyst 9200 Series is made up of modular (C9200) and fixed (C9200L) switch models.

Cisco Catalyst 9200 Series switches_A

 

Cisco Catalyst 9200 Series switches_B

Figure 1.            

Cisco Catalyst 9200 Series switches

Table 1.        Cisco Catalyst 9200 Series Switch configurations

Switch model

Downlinks total 10/100/1000 or PoE+ copper ports

Uplink configuration

Default primary AC power supply

Fans

Modular uplink models

C9200-24T

24 ports data

Modular uplink options

PWR-C6-125WAC

FRU redundant

C9200-24P

24 ports full PoE+

Modular uplink options

PWR-C6-600WAC

FRU redundant

C9200-24PB

24 ports full PoE+

Modular uplink options

PWR-C6-600WAC

FRU redundant

C9200-24PXG

24 ports full PoE+ (8 mGig ports up to 10G, 16 ports up to 1G)

Modular uplink options

PWR-C6-600WAC

FRU redundant

C9200-48T

48 ports data

Modular uplink options

PWR-C6-125WAC

FRU redundant

C9200-48P

48 ports full PoE+

Modular uplink options

PWR-C6-1KWAC

FRU redundant

C9200-48PL

48 Ports partial PoE+

Modular uplink options

PWR-C6-600WAC

FRU redundant

C9200-48PB

48 ports full PoE+

Modular uplink options

PWR-C6-1KWAC

FRU redundant

C9200-48PXG

48 ports full PoE+ (8 mGig ports up to 10G, 40 ports up to 1G)

Modular uplink options

PWR-C6-1KWAC

FRU redundant

Fixed uplink models

C9200L-24T-4G

24 ports data

4x 1G fixed uplinks

PWR-C5-125WAC

Fixed redundant

C9200L-24P-4G

24 ports full PoE+

4x 1G fixed uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-48T-4G

48 ports data

4x 1G fixed uplinks

PWR-C5-125WAC

Fixed redundant

C9200L-48P-4G

48 ports full POE+

4x 1G fixed uplinks

PWR-C5-1KWAC

Fixed redundant

C9200L-48PL-4G

48 Ports partial PoE+

4X 1G Fixed uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-24T-4X

24 ports data

4x 1/10G fixed uplinks

PWR-C5-125WAC

Fixed redundant

C9200L-24P-4X

24 ports full PoE+

4x 1/10G fixed uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-48T-4X

48 ports data

4x 1/10G fixed uplinks

PWR-C5-125WAC

Fixed redundant

C9200L-48P-4X

48 ports full PoE+

4x 1/10G fixed uplinks

PWR-C5-1KWAC

Fixed redundant

C9200L-48PL-4X

48 Port partial PoE+

4X 1/10G Fixed Uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-24PXG-4X

24 ports full PoE+ (8 mGig ports up to 10G, 16 ports up to 1G)

4x 1/10G fixed uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-48PXG-4X

48 ports full POE+ (12 mGig ports up to 10G, 36 ports up to 1G)

4x 1/10G fixed uplinks

PWR-C5-1KWAC

Fixed redundant

C9200L-24PXG-2Y

24 ports full PoE+ (8 mGig ports up to 10G, 16 ports up to 1G)

2x 1/10/25G fixed uplinks

PWR-C5-600WAC

Fixed redundant

C9200L-48PXG-2Y

48 ports full POE+ (8 mGig ports up to 10G, 40 ports up to 1G)

2x 1/10/25G fixed uplinks

PWR-C5-1KWAC

Fixed redundant

Network modules

Cisco Catalyst 9200 Series switches come with modular or fixed uplinks as indicated in Table 1. With modular SKUs, the field-replaceable network modules provide infrastructure investment protection by allowing a nondisruptive migration from 1G to 10G and beyond. When you purchase the switch, you can choose from the network modules described in Table 2.

Cisco Catalyst 9200 Series Switch network modules

Figure 2.            

Cisco Catalyst 9200 Series Switch network modules

Table 2.        Network module part numbers and descriptions

Network module

Description

C9200-NM-2Y1

2 x 25G Network Module

C9200-NM-2Q1

2 x 40G Network Module

C9200-NM-4G2

4x 1G network module

C9200-NM-4X

4x 1G/10G network module

C9200-NM-BLANK

No network module

1 Supported only on C9200-24PXG, C9200-48PXG

2 Not supported on C9200-24PXG, C9200-48PXG

For additional details, please read our FAQ: https://www.cisco.com/c/dam/en/us/products/collateral/switches/catalyst-9000/nb-09-cat-9k-faq-cte-en.pdf.

Platform resiliency

Power supplies

Cisco Catalyst 9200 Series switches support dual field-replaceable power supplies (Figure 3). Each switch ships with one default power supply, and a second identical power supply can be purchased with the initial order or can be added at a later time. The second power supply can provide redundancy or additional power to PoE+ ports as needed.

Intelligent PoE+

●      IEEE 802.3at PoE+ (up to 30W per port) is supported on Cisco Catalyst 9200 Series switches to lower the total cost of ownership for deployments that incorporate Cisco IP phones, Cisco Aironet® wireless access points, or other standards-compliant PoE+ end devices. PoE+ removes the need to supply wall power to PoE-enabled devices and eliminates the cost of adding electrical cabling and circuits that would otherwise be necessary in IP phone and WLAN deployments. With Cisco Catalyst 9200 Series switches, PoE+ power allocation is dynamic, and power mapping scales up to a maximum of 1440W of PoE+ power.

●      Perpetual PoE is supported on Cisco Catalyst 9200 Series switches, and maintains the PoE+ power during a switch reload. This is important for critical endpoints such as medical devices and for Internet of Things (IoT) endpoints such as PoE-powered lights, so that there is no disruption during a switch reboot.

●      Fast PoE: When power is restored to a switch, Fast PoE starts delivering power to endpoints without waiting for the operating system to fully load, thereby speeding up the time for the endpoint to start up.

Cisco Catalyst 9200 Series Switch dual redundant power supplies_A

 

Cisco Catalyst 9200 Series Switch dual redundant power supplies_B

Figure 3.            

Cisco Catalyst 9200 Series Switch dual redundant power supplies

Table 3 lists the PoE+ power availability for each model.

Table 3.        PoE+ Power with primary and secondary power supplies

Model

Default primary power supply

Available PoE power with single primary power supply only*

Optional secondary power supply

Available PoE power with additional secondary power supply*

C9200-24P

PWR-C6-600WAC

370W

PWR-C6-600WAC

740W

C9200-24PB

PWR-C6-600WAC

370W

PWR-C6-600WAC

740W

C9200-24PXG

PWR-C6-600WAC

370W

PWR-C6-600WAC

740W

C9200-48P

PWR-C6-1KWAC

740W

PWR-C6-1KWAC

1440W

C9200-48PL

PWR-C6-600WAC

370W

PWR-C6-600WAC

740W

C9200-48PB

PWR-C6-1KWAC

740W

PWR-C6-1KWAC

1440W

C9200-48PXG

PWR-C6-1KWAC

740W

PWR-C6-1KWAC

1440W

C9200L-24P-4G

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-24P-4X

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-48P-4G

PWR-C5-1KWAC

740W

PWR-C5-1KWAC

1440W

C9200L-48PL-4G

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-48P-4X

PWR-C5-1KWAC

740W

PWR-C5-1KWAC

1440W

C9200L-48PL-4X

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-24PXG-4X

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-48PXG-4X

PWR-C5-1KWAC

740W

PWR-C5-1KWAC

1440W

C9200L-24PXG-2Y

PWR-C5-600WAC

370W

PWR-C5-600WAC

740W

C9200L-48PXG-2Y

PWR-C5-1KWAC

740W

PWR-C5-1KWAC

1440W

* Limited by port number and port rating (for example, 24 PoE+ 30W ports = 720W)

Stacking

Cisco Catalyst 9200 Series switch models are designed for stacking switches as a single virtual switch, enabling customers to have a single management plane and control plane for up to 384 access ports.

Table 4 lists the supported stacking options.

Table 4.        Supported stacking options

Model

Stacking support

Stacking bandwidth support

Stacking hardware

Number of members

Supported stack members

Modular models (C9200 SKUs)

StackWise®-160

160 Gbps

C9200-STACK-KIT

8

Other C9200 models with same license level

Fixed models (C9200L SKUs)

StackWise-80

80 Gbps

C9200L-STACK-KIT

8

Other C9200L models with same license level

Mixed stacking is not supported. You cannot stack fixed (C9200L SKUs) with modular (C9200 SKUs) models, or other Catalyst switches, e.g. Cisco Catalyst 2960-X and 2960-XR Series.

The optional StackWise-160 and StackWise-80 kits consist of two adapters and a stacking cable. The default stacking cable is 0.5 m, but options of 1 m and 3 m are also available. Table 5 lists the stacking accessories.

Table 5.        Stacking accessories

Model

Description

C9200-STACK-KIT

Stack kit for C9200 SKUs only: Two data stack adapters and one data stack cable

C9200L-STACK-KIT

Stack kit for C9200L SKUs only: Two data stack adapters and one data stack cable

STACK-T4-50CM

Data stack 50 cm cable (default cable with Stack Kit)

STACK-T4-1M

Data stack 1 m cable

STACK T4-3M

Data stack 3 m cable

Cisco Catalyst 9200 Series Switch stacked units

Figure 4.            

Cisco Catalyst 9200 Series Switch stacked units

Fan

Cisco Catalyst 9200 Series switches also come with dual fans and support redundancy. Cisco Catalyst 9200 Series switches support redundancy with dual fans. On the C9200 SKUs, the fan units are field-replaceable, whereas on the fixed C9200L SKUs, the fan units are fixed. Table 5 lists the fan module part number.

Table 6.        Fan modules

Model

Description

FAN-T2=

Fan module

Performance and scalability

Table 7 lists performance and scalability metrics for Cisco Catalyst 9200 Series switches. Table 8 lists the bandwidth specifications.

Table 7.             Performance specifications

Description

C9200-24T, C9200-24P, C9200-48T, C9200-48P, C9200-24PB, C9200-48PB, C9200-24PXG,C9200-48PXG, C9200-48PL

C9200L-24T-4G, C9200L-24P-4G, C9200L-48T-4G, C9200L-48P-4G, C9200L-24T-4X, C9200L-24P-4X C9200L-48T-4X, C9200L-48P-4X, C9200L-24PXG-4X, C9200L-48PXG-4X, C9200L-24PXG-2Y, C9200L-24PXG-4X, C9200L-48PL-4X, C9200L-48PL-4G

Virtual Networks

4 for C9200-24T, C9200-24P, C9200-48T, C9200-48P, C9200-24PXG, C9200-48PXG, C9200-48PL

32 for C9200-24PB, C9200-48PB

1

Stacking bandwidth

160 Gbps

80 Gbps

Total number of MAC addresses

32,000

16,000

Total number of IPv4 routes (ARP plus learned routes)

14,000 (10,000 direct routes and 4,000 indirect routes)

11,000 (8,000 direct routes and 3,000 indirect routes)

IPv4 routing entries

4,000

3,000

IPv6 routing entries

2,000

1,500

Multicast routing scale

1,000

1,000

QoS scale entries

1,000

1,000

ACL scale entries

1,600

1,500

Packet buffer per SKU

6 MB buffers for 24- or 48-port Gigabit Ethernet models , 12MB buffers for 24 or 48 port multigigabit models

6 MB buffers for 24- or 48-port Gigabit Ethernet models, 12 MB buffers for 24 or 48 port multigigabit models

Flexible NetFlow (FNF) entries

16,000 flows on 24- and 48-port Gigabit Ethernet models

16,000 flows on 24- and 48-port Gigabit Ethernet models, 32,000 flows on 24 or 48 port multigigabit models

DRAM

4 GB

2 GB

Flash

4 GB

4 GB

VLAN IDs

4096

4096

PVST Instances

128

128

STP Virtual Ports (Port * VLANs) for PVST

13,000

13,000

STP Virtual Ports (Port * VALNs) for MST

13,000

13,000

Total Switched Virtual Interfaces (SVIs)

1000

512

Jumbo frames

9198 bytes

9198 bytes

Wireless bandwidth per switch

Up to 48 Gbps on 24-port and 48-port Gigabit Ethernet model

N/A

IP SGT binding scale

8K

10K

Number of IPv4 bindings

8K

10K

Number of SGT/DGT policies

2K

2K

Number of SXP Sessions

200

200

Table 8.        Bandwidth specifications

Description

Switching capacity

Switch capacity with Stacking

Forwarding rate

Forwarding rate with Stacking

C9200-24T

128 Gbps

288 Gbps

95.23 Mpps

214 Mpps

C9200-24P

128 Gbps

288 Gbps

95.23 Mpps

214 Mpps

C9200-24PB

128 Gbps

288 Gbps

95.23 Mpps

214 Mpps

C9200-24PXG

352 Gbps

532 Gbps

261.90 Mpps

395 Mpps

C9200-48T

176 Gbps

336 Gbps

130.95 Mpps

250 Mpps

C9200-48P

176 Gbps

336 Gbps

130.95 Mpps

250 Mpps

C9200-48PL

176 Gbps

336 Gbps

130.95 Mpps

250 Mpps

C9200-48PB

176 Gbps

336 Gbps

130.95 Mpps

250 Mpps

C9200-48PXG

400 Gbps

580 Gbps

297.61 Mpps

431 Mpps

C9200L-24T-4G

56 Gbps

136 Gbps

41.66 Mpps

101 Mpps

C9200L-24P-4G

56 Gbps

136 Gbps

41.66 Mpps

101 Mpps

C9200L-48T-4G

104 Gbps

184 Gbps

77.38 Mpps

137 Mpps

C9200L-48P-4G

104 Gbps

184 Gbps

77.38 Mpps

137 Mpps

C9200L-48PL-4G

104 Gbps

184 Gbps

77.38 Mpps

137 Mpps

C9200L-24T-4X

128 Gbps

208 Gbps

95.23 Mpps

155 Mpps

C9200L-24P-4X

128 Gbps

208 Gbps

95.23 Mpps

155 Mpps

C9200L-48T-4X

176 Gbps

256 Gbps

130.95 Mpps

190 Mpps

C9200L-48P-4X

176 Gbps

256 Gbps

130.95 Mpps

190 Mpps

C9200L-48PL-4X

176 Gbps

256 Gbps

130.95 Mpps

190 Mpps

C9200L-24PXG-4X

272 Gbps

352 Gbps

214.28 Mpps

262 Mpps

C9200L-24PXG-2Y

292 Gbps

372 Gbps

229.16 Mpps

277 Mpps

C9200L-48PXG-4X

392 Gbps

472 Gbps

291.66 Mpps

351 Mpps

C9200L-48PXG-2Y

340 Gbps

420 Gbps

252.97 Mpps

313 Mpps

*Measured with 64 byte packets

Software

Platform software benefits

Cisco IOS XE

Cisco IOS XE Software opens a completely new paradigm in network configuration, operation, and monitoring through network automation. Cisco’s automation solution is open, standards-based, and extensible across the entire lifecycle of a network device. The various automation mechanisms are outlined below.

●    Automated device provisioning is the ability to automate the process of upgrading software images and installing configuration files on Cisco Catalyst switches when they are being deployed in the network for the first time. Cisco provides turnkey solutions such as Plug and Play and Preboot Execution Environment (PXE) that enable an effortless and automated deployment.

●    API-driven configuration is available with modern network switches such as Cisco Catalyst 9200 Series switches. It supports a wide range of automation features and provides robust open APIs over NETCONF and RESTCONF using YANG data models for external tools, both off the shelf and custom built, to automatically provision network resources.

●    Granular visibility enables model-driven telemetry to stream data from a switch to a destination. The data to be streamed is identified through subscription to a data set in a YANG model. The subscribed data set is streamed to the destination at specified intervals. Additionally, Cisco IOS XE enables the push model. It provides near-real-time monitoring of the network, leading to quick detection and rectification of failures.

●    Seamless software upgrades and patching supports OS resilience. On Cisco Catalyst 9200 Series switches Cisco IOS XE supports cold patching with reboot, which provides fixes for critical bugs and security vulnerabilities between regular maintenance releases. This support lets you add patches without having to wait for the next maintenance release. Cold patching requires the switch to be rebooted after patching to allow the changes to take effect.

●    Trustworthy solutions built with Cisco Trust Anchor Technologies provide a highly secure foundation for Cisco products. With Cisco Catalyst 9200 Series switches, these technologies enable hardware and software authenticity assurance for supply chain trust and strong mitigation against man-in-the-middle attacks that compromise software and firmware. Trust Anchor capabilities include image signing, Secure Boot, and Cisco Trust Anchor module.

●    High availability: Cisco Catalyst 9200 Series switches support high-availability features, including the following:

◦    Cross-stack EtherChannel provides the ability to configure Cisco EtherChannel technology across different members of the stack for high resiliency.

◦    IEEE 802.1s Multiple Spanning Tree Protocol (MSTP) provides rapid spanning tree convergence independent of spanning tree timers and also offers the benefit of Layer 2 load balancing and distributed processing.

◦    Per-VLAN Rapid Spanning Tree (PVRST+) allows rapid spanning tree (IEEE 802.1w) reconvergence on a per-VLAN spanning tree basis, providing simpler configuration than MSTP. In both MSTP and PVRST+ modes, stacked units behave as a single spanning tree node.

◦    Switch-port auto-recovery (“err-disable” recovery) automatically attempts to reactivate a link that is disabled because of a network error.

◦    The Catalyst 9200 Series platform delivers the best SSO resiliency architecture in a stackable solution with sub-50-ms failover.

The Foundation of Software-Defined Access

Secure Segmentation with SD-Access

The enterprise network lies at the heart of digital transformation. A network that is open, programmable, integrated, and secure maximizes business agility, allowing new business opportunities to be pursued and captured.

Cisco DNA with SD-Access is the network fabric that powers business. It is an open and extensible software-driven architecture that accelerates and simplifies your enterprise network operations. The programmable architecture frees your IT staff from time-consuming, repetitive network configuration tasks so they can focus instead on innovation that positively transforms your business. SD-Access enables policy-based automation from edge to cloud with foundational capabilities. These include:

●      Simplified device deployment

●      Unified management of wired and wireless networks

●      Network virtualization and segmentation

●      Group-based policies

●      Context-based analytics

●      SD-Access: Cisco Catalyst 9200 Series switches are the entry-level devices for SD-Access, Cisco’s lead enterprise architecture, with policy-based automation from edge to cloud.

◦    Simplified segmentation and micro-segmentation, with predictable performance and scalability

◦    Automation through Cisco DNA Center

◦    Policy handled through the Cisco Identity Services Engine (ISE)

◦    Faster launch of new business services and significantly improved issue resolution time

●      Assurance

◦    Full network visibility and monitoring

◦    End-to-end Quality of Experience (QoE)

◦    Fast issue resolution and network remediation

●      Plug and Play (PnP) enabled: A simple, secure, unified, and integrated offering to ease new branch or campus device rollouts or updates to an existing network

Cloud Security

●      Umbrella Integration:

Small to midsize networks reliant on managed service providers can now host Cisco Umbrella agent directly on their Catalyst 9200 series switches. This allows the business to easily customize their DNS filtering policies to prevent BYOD or IoT guest or corporate users from accessing malicious or inappropriate websites, without having to rely on the MSP to push the policies out. It also lets them optimize use of bandwidth by allowing direct cloud access for trusted apps. Requires DNA-Advantage License and Umbrella License per device.

Full Flexible NetFlow

●    Full Flexible NetFlow (FNF): Cisco IOS FNF is the next generation in flow visibility technology. It enables optimization of the network infrastructure, reduces operation costs, and improves capacity planning and security incident detection with increased flexibility and scalability. Catalyst 9200 Series switches are capable of up to 16,000 flow entries on 48-port and 24 port models.

QoS

●    Superior QoS: Cisco Catalyst 9200 Series switches offer Gigabit Ethernet speeds with intelligent services that keep traffic flowing smoothly, even at 10 times the normal network speed. Industry-leading mechanisms for cross-stack marking, classification, and scheduling deliver superior performance for data, voice, and video traffic at wire speed. Superior QoS includes granular wireless bandwidth management and fair sharing, 802.1p Class of Service (CoS) and Differentiated Services Code Point (DSCP) field classification, Shaped Round Robin (SRR) scheduling, Committed Information Rate (CIR), and eight egress queues per port.

Smart operation

●      WebUI:

WebUI is an embedded GUI-based device-management tool that provides the ability to provision the device, to simplify device deployment and manageability, and to enhance the user experience. It comes with the default image, so there is no need to enable anything or install any license on the device. You can use WebUI to build configurations, and to monitor and troubleshoot the device without having CLI expertise.

●      RFID tags:

Cisco Catalyst 9200 Series switches have an embedded RFID tag that facilitates easy asset and inventory management using commercial RFID readers.

●      Blue beacon:

Cisco Catalyst 9200 Series switches support both front and back blue beacon LEDs for easy identification of the switch being accessed.

●      Efficient switch operation*:

Cisco Catalyst 9200 Series switches provide optimum power saving with Energy Efficient Ethernet (EEE) on the RJ-45 ports and low-power operations for industry best-in-class power management and power consumption capabilities. The ports support reduced power modes so that ports not in use can move into a lower power utilization state. Other efficient switch operation features are as follows:

◦    Per-port power consumption command allows customers to specify a maximum power setting on an individual port.

◦    Per-port PoE power sensing measures actual power being drawn, enabling more intelligent control of powered devices. The PoE MIB provides proactive visibility into power usage and allows you to set different power-level thresholds.

●      Bluetooth ready:

Cisco Catalyst 9200 Series switches have hardware support to connect a Bluetooth dongle to your switch, enabling you to use this wireless interface as an IP management port interface. The port can be used for configuration and troubleshooting using WebUI or the Command-Line Interface (CLI), and to transfer images and configurations.

High-performance ip routing

The Cisco Express Forwarding hardware routing architecture delivers extremely high-performance IP routing in Cisco Catalyst 9200 Series switches, based on:

●      IP unicast routing protocols (including static, Routing Information Protocol Version 1 [RIPv1], RIPv2, RIPng, and Open Shortest Path First [OSPF], Routed Access) are supported for small network routing applications with the Network Essentials stack. Equal-cost routing facilitates Layer 3 load balancing and redundancy across the stack.

●      Advanced IP unicast routing protocols (including Full [OSPF], Enhanced Interior Gateway Routing Protocol [EIGRP], and Intermediate System-to-Intermediate System Version 4 [IS-ISv4]) are supported for load balancing and for constructing scalable LANs. Ipv6 routing (using OSPFv3 and EIGRPv6) is supported in hardware for maximum performance.

●      Protocol-Independent Multicast (PIM) for IP multicast routing is supported, including PIM sparse mode (PIM SM), and Source-Specific Multicast (SSM).

●      Ipv6 addressing is supported on interfaces with appropriate show commands for monitoring and troubleshooting.

Licensing

Introduction to Smart Licensing

Cisco Smart Licensing is a flexible licensing model that provides you with an easier, faster, and more consistent way to purchase and manage software across the Cisco portfolio and across your organization. And it’s secure – you control what users can access. With Smart Licensing you get:

●      Easy Activation: Smart Licensing establishes a pool of software licenses that can be used across the entire organization—no more PAKs (Product Activation Keys).

●      Unified Management: My Cisco Entitlements (MCE) provides a complete view into all of your Cisco products and services in an easy-to-use portal, so you always know what you have and what you are using.

●      License Flexibility: Your software is not node-locked to your hardware, so you can easily use and transfer licenses as needed.

To use Smart Licensing, you must first set up a Smart Account on Cisco Software Central (software.cisco.com).

For a more detailed overview on Cisco Licensing, go to cisco.com/go/licensingguide

Packaging: Network and Cisco DNA licensing

The Cisco Catalyst 9000 family of switches introduces a new and simplified licensing package in the form of base and add-on licenses.

●      The perpetual licensing package includes the Network Essentials and Network Advantage licensing options that are tied to the hardware. Between them, the base licensing packages cover switching fundamentals, management automation, troubleshooting, and advanced switching features. These Network licenses are perpetual.

●      The subscription licensing package includes the Cisco DNA Essentials and Cisco DNA Advantage options. In addition to on-box capabilities, the features available with this package provide Cisco innovations on the switch, as well as on Cisco DNA Center. The Cisco DNA subscription licenses are mandatory at the time of configuration. With Cisco DNA software licenses, customers receive embedded SWSS – which covers 24x7x365 Cisco Technical Assistance Center (TAC) support, software release updates, advanced support analytics, and designated service management. This is valid only for the Cisco DNA software subscription stacks (Cisco DNA Essentials, Advantage, and Premier).

Note:      For full hardware support, including the perpetual network stack customers will require Smart Net Total Care for 24x7x365 Cisco Technical Assistance Center (TAC) support, proactive security and product alerts, and product lifecycle management. An additional option for hardware support is Solution Support for your multivendor Cisco solution environment

License consumption is easily determined by the package itself. While perpetual licenses are always permanent and without an expiration date, subscription licenses have to be purchased for a 3-, 5-, or 7-year term (and hence are also known as term-based licenses). Table 12 shows the combinations of perpetual and subscription licenses that must be purchased.

Supported licensing combinations

Table 9.        Licensing combinations

 

Cisco DNA Premier

Cisco DNA Advantage

Cisco DNA Essentials

Network Essentials

No

No

Yes

Network Advantage

Yes

Yes

No*

* At the time of license renewal, the Cisco DNA Essentials license can be purchased to be used with Network Advantage.

Cisco DNA Premier subscription

Cisco DNA Premier subscriptions offer a flexible way to buy software for the access, WAN, and data center domains. At each stage in the product lifecycle, Cisco DNA Premier subscriptions help make buying, managing, and upgrading your network and infrastructure software easier. Cisco DNA Premier subscriptions provide:

●      Flexible licensing models to smoothly distribute customers’ software spending over time

●      Investment protection for software purchases through software services–enabled license portability

●      Lower cost of entry with the new Cisco DNA Premier Subscription for Switching model

For ordering information for Cisco DNA Premier Software for Cisco Catalyst 9200 Series switches, go to: https://www.cisco.com/c/en/us/products/collateral/software/one-subscription-switching/nb-06-dna-sw-sub-access-sw-ctp-en.html.

Cisco Catalyst 9200 Series switches run on Cisco IOS XE Release 16.9.2 or later. This software release includes all the features listed earlier in the Platform Software Benefits section.

Managing licenses with Smart Accounts: Creating Smart Accounts by using the Cisco Smart Software Manager (Cisco SSM) enables you to order devices and licensing packages and also manage your software licenses from a centralized website. You can set up Cisco SSM to receive daily email alerts and to be notified of expiring add-on licenses that you want to renew.

You must order an add-on license in order to purchase a switch. When the license term expires, you can either renew the add-on license to continue using it or deactivate the add-on license and then reload the switch to continue operating with the base license capabilities.

Both the base and add-on licenses are also available for a 90-day evaluation period. An evaluation license is activated temporarily, without purchase. An expired evaluation license cannot be reactivated after reload.

Note:      It is not required to deploy Cisco DNA Center, just to use one of the above packages.

Tables 10 shows the features included in the Network Essentials and Advantage packages.

Table 11 shows the features included in the Cisco DNA Essentials and Advantage packages.

Network licensing

Table 10.     Network essentials and advantage package features

Features on Cisco Catalyst uplink switches

Network Essentials

Network Advantage

Switch fundamentals

Layer 2, Routed Access (RIP, EIGRP Stub, OSPF -– 1000 routes), PBR, PIM Stub Multicast (1000 routes), PVLAN, VRRP, PBR, CDP, QoS, FHS, 802.1X, MACsec-128, CoPP, SXP, IP SLA Responder,SSO

Advanced switch capabilities and scale

EIGRP, HSRP, IS-IS, BSR, MSDP, IP SLA, OSPF

Network segmentation

VRF, VXLAN, LISP, SGT

Automation

NETCONF, RESTCONF, YANG, PnP Agent, PnP

Telemetry and visibility

Sours: https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9200-series-switches/nb-06-cat9200-ser-data-sheet-cte-en.html
  1. Lexus lx rims
  2. Freertos interrupt
  3. Vanity art ravenna

Stack Manager and High Availability Configuration Guide, Cisco IOS XE Gibraltar 16.10.x (Catalyst 9200 Switches)

Feature History and Information for Switch Stacks

The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

Use the Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required

Feature Name

Release

Feature Information

Switch Stack

Cisco IOS XE Fuji 16.9.1

A switch stack can have up to eight stacking-capable switches connected through their StackWise ports. The stack members work together as a unified system. Layer 2 and Layer 3 protocols present the entire switch stack as a single entity to the network.

Sours: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/software/release/16-10/configuration_guide/stck_mgr_ha/b_1610_stck_mgr_ha_9200_cg/managing_switch_stacks.html
Unboxing - Cisco C9200L-48T-4G-E

Stack Manager and High Availability Configuration Guide, Cisco IOS XE Amsterdam 17.2.x (Catalyst 9200 Switches)

Feature History and Information for Switch Stacks

The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

Use the Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required

Feature Name

Release

Feature Information

Switch Stack

Cisco IOS XE Fuji 16.9.1

A switch stack can have up to eight stacking-capable switches connected through their StackWise ports. The stack members work together as a unified system. Layer 2 and Layer 3 protocols present the entire switch stack as a single entity to the network.

Switch Stack

Cisco IOS XE Amsterdam 17.2.1

C9200-24PB and C9200-48PB switch models can only be stacked with each other and not with other models of the Cisco Catalyst 9200 Series Switches.

Sours: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/software/release/17-2/configuration_guide/ha/b_172_stck_mgr_ha_9200_cg/managing_switch_stacks.html

Stacking cisco 9200

Cisco Catalyst 9200 Series Switches Hardware Installation Guide

Connecting Devices to the Ethernet Ports

10/100/1000/Multigigabit Ethernet Port Connections

The switch 10/100/1000 and Multigigabit Ethernet port configuration changes to operate at the speed of the attached device. If the attached ports do not support autonegotiation, you can manually set the speed and duplex parameters. Connecting devices that do not autonegotiate or that have the speed and duplex parameters manually set can reduce performance or result in no linkage.

To maximize performance, choose one of these methods for configuring the Ethernet ports:

  • Let the ports autonegotiate both speed and duplex.

  • Set the interface speed and duplex parameters on both ends of the connection.

Auto-MDIX Connections

The autonegotiation and the auto-MDIX features are enabled by default on the switch.

With autonegotiation, the switch port configurations change to operate at the speed of the attached device. If the attached device does not support autonegotiation, you can manually set the switch interface speed and duplex parameters.

With auto-MDIX, the switch detects the required cable type for copper Ethernet connections and configures the interface accordingly.

If auto-MDIX is disabled, use the guidelines in this table to select the correct cable.

Device

Crossover Cable

Straight-Through Cable

Switch to switch

Yes

No

Switch to hub

Yes

No

Switch to computer or server

No

Yes

Switch to router

No

Yes

Switch to IP phone

No

Yes

PoE and POE+ Port Connections

The 10/100/1000 PoE and PoE+ ports have the same autonegotiation settings and cabling requirements that are described in the 10/100/1000 Ports. These ports can provide PoE and PoE+ inline power.

PoE inline power supports devices compliant with the IEEE 802.3af standard, as well as prestandard Cisco IP Phones and Cisco Aironet Access Points. Each port can deliver up to 15.4 W of PoE. PoE+ inline power supports devices compliant with the IEEE 802.3at standard, by delivering up to 30 W of PoE+ power per port to all switch ports.

See Power Supply Modules for the power supply modules required to support PoE and PoE+ on 24- and 48-port switches.


Warning

Voltages that present a shock hazard may exist on Power over Ethernet (PoE) circuits if interconnections are made using uninsulated exposed metal contacts, conductors, or terminals. Avoid using such interconnection methods, unless the exposed metal parts are located within a restricted access location and users and service people who are authorized within the restricted access location are made aware of the hazard. A restricted access area can be accessed only through the use of a special tool, lock and key or other means of security. Statement 1072

Warning

Voice over IP (VoIP) service and the emergency calling service do not function if power fails or is disrupted. After power is restored, you might have to reset or reconfigure equipment to regain access to VoIP and the emergency calling service. In the USA, this emergency number is 911. You need to be aware of the emergency number in your country. Statement 371

Caution

Category 5e and Category 6 cables can store high levels of static electricity. Always ground the cables to a suitable and safe earth ground before connecting them to the switch or other devices.

Caution

Noncompliant cabling or powered devices can cause a PoE port fault. Use only standard-compliant cabling to connect Cisco prestandard IP Phones and wireless access points, IEEE 802.3af, or 802.3at (PoE+)-compliant devices. You must remove any cable or device that causes a PoE fault.
Sours: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9200/hardware/install/b-c9200-hig/b-c9200l-hig_chapter_010.html
Cisco Stackwise \u0026 Stackpower Guide

Put your finger in there. - They put the fish on the hook, - Lena smiled slyly, - The main thing is that he calmed down right away. Hes afraid to move, said Masha. - Yeah, - Lena nodded, - You can do with him what you want. For example, wash the scrotum again.

Similar news:

Don't mix it up. And also leave comments. Your opinion is very interesting.



4317 4318 4319 4320 4321