Sunday 2 March 2014

CUCM Licensing Overview

To better understand the licensing of CUCM, lets start from the beginning.
CUCM was known as Selsius Call Manager before Cisco acquired it and re-christened it to Cisco Call Manager starting with Version 3.0. At that time, for CallManager 3.x and 4.x, we had to buy licenses with phones. There was no method to enforce licensing. A spare license was cheaper than original license and companies started to exploit this. CUCM was running on Windows server. Also, if we had CUCM software, we could install as many as nodes in a cluster as required.
From CallManager 5.x, it runs as an Appliance on a customized RHEL with a different shell. Cisco started the concept of DLUs. Each device required different number of DLUs depending on the device features. Each DLUs cost about 25$. Also introduced was the Node licensing. So installing a new subscriber required a license.
From CUCM 6.x, Cisco renamed CallManager to CUCM and introduces Feature license. You can't upgrade to any major version without feature license. You cannot start the CUCM service without a feature license. When upgrading between major versions, the DLU count would become negative and you need a feature license to get back to the original count.
CUCM 8.x onwards due to the large number of collaboration devies and licenses, Cisco introduced a new CUWL (Cisco Unified Workspace Licensing) {pronounced as COOL}. It consists of a license based on users rather than on devices i.e. if a user has access to only one device then you can get a basic license can scale upto 10 devices per user which is a professional license.
Here is a list of CUWL licenses made available:
Essential : 1 device/user (analog/Voice)
Basic : 1 device/user (Voice)
Enhanced : 1 device/user (Video)
Advanced : 2 devices/user (Video)
Standard : 2 devices/user (Video)
Premium : 4 devices/user (Video)
Professional : 10 devices/user (Video)
Also introduced was Cisco User Connect Licensing (UCL) provides user-based licensing for individual Cisco Unified Communication products. UCL is further broken down by device type for Communications Manager and is viewed as the replacement for the ‘DLU'.
Till 8.x, the licensing was done by Publisher. From 9.0, the licensing is done by ELM (Enterprise License Manager).
ELM is installed by default on every node. It can be installed as a standalone using CUCM OVA template and it can be used to license multiple servers.
- Abhinay Mylavarapu

Troubleshooting CUCM Database Replication

CUCM uses IBM Informix for database needs. We have no native access to this (unless you are a Cisco TAC Engineer who can gain temporary access to root).
If DB replication breaks, we see many symptoms in our IPT network like Phone registered to a Subscriber unable to make calls to phones registered on the other subscriber, unable to login to extension mobility etc.

We can make use on any one of these methods to check replication status:
  •  Open RTMT -- Click on : Call Manager --> Service --> Database Summary
  • Open a web browser, type in https://<IP address>:8443/cucreports/ , Enter your authorized username and password.
    Go to : Database Status Report
  • Using putty, SSH to the CUCM to take CLI access and run this command : utils dbreplication runtimestate --> REPl.LOOP? is the current state.

Here is what the replication state means :
0 - Initialization state : This state indicates that replication is in the process of trying to  setup. Being in this state for a period longer than an hour could  indicate a failure in setup.

1 - Number of replicates not correct : This state is rarely seen, can indicate its still in the setup process. Being in this state for a period longer than an hour could indicate a failure in setup.

2 - Replication is good : All is well in paradise :)

3 - Tables are suspect : Logical connections have been established but we are unsure if tables match. This can happen because the other servers are unsure if  there is an update to a user facing feature that has not been passed  from that sub to the other device in the cluster.

4 - Setup failed / Dropped : The server no longer has an active logical connection to receive  database table across. No replication is occurring in this state.

If State is other than 2, check:
  • Server & Cluster connectivity : Check TCP/UDP ports needed to be opened on the network. To get the port list for your CUCM version, just google : CUCM < your version> ports
  • Configuration files (if on older CUCM - extremely rare) :
  1. /etc/hosts ---> local resolution of hostnames to IP addresses
  2. /home/informix/.rhosts ---> hosts trusted to make database connections
  3. $INFORMIXDIR/etc/sqlhosts ---> full list of CCM servers for replication
  • Check if server times are correct and synced (NTP working fine)
  • DNS not configured properly (forward/reverse lookup)
  • NTP not reachable/time drift between server
  • A Cisco DB replicator service not running/not working
  • Cisco Database Layer monitor (DbMon) hung/stopped

Useful Commands :


  • utils dbreplication setrepltimeout -- The default value is set to 300 seconds. You can validate this by running "show tech repltimeout". This is the timer used to put multiple servers into one run of the data sync. In other words, it is the "batching" timer. This affects when the broadcast realize template and data sync will fire (n seconds from the end of the first defined server). Clustering over WAN (CoW) long delays can cause the data sync process to be exponentially longer. Try to sync the local servers first.
 
  • utils dbreplication repair -- in CUCM 5.x, this command meant a reset of the replication, whereas, in CUCM 6.x and higher versions, this means a repair of the data. It runs a repair process on all tables in the replication for all servers that are included in the command. Run this command when RTMT = 2, not when RTMT = 0 or 3.
 
  • utils dbreplication repairtable / repairreplicate -- This command essentially does the same thing as the repair command, but runs on only one table / replicate, hence making the process much faster. It fixes the out of sync data for that table / replicate. You can verify by running "utils dbreplication status" to see if there are any mismatches or errors found. It is particularly useful on large CUCM clusters. Run this command when RTMT = 2, not when RTMT = 0 or 3.
 
  • utils dbreplication stop -- You should only be running this if you want to stop replication setup. The only way to recover from a stop is with a reset. This command removes the set-up indicator file i.e. the dbmonpreflightcheck file and kills the currently running replication commands. It pauses for the duration of repltimeout timer, so if you run replication commands soon after running a stop, it could kill the commands again. Run this command when RTMT = 0, not when RTMT = 3
 
  • utils dbreplication reset -- This command causes replication to be torn down and then set-up. You should run this command when RTMT = 4 or when you have issued stop. Successful completion of this process results in RTMT = 2.
 
  • utils dbreplication clusterreset -- Avoid running this command. It is for debugging replication set-up problems. It bypasses the RTMT settings, cluster requirements and normal CUCM set-up. It causes services to go out of sync with the database because it syncs data without change notification. The services need to be restarted when this command is run, no exceptions!
 
  • utils dbreplication dropadmindb -- Run this command when there is a looping attempt to define a server in replication. It's usually not the server that's failing, it's the pub which is corrupted as a result of an attempt or the sub, prior to the current one attempting set-up.
 
  • utils dbreplication forcedatasyncsub -- This command takes a backup of the publisher and restores it to the subscriber(s) and resets up replication. It requires a serivces restart on the subscriber so they get the new values.
 New Commands and Database Improvements in CUCM 9.x: 
  • Re-engineered CLI forcedatasyncsuball (Lightening fast) -- This command can now restore a larger cluster in a shorter period of time!

  • New CLI rebuild is a stop, drop and reset all in one (and faster) -- The architecture of Rebuild is multi-threaded, the total operation time is much shorter than executing three different CLI commands (stop / drop / reset). Rebuild, is a master command that will stop, delete and trigger the replication setup signal across the cluster automatically and in parallel:
  1. Stop DB Replication – stop the current replication setup process if exists
  1. Remove server from database – Remove replication from the network by either “cdr delete”, dropping the syscdr database or renaming the syscdr database remotely
  1. Trigger Dbmon on the subscriber to submit a replication setup request on to publisher.
 
  • New CLI utils replication status table/replicate -- The "utils dbreplication status" command is lengthy when it runs. If only one table is suspect, then you have to wait for all the tables to check. Being able to check one table speeds up checking of replication.

 
  • Better Log Collection -- "utils create report database" collects all the database logs in one go. Also, ercollect.sh script is embedded into the server for IBM root cause cases. The script is on the server now, no need to transfer and change permissions. It is accessible via root access only.
 
  • Faster and more accurate Runtimestate CLI -- This command is now multithreaded, making it much faster. The output will also be logged for historical RCA. If there are any unreachable servers in the cluster, this command will no longer hang. Some additional information will be included in it such as repltimeout and IDS server number.

- Abhinay Mylavarapu

Thursday 30 January 2014

IP Phone Boot Up and Registration

Hi Reader and welcome to this blog on IP Phone Boot up process and Registration. I will explain to you all the things involved in registering an IP Phone and go through the entire process in detail. So here goes...

The very first thing that happens after we connect an IP Phone to a switch-port is that it receives power from the port.. Now, there are two ways in which an IP Phone can receive power and those are:

Inline Power:
  • When the phone is connect to switch-port, the port sends FLPs (Fast Link Pulse) which are bursts of electric signal lasting 125 microseconds on the transmit pair.
  • The switch-port expects to receive this back on the receive port.
  • Devices that need power have a LPF (Low Pass Filter) between the transmit and receive pairs designed to pass only FLPs.
  • On receiving the FLP, power is applied
  • The port is taken out of discovery mode and put into ethernet auto-negotiate.
  • "Wait for Link" timer of 5 seconds starts. If the phone link is detected, power is kept on else power is switched off and port goes back to discovery mode.
Power over Ethernet:

(Industry standard - IEEE 802.3af delivers upto 15.4 Watts of power per port, using 48 volts)
  • When the phone is connected to switch-port, 2.7 V to 10 V is applied between the transmit and receive pairs.
  • Devices that need power have a 25K ohms resistance between the transmit and receive pairs.
  • If this is detected, then power is applied to the device if enough is available.
  • If we disconnect the phone from the port, the power connection is terminated in less than 250ms.

Note: The switch-port doesn't have a power ON mode to protect the NICs (Network Interface cards) which do not expect to receive power from the network.The switch queries its NMP (Network Management Processor) to determine whether enough power is available to power a device. Power can be sent through the cable on spare pairs or data pairs (phantom power). CDP is used for power negotiation after the link is up and phone has booted on. If switch doesn't support CDP, there is no reduction in power.
       Contact               Media Direct Interface Signal
   1                            Transmit +ve
            2                                   Transmit -ve
            3                                Receive +ve
            4                                  -ve
            5                                  -ve
            6                            Receive -ve
            7                                  +ve
8      +ve

Upon receiving power , the IP phone:
1) The phone obtains Voice VLAN information from the switch to which the IP phone is connected or uses statically defined Voice VLAN information [Admin VLAN ID]. Phone sends a maximum of 3 CDP messages requesting Voice VLAN ID. If it receives a response from switch with the VLAN, the packets are sent tagged, else the packets are untagged.

2) Requests DHCP information / uses statically configured settings. It provides the IP phone with the IP address of the TFTP server.
Settings > 3- Network Configuration > DHCP Enabled > YES -- AUTO
                                                                                         NO -- MANUAL

3)  Contacts TFTP server to download a configuration file - SEP<devicemac>.cnf.xml / SEPDefault.cnf.xml (if 1st one is not found)

These files contain :
    • List of three call managers plus an optional SRST router for IP phone to connect.
    • Load information specifying firmware version IP phone should be using.
    • URLs for directories, services, information and idle.
4) Phone reads load ID and version stamp and compares with current load.
If mismatched, loads new load from TFTP, IP phone resets and repeats entire boot process.

5) Phone attempts registration to its primary CUCM, If it is unavailable, IP phone tries the subsequent servers from the list provided via TFTP.

SKINNY CLIENT REGISTRATION :
- IP phone sets up a TCP connection with the CUCM
- IP phone sends a Alarm Message which contains Device name, load of the IP phone, Alarm status, Phone's IP address.
- IP phone sends a stationRegisterMessage containing Device name, Device type and IP.
- IP phone sends a StationIPPortMessage containing the TCP port on which it is listening.
- CUCM sends a StationRegisterAck to the IP phone to indicate successful registration.
- CUCM queries the IP phone about its capabilities to which the IP phone replies.
- IP phone requests for Softkey template, Button template, Softkey set, line status, speed-dial status and date and time.
- Apart from the above, CUCM sends the output prompt display message - Your Current Options.

I hope you understood the above and any doubts you can always comment here and I will help to solve them.

- Abhinay Mylavarapu.