End to End UC Deployment - 00 - Overview

There's always been the intention of doing a documented end to end deployment of the Cisco UC Stack in a lab, and the end of year break seems like a good enough time than any.

Concept is to cover the installation of a standard office UC backend, including integration into a Microsoft Environment (AD / Exchange etc).

The UC Environment will include the following systems:
 - CallManager 10.5
 - Unity Connection 10.5
 - IM and Presence 10.5
 - UCCX 10.6
 - VCS Expressway for MRA.

Our Microsoft Environment will consist of the following:
 - Windows 2012 R2 Domain Controller.
 - Exchange 2010 for EMail.

Other services we'll deploy:
 - A Linux box for UC Backups
 - A CDR Collection System based on Kibana and Elasticsearch.

I'll be using trial / demo licenses for all the above systems where necessary.

Lets begin...

Syslog not logging events though the server is receiving syslog packets

Recent issue with a new centos syslog-ng box which was configured as a syslog destination for a number of devices, but we weren't seeing logs being generated.

Things we tested

 - Checked syslog-ng.conf - all good.

 - Sent test syslogs via a command line syslog generator - all good.

 - Ran tcpdump on the interface receiving syslog via 'tcpdump -A -i bond1 udp port 514' and could see traffic for syslog successfully.
 - Re-ran syslog, still no traffic

Solution

 - Issue was resolved after adding the following to /etc/sysctl.conf and running sysctl -p.

# Drop packets that look like they are spoofed.  Default = 1.
# 0 = don't check;
# 1 = drop packets that as sourced at a directly connected interface
#     but were input from another interface;
# 2 = drop any packets that look spoofed.
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.bond1.rp_filter = 0

tac_plus install on Ubuntu - Fixing the crypt error message during build

I'm going to add a couple of posts later on in regards to the process of installing tac_plus on Ubuntu but I thought I'd pre-empt the full post with a quick fix if you're trying to install and hit the following error:

authen.cpp: In member function 'int tauthen::changepass(int)':
authen.cpp:1057:52: error: 'crypt' was not declared in this scope
I initially added #define <crypt.h>' to authen.cpp but was finding that the passwords being generated by the function wasn't matching what the tac_plus web front end was generating.

Ended up tracing the fix back to adding #include <unistd.h> to the define section in tac_plus.h.

Testing following a successful build showed passwords being generated successfully by both the tac_plus program and the php web front end.

Hope this shortcuts the build process for people.

tac_plus Deep-Dive #1 - Command Authorization and db tables

First in a few posts I hope to get up to here covering tac_plus (using the version from networkforums.com) which I've been looking into over the weekend.  All the work I've been doing so far has been in relation to cisco devices

Primary interest has been how we get command authorization working on our network equipment.  Having someone 'reload' a core device in the middle of the day is only something we want to happen once...

So as a quick overview of how some of the values in the database come together:

1. The column 'cmd_dflt' in the user table marks whether an unlisted command will be permitted or denied if it doesn't exist explicitly in the node table.  0 is the default value and means non explicit commands will be denied.  Set to 1 and commands not explicitly included will be allowed.
2. The node table contains the command sets applied to users/groups.
3. Entries with the 'service' column set to 52 are returned to the device on exec (eg once logged in).  It's here you can return the priviledge level.  (value1 would be 'priv-lvl' and value is '1' or '15'.
4. Entries in relation to command authorization will have a service value of 56.  Type value set to 57 means permitted, type value set to 58 is denied.

Tracing an Outbound UDP Flood

Had a client running on our virtual machine infrastructure who we saw starting to generate really high outbound traffic.  This in turn would affect other infrastructure as it attempted to cope with 1Gb of outbound traffic.

After some basic analysis we determined the outbound traffic was UDP, originating from the clients server and heading back to the internet (in this case a host in Brazil).  After doing an initial packet capture from the host we were seeing that the UDP Traffic data was all 'X' characters.  Google searches for anything that may generate this kind of traffic was inconslusive,  What I did find however in searching for outbound UDP flooding was a blog entry that discussed the use of a PHP script (http://scott.cm/outbound-dos/) which pointed me in the direction of looking for a script kiddie script.

In going through the packet capture again, I found that we'd actually (luckily) captured the initial http request from the external party who was kicking off the flood (it was luck becuase I was originally only kicking off the capture when I saw the flooding begin on our real time graphing).

What we saw:

http://www.host.sample/403.php?act=phptools&host=x.x.x.x&time=460&port=80

Given that x.x.x.x was then the host that we were flooding UDP out to, kind of a coincidence to see the ip was in the request.

Taking a look on the server then yielded the 403.php file.  It was pretty simple, and ultimately did the following:

$fp = fsockopen('udp://'.$host, $rand, $errno, $errstr, 5);

If you google this string you'll find it around in a number of scripts.  Additional to 403.php we found 404.php which appears to be a gzencoded shell app which once running on the server allows you to do a number of items.  Additional to this was 'a.php' which looks like it was a vulnerability scanner.

Ultimately we cleaned off the scripts, alerted the client of the issue and moved on with real work.

CUCM SQL Query for Users set to reset passwords

The following query run from the SSH CLI will give you a list of users whose accounts are set to enforce a password change:

run sql ccm select u.userid, c.credmustchange from enduser as u inner join credential as c on c.fkenduser = u.pkid where tkcredential = 3 and c.credmustchange = 't'

Debugging VoIP on Cisco Routers

Debugging VoIP on Cisco Routers

Just a basic collection of debugs that may be useful to run when troubleshooting VoIP issues on Cisco Routers.
As per all debugs, enabling debugs on routers can cause CPU overload causing the unit to become unresponsive.  Enable debugs carefully on busy systems.

General Dialpeer / Call Flow

These debugs will give you a call flow including pattern matching on dialpeer's and which was the ingress and egress dial peers:

• debug voip dialpeer inout
• debug voip ccapi inout

H323 Useful Debugs

For a deeper look at H323 setups and call flows:

• debug cch323 h225
• debug h245 asn1
• debug cch323 session
• debug cch323 error

SIP useful debugs

• debug ccsip messages
• debug ccsip ?

Other Notes

• If you're running a newer version of IOS make sure that your Voice service config is allowing traffic through from your remote ip's.  Saw an issue where H323 wasn't showing any debug traffic when a call was being sent through.  Issue was an with voice service not accepting connections because the remote ip wasn't in the list of allowed IP's.
• Dialpeer matching logic useful doc: http://www.cisco.com/en/US/docs/ios/12_3/vvf_c/voice_troubleshooting/old/vts_appa.html

• An awesome debug doc from cisco: https://supportforums.cisco.com/docs/DOC-16310