Tracing an Outbound UDP Flood

Had a client running on our virtual machine infrastructure who we saw starting to generate really high outbound traffic.  This in turn would affect other infrastructure as it attempted to cope with 1Gb of outbound traffic.

After some basic analysis we determined the outbound traffic was UDP, originating from the clients server and heading back to the internet (in this case a host in Brazil).  After doing an initial packet capture from the host we were seeing that the UDP Traffic data was all 'X' characters.  Google searches for anything that may generate this kind of traffic was inconslusive,  What I did find however in searching for outbound UDP flooding was a blog entry that discussed the use of a PHP script (http://scott.cm/outbound-dos/) which pointed me in the direction of looking for a script kiddie script.

In going through the packet capture again, I found that we'd actually (luckily) captured the initial http request from the external party who was kicking off the flood (it was luck becuase I was originally only kicking off the capture when I saw the flooding begin on our real time graphing).

What we saw:

http://www.host.sample/403.php?act=phptools&host=x.x.x.x&time=460&port=80

Given that x.x.x.x was then the host that we were flooding UDP out to, kind of a coincidence to see the ip was in the request.

Taking a look on the server then yielded the 403.php file.  It was pretty simple, and ultimately did the following:

$fp = fsockopen('udp://'.$host, $rand, $errno, $errstr, 5);

If you google this string you'll find it around in a number of scripts.  Additional to 403.php we found 404.php which appears to be a gzencoded shell app which once running on the server allows you to do a number of items.  Additional to this was 'a.php' which looks like it was a vulnerability scanner.

Ultimately we cleaned off the scripts, alerted the client of the issue and moved on with real work.

CUCM SQL Query for Users set to reset passwords

The following query run from the SSH CLI will give you a list of users whose accounts are set to enforce a password change:

run sql ccm select u.userid, c.credmustchange from enduser as u inner join credential as c on c.fkenduser = u.pkid where tkcredential = 3 and c.credmustchange = 't'

Debugging VoIP on Cisco Routers

Debugging VoIP on Cisco Routers

Just a basic collection of debugs that may be useful to run when troubleshooting VoIP issues on Cisco Routers.
As per all debugs, enabling debugs on routers can cause CPU overload causing the unit to become unresponsive.  Enable debugs carefully on busy systems.

General Dialpeer / Call Flow

These debugs will give you a call flow including pattern matching on dialpeer's and which was the ingress and egress dial peers:

• debug voip dialpeer inout
• debug voip ccapi inout

H323 Useful Debugs

For a deeper look at H323 setups and call flows:

• debug cch323 h225
• debug h245 asn1
• debug cch323 session
• debug cch323 error

SIP useful debugs

• debug ccsip messages
• debug ccsip ?

Other Notes

• If you're running a newer version of IOS make sure that your Voice service config is allowing traffic through from your remote ip's.  Saw an issue where H323 wasn't showing any debug traffic when a call was being sent through.  Issue was an with voice service not accepting connections because the remote ip wasn't in the list of allowed IP's.
• Dialpeer matching logic useful doc: http://www.cisco.com/en/US/docs/ios/12_3/vvf_c/voice_troubleshooting/old/vts_appa.html

• An awesome debug doc from cisco: https://supportforums.cisco.com/docs/DOC-16310

Opsview 64bit Agent on Ubuntu 12.04 (Precise)

Had some real issues finding the agent and getting it easily installed on a new install of Ubuntu 12.04 (Precise) I'd stood up as a backup target for UC Backups.

Steps:

• Edit /etc/apt/sources.list and add 'deb http://downloads.opsview.com/opsview-core/latest/apt precise main' at the end of the file.
• run 'apt-get update'.
• Run 'apt-get install opsview-agent'.
• Should be good to go.

CUCM SQL Queries for users with IPCC Extension and Primary Extension asigned

Run the following in a CLI session on your CUCM Server to generate a list of users with IPCC extensions assigned in their user profile:

run sql select e.userid, n.dnorpattern from enduser as e, numplan as n, endusernumplanmap as x where (e.pkid = x.fkenduser and x.fknumplan = n.pkid and tkdnusage=2)

Run the following query for users with primary extensions assigned:

run sql select e.userid, n.dnorpattern from enduser as e, numplan as n, endusernumplanmap as x where (e.pkid = x.fkenduser and x.fknumplan = n.pkid and tkdnusage=1)

Getting Cisco Phone Designer working on x64 Windows 7

Have to give credit to the guys on the cisco forums for this tip.  Cisco Phone designer doesn't support x64 Windows 7.  To get it to run:

• Open up command prompt as administrator and type this command:
• C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Ldr64.exe SetWoW
• Now you should be able to run apps that use only the .NET 32 bit CLR

• To revert back to the default 64 bit framework run
• C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Ldr64.exe Set64

Issues with SCCP Phone Registration through a Juniper Netscreen

Deployed an NG5-GT out to a remote site to get a phone up and running behind it via ipsec VPN.  Tested everything locally before deployment and phone registration was all good.  Once shipped to site, the phone wouldn't register.  We'd see attempts to register in CUCM logs but we'd see it marked as 'rejected' and log analysis would show a Transient Connection and Registration Failure.

Ended up switching off the ALG for SCCP (Security > ALG) and the phone registered fine.

Just in case anyone ever has the same issue.

To login and logout of Extension Mobility via a URL:

https://x.x.x.x:8443/emapp/EMAppServlet?device=SEP6400F1111111&&userid=userid&seq=PIN
https://x.x.x.x:8443/emapp/EMAppServlet?device=SEP6400F1111111&doLogout=true

For when your provisioning app hits the craps and you need to do it remotely :).

9971 Phones need at least one DN assigned to register

Yet another knocking head against the wall issue.  We've been running phones without extensions and EM enabled so that users could log in to get their correct Lines.  Came to put a 9971 into production and the phone wouldn't register.  Status message on the screen was simply 'Phone not registered'.

Went through all the debugging steps.  Status messages on the phone showed no obvious issues.  Trust list was being updated successfully.  TFTP load was happening correctly.  Uploaded new firmware to the cluster and the phones pulled it down correctly.  Basic network traces showed the phone booting, doing 3 dns requests, pulling config down via HTTP from the TFTP server (a few requests) and then heading off to the NTP server.  We never saw any SIP requestes (port 5060).

The only thing of real note was this line in the status logs:

7301 NOT 13:25:22.214160 CVM-cfg.line1NotConfigured: [thread=configmgr MQThread][class=cip.cfg.j] sipMinCfgCheck():Line 1 not configured..

After spending a couple of hours on the phone with TAC, managed to narrow the issue down to the phone having no DN assigned.  Assigned a DN and the phone registers fine.

If you hit this issue and are using EM, don't forget to update the EM logout profile on the phone, it takes precedence over the physical phone profile.  If you only update the physical device record and not the Device Profile it won't register.

Hope this saves someone some time.

Ending a Call via URL with CUCM Webdialer

Had a customer who was wanting to integrate their custom CRM application with CUCM.  Easiest way to do this was via the Webdialer.  Their current setup was to use a modem placed in front of an analogue phone which was then controlled by the CRM app via a dialer string.

To use the webdialer process we can fire off the the URL 'https://x.x.x.x:8443/webdialer/Webdialer?destination=0123456789' from within the application to cause a call to kick off. (Note that you need to have logged the end user into webdialer at least once previously so that settings and login details are saved).

Next request was to be able to hang the call up via a button in the app so that the call center agent never has to physically touch the phone (or softphone) to hang up calls.

After mucking around with Fiddler, seems that you can use the following url string to accomplish the hangup: 'https://x.x.x.x:8443/webdialer/Webdialer?cmd=doEndCall'.  This will hang up the call (assuming there is one in progress) on the device managed by the webdialer setup.  If no call is in progress then nothing untoward happens.

Now to get the CRM app updated with this code and we'll see how it goes.