Default NetScaler changes

When deploying a NetScaler, Citrix recommend that you make these changes by default (https://support.citrix.com/article/CTX121149 ).

The Windows Scaling one was particularly useful on a customer site recently where connections over CAG would drop due to window size on the TCP stream not being negotiated with the ASA filrewall correctly. Enable windows scaling and the issue went right away.

GSLB Site IP Already Exists

So, I’m building a Global Server Load balancing solution based on NetScaler and I made a mistake entering in the IP address for the local GSLB site. I deleted the site and then went to create a new local site but whatever I do the site creation fails with the following error.

 

It turns out that, when the site is created NetScaler records the GSLB local site IP in its list of IP addresses. The RTM version of NetScaler 10.0 (build 54.6) has a bug in that it doesn’t delete this IP address.

Bacause a Global Site IP already exists, you cannot “add” another one. So, if you need to change the IP address used for the local GSLB site, you just need to delete the IP address recorded here and you are good to go. The other choice it to update the firmware to the current version as this bug is fixed in build 54.7.

 

Default URLs and Passwords for Citrix VDI in a Box

Access Gateway Enterprise Edition Nested Group Extraction and Publication of Resources

Netscaler, with is Access Gateway Enterprise Edition (AGEE) functionality, allows you to publish resources to users, such as shares and access to internal web sites, when they are connecting externally to the network. These may include shares which are user specific. Publishing such items is relatively easy; simply create a bookmark that connects to \LocationOfResource%username% for later editions of AGEE or \LocationOfResources#<username> for earlier editions. However, what if the resource is held in a location that depends on the users membership of a department ? For example, finance department home drives are held on one server and sales department home drives are held on a different server ?

Similarly, that should be relatively easy. AGEE will extract the users group membership and you can simply publish the resource to that group on the basis that most administrators create Active Directory groups to represent a department in the organisation. To do this, access the policy manager in the “Access Gateway” node of AGEE.

Right click the Groups node under Configured Policies / Resources and select “Add”.

Then simply enter the name of the Active Directory Group (case sensitive).

Click on create and the group is added. You can then just create a bookmark and drag and drop it on to the group to publish that resource to that group. This works fine if the user is a direct member of the group that has the resource published to them. For example, if the group is “Sales” and the user is a member of “Sales” then they will be able to access the resource.

However, what happens if the user is not a member of the group Sales, but a member of a sub group, for example MajorAccounts ? In this case the user will not be able to access the resource. To overcome this we could just create a group for each resource and add users to each of those groups but for larger organisations that would be an administrative nightmare. Instead, we can use nested group extraction to find the ancestors of those groups which the user is a member of. That is, if the user is a member of the MajorAccounts group and this group is a member of the Sales group then they will have access to any resources published to the Sales group.

Configuring nested group extraction is quite simple. We simply amend the authentication server attached to our policy to enable nested group extraction. For Windows Active Directory the settings are as below:

NOTE: The Maximum Nesting Level setting determines how many levels “up” will be checked. The more levels checked will require more resources form the Netscaler which may have an effect on the scalability of the solution in large, busy deployments.

Now, this is all well and good when it works. But what if it doesn’t ? There is a caveat to this working which is not mentioned in the AGEE documentation. The search scope of the authentication policy must also include the location in Active Directory where the groups are held. For example, if the LDAP server object used by the authentication policy is scoped to the whole domain then all will be fine.

The issues with this is that, technically, this will allow any account within Active Directory to be authenticated, including service accounts and other accounts with Administrative privileges. If these accounts are compromised then, in the case of administrative users, they may also have applications published to them via XenApp which an attacker could utilise. One way to overcome this would be to not publish those applications if an Access Gateway connection is used which is easy enough to do by clearing the appropriate check box in the application properties in XenApp. Far better to never allow logons from those accounts in the first place.

This can be achieved by placing all of the users in a group and scoping the authentication to that group using a search filter. For example, we can create a group call Remote Access and add all the user accounts we want to be able to log in to that group. An example of a string for a search filter is given below:

Again, this is all well and good and easy to set up if you have a discrete set of users you want to grant access to. If it’s every user except administrative accounts then you have to remember to add the user account each and every time a new person joins the organization which is almost certain to fail on occasion. Even worse, if you have a tool for creating user accounts, such as when thousands of students enrol at the start of a new academic year, then this again increases administrative overhead, risks calls being raised where users aren’t added in or requires a re-write of the user creation tool. So, wouldn’t it be nice to use existing groups (departmental groups for example) and add those sub (departmental) groups into this group which grants rights to log on remotely ? That way, you can continue to use your user creation deployment tools and automatically grant these non-administrative accounts the rights to log on when working externally.

Unfortunately, that doesn’t work ! It only works where users are added directly to the group being filtered on, not where the membership of this group is other groups rather than the user accounts themselves. What can be done ?

The solution is to remove the search filter above but only allow certain user groups to log on. To do this we revert the above settings so that our authentication server will have settings as below:

This essentially grants remote logon rights to all users, including administrative accounts. We limit this behaviour by accessing the profile for the session policy applied to the authenticating users. We open the policy manager once again in the “Access Gateway” node of AGEE.

We access the session profile and choose to modify the Request Profile.

On the Security tab of the profile, click on the Advanced link.

Enable the Groups Allowed To Login section and add the name of the group we want to be able to login.

Above I have created a single group and nested sub groups within that group within Active Directory. In the example above, the Remote Access group contains the Sales group. This Sales group in turn contains the sub-groups holding the user accounts such as MajorAccounts, NorthernSales, SouthernSales, EMEA etc. I have also configured Nested Group Extraction as above. Now, if a user is a member of one of the nested groups they are allowed to login. This allows us to scope the search at the root level and thus ensure that all groups (for publishing resources) are within the scope of the search. This civers the situation where thereis a “flat” Active Directory structures where there is no single Organizational Unit which groups together user  group OUs to allow us a common point of entry to conduct searches. As we can now extract nested group membership we can publish resources based on that group membership while restricting logins for administrative accounts and removing the administrative burden of having to individually add users to a group just to allow remote access.

Note: If you prefer, multiple groups can be added to the above field by separating the group names with comma’s.

If you want to troubleshoot nested group extraction, or at least check that groups are being assigned to users, you can use the built in Netscaler tools to monitor the logon process. To do this create an SSH connection to the active Netscaler device using your favourite client, Putty for example. Once you have logged on using the nsroot (or similar) credentials, connect with the operating system shell by typing in shell and pressing return. Then enter the command cat /tmp/aaad.debug and press return.

This will show the debug log for logons. Log on through the AGEE logon page, output similar to the below will be created:

In the example above, the user is only in the ITServices group. This is a nested member of the All Staff group which is itself a member of the Remote Access group. If you cannot see the enumerated ancestor groups (or sufficient groups) then either nested group extraction is misconfigured or you may need to increase the number of levels enumerated.

Optimise XenApp RAM and CPU Part 2

Following my earlier post (https://philipflint.com/2011/12/29/optimise-xenapp-ram-and-cpu/) I have done some digging and, thanks to the hard work done by Jeremy at J House Consulting and the advice he gives (http://www.jhouseconsulting.com/2008/05/13/processor-scheduling-20), I can give what I think is a correct explanation of why the RAM and CPU are optimized in the way they are in the majority of XenApp deployment.

In a Windows 2003 32 bit environment server memory can be optimised either for System Cache or for Programs. The LargeSystemCache registry value ( found ain the registry at HKLMSYSTEMCurrentControlSetControlSession ManagerMemory Management) specifies whether the system maintains a standard size or a large size file system cache, and influences how often the system writes changed pages to disk.

Increasing the size of the file system cache generally improves server performance, but it reduces the physical memory space available to applications and services. Similarly, writing system data less frequently minimizes use of the disk subsystem, but the changed pages occupy memory that might otherwise be used by applications.

The value set for this key can be viewed in the server GUI by accessing the performance settings on the advanced tab of the system properties careen. There are two possible values for this key which are:

Value Meaning
0 Establishes a   standard size file-system cache of approximately 8 MB. The system allows   changed pages to remain in physical memory until the number of available   pages drops to approximately 1,000. This setting is recommended for servers   running applications that do their own memory caching, such as Microsoft SQL   Server, and for applications that perform best with ample memory, such as   Internet Information Services (IIS).
1 Establishes a large   system cache working set that can expand to physical memory, minus 4 MB, if   needed. The system allows changed pages to remain in physical memory until   the number of available pages drops to approximately 250. This setting is   recommended for most computers running Windows Server 2003 on large networks.

 

The LargeSystemCache is allocated from kernel memory area, which is shared with the paged pool memory and system page table entries. In a 32 bit environment paged pool memory is limited to a maximum of approximately 650MB however much physical RAM is installed on the server. Similar limitations exist for system page table entries. Reducing either of these in a Citrix environment may lead to increased paging and poorer performance.

In a 32 bit Citrix Presentation Server environment it is recommended that memory is optimised for Applications over System Cache to ensure that the maximum amount of RAM is available for kernel operations.

In a 64 bit environment the above limits have been increased and while the LargeSystemCache entry still exists in the registry its use has been deprecated.

In a Windows 2003 32 bit environment processor scheduling can be optimised either for Programs or Background Services.

Multi-tasking Operating Systems switch from task to task using various algorithms and heuristics giving the impression that they are multi-tasking whereas they are really processing each thread in turn. The time allocated to processing each thread is determined by the CPU scheduler. The choice of scheduling algorithm can be immensely important when it comes to determining the performance of a Server based Computing system such as Citrix Presentation Server.

Changing the processor scheduling option modifies the Win32PrioritySeparation value under the HKEY_LOCAL_MACHINESystemCurrentControlSetControlPriorityControl key, which consists of 6 bits (AABBCC).

  • Where AA =
    01 – longer timeslice interval
    10 – shorter intervals
  • Where BB =
    01 – timeslice can have variable length
    10 – timeslice has fixed length
  • Where CC =
    00 – foreground/background processes have same priority
    01 – foreground process gets 2 x boost compared to background process
    10 – foreground process gets 3 x boost compared to background process

When you set the performance option for processor scheduling in the GUI, you only see two possible choices for the duration of time allocated to a process:

  1. Programs – Registry value is 38 decimal, binary is 100110 = shorter intervals, variable timeslice length, 3 x boost
  2. Background Services – registry value is 24 decimal, binary is 011000 = longer timeslice interval, timeslice fixed length, no boost

Neither of these settings are optimal for a Remote Desktop Services Server, although the Programs option is the better of the two simply because shorter timeslices are mandatory for a Terminal Server environment.

However, in a Citrix Presentation Server environment the situation changes as Presentation Server provides functionality to optimize the environment over and above that provided by RDS.

If the CPU Utilization Management Feature introduced Citrix Presentation Server 4.0 Enterprise Edition and above has been enabled (which it is by default), the 3 x boost for the default Programs value makes this feature somewhat less effective. In these circumstances the variable timeslice length is often better fixed. Under these conditions it is suggested that the optimum value may indeed be 40 decimal, 101000 binary. That gives us small, fixed length timeslices, allowing the CPU Utilization Management Feature to efficiently do its job by giving each user a fair share of the CPU by modifying the normal job priority scheduling in the operating system.

Setting the Win32PrioritySeparation value to 40 decimal will produce a message similar to the following in the Application Event Logs once the “Citrix CPU Utilization Mgmt/Resource Mgmt” service is next restarted.

———————————————————————

Event Type: Warning
Event Source: CTXCPUUtilMgmt
Event Category: (1)
Event ID: 1591
Date:  22/03/2008
Time:  12:34:28 AM
User:  N/A
Computer: CTX001

Description:
Windows is using a custom priority separation value and CPU Utilization Management performance may be degraded. To optimize CPU Utilization Management performance, on the Advanced tab of the System Properties dialog, open Performance Options and select Background Services. Then restart the CPU Utilization Management service.
———————————————————————

From the above it can be seen that Citrix Best Practice when CPU Utilization Management is enabled is to set Processor Scheduling to favour Background Services. If processor performance remains a bottleneck or a very high number of conext switches are observed then it may be worthwhile changing the value of this key to 40.

Conclusion

In short it can be seen that the “default” settings in a 32 bit XenApp environment should be to optimize RAM for Programs and CPU for Background Services and in a 64 bit environment optimize CPU for Background Services UNLESS CPU Optimization has been disabled.

HDX Mediastream and Adaptive Display

Citrus recommend  that media is streamed to the end point whenever available bandwidth is higher than the bit rate of the video. Where there is not sufficient bandwidth Adaptive Display should be used. But what is the bit rate of differ video types ? There’s a handy table below obtained fromWikipedia:

[edit]World Wide Web HD resolutions

Source Codec Highest resolution (W×H) Total bit rate/bandwidth Video bit rate Audio bit rate
Amazon Video On Demand(formerly “Unbox”) VC-1[2] 1,280×720[3] 2.5 Mbit/s[3]
BBC iPlayer H.264[4] 1,280×720[5] 3.2 Mbit/s[4] 3 Mbit/s[4] 192 kbit/s[4]
Blockbuster Online
CBS.com/TV.com (720p) 1,280×720[6] 2.5 Mbit/s[6]
CBS.com/TV.com (1080p) 1,920×1,080[6] 3.5 Mbit/s[6]
DaCast VP6H.264[7] 7680×4320 5 Mbit/s[8]
Hulu On2 Flash VP6[9] 1,280×720[10] 2.5 Mbit/s[11]
iPlayerHD FLVQuicktimeH.264MP4 H.264[12] 1,920×1,080[13] 5 Mbit/s[14]
iTunes/Apple TV QuickTime H.264[15] 1,280×720[15] 4Mbit/s[16]
Netflix Watch Instantly VC-1[17] 1,280×720[18] 5 Mbit/s[19] 2.6 Mbit/s and 3.8 Mbit/s[20]
PlayStationStore Movies & TV Shows H.264/MPEG-4 AVC[21] 1,920×1,080[21] 8 Mbit/s[21] 256 kbit/s[21]
Vimeo H.264[22] 1,920×1,080[23] 4 Mbit/s[24] 320 kbit/s[25]
Vudu H.264[26] 1,920×1,080[27] 4.5 Mbit/s[28]
Zune Video (formerly “Xbox Live Marketplace Video Store”) 1,920×1,080[29] 3 Mbit/s[30]
YouTube H.264/MPEG-4 AVC 4,096×2,304[31]

As you can see, a user should have a minimum of 2.5Mbps before you even think of redirecting the video.