+420 273 130 426
Copyright ©
2024 EXPINIT, s.r.o..
All rights reserved
Recently, as a Citrix partner, we were notified on a priority basis that the following vulnerabilities exist in specific versions of Citrix NetScaler, which was subsequently confirmed by checking the relevant GWs in Citrix ADM.
Since there are no so-called “mitigation steps” for these CVEs, the only solution was to upgrade the firmware version of the respective NetScaler GWs – in our case it was two VPX appliance running on the Hyper-V platform (which is probably not the most important information). In this particular case, the two appliance are running separately, meaning they are not connected in an HA pair. Each running a different firmware version – 12.x-yy and 13.x -yy. Since we have been performing such upgrades since NetScaler was still Citrix Access Gateway, we suspected that it would not be as easy as the manufacturer claims and after several weeks of email correspondence with Citrix support, we decided to perform the upgrade on our own – the already proven way. It should be noted here that both GWs are highly customized, from the themes used to the configuration. Thus, the classic process of uploading a package with the latest FW (trying to perform this operation via the GUI is out of the question) was not an option. During the first bold attempts (after all, you need to give the official procedure a chance at least once) both appliance ended up with the error “<daemon.err> monit[216]: ‘httpd’ failed to start”. Citrix tech support of course requested logs and other information from the upgraded appliance. What the heck, one could provide this information without any problems, unfortunately one could not log into the appliance after the upgrade. No way – directly on the console, via SSH, or via the web interface, which according to the bug mentioned above was of course completely inaccessible, just like all vServers. Enough, that’s not the way to go. Let’s start from scratch by cleanly installing a new appliance in the latest available version, transferring the configuration and other essentials such as graphical themes of each web interface, certificates and in short everything needed to get the environment working again. I highly recommend not to use the built-in configuration transfer options, e.g. as described in the following blog it will not work as expected again. Partial inspiration can be drawn from the blog by using the “Running configuration” export to a text file. This file then contains a sequence of commands allowing the CLI to set the new appliance to the desired state. It should be noted, however, that ctrl+a –> SSH session –> paste again will not solve the seamless transfer of the configuration. One of the pitfalls in the exported file is, for example, a sequence of commands that do not logically follow each other and use entities or configuration items that do not yet exist and are created in subsequent steps, refer to non-existent certificates that need to be uploaded manually, and other similar “niceties”. Attention is also required by the (likely) change of the encryption algorithm or hashing methods, so the transfer of user passwords or LDAP passwords or other accounts is again a mystery. It’s also good to remember that in the latest versions of Citrix NetScaler you can configure an option regarding the use of SSL profiles for individual virtual servers, which is a rather useful feature when using different certificates and CIPHER group x number of virtual servers combinations. Bottom line is that we were not able to upgrade any of the appliance in any of the official ways without any problems, even with the help (if I can say so) of the Citrix NetScaler support experts. In our case, the only possible way was partial transfer of configuration to the new appliance, including immediate reverse checking of the performed steps, mainly due to unstructured commands contained in the exported configuration file. The result of the efforts and partial satisfaction was a fully functional appliance running on the latest firmware version with the required configuration. Due to the virtualization of the solution and the fact that the entire process was carried out in LAB, which is identical to the live environment in terms of setting up the network infrastructure, the transfer to production was a matter of moments. However, the question that never ceases to bother me is how one would restore the configuration of such an environment using the build-in tools, i.e. by backing up and then uploading the “Running configuration”. …and in our case the configuration file of one appliance had “only” 2600 lines. 🙂 For more knowledge not only from the world of Citrix looking forward to reading Jan HanuÅ¡.