syslog

Tuesday saw this year’s UAS Conference taking place in the University Examination Schools. This annual event brings together staff from around the University with the aim of exchanging information about things happening in the University Administration and Services Division. OUCS officially joined UAS in January of this year (moving from ASUC), so although we have been invited to present / attend in the past, there was perhaps a sense in which our presence was cast in a different light this time around.

The UAS Conference and ICTF Conference are quite different affairs. Where the ICTF Conference has a backbone of plenaries with workshops woven around them, the UAS Conference programme is much more open with up to 11 concurrent workshops (either 30 minutes or hour-long). These are organised in four hour-long slots, with an opening plenary (this year given by the Registrar under whom UAS sits), and a closing panel Q&A session.

Workshop topics are quite diverse, as you might expect given the diversity of operations within UAS itself. Some, such as “Using assertiveness to improve communication in the workplace”, are bite-sized training sessions, whilst others such as “X5 Project: an update”, aim to communicate about current activities and future plans.

I made it along to 5 sessions, interspersed with lots of ad hoc discussions as I bumped into colleagues from all parts of the University.

Toward Coordinated Central ICT provided an update on work to bring together BSP, ICTST, and OUCS. The presentation was given by Anne Trefethen who is leading the project, and outlined key features of the project and current thinking. The project’s objectives are to create a more coordinated service for users, and to increase efficiency and effectiveness. A high-level schedule was outlined: identify opportunies 0 – 6 months; quick wins, planning, revised governance model, and initial consolidation 6 – 12 months; implementation of plans 12 – 24 months; completing implementation of new organisational structure 24 – 36 months, and we were given an indication of what the revised governance model might look like.

Understanding the JRAM and the Infrastructure Charge – second time round for me – showed how incoming funds such as student fees are divided and distributed amongst the academic divisions, departments, and colleges, and how a portion of this is then claimed back from divisions to fund central University operations. Some fine animated visuals made it easy to understand what is in fact quite a complex model (the JRAM). The workshop went on to describe how the 123 Infrastructure Charge then determines how much should be pulled back from academic departments, and on this I will attempt to provide a summary.

The 123 Infrastructure Charge works on the basis that the cost of centrally provided services are related to academic departments in one of 3 main ways. Type 1 services (e.g. divisional office) are consumed by specific divisions, so the cost is essentially a direct cost to that division. Type 2 services (e.g. student admin, estates) are shared by several departments, and the cost is distributed amongst these according to an appropriate measure of “level of use” which may vary depending on the service (e.g. student numbers, floor space, annual turnover, etc). Type 3 services do not have a usage-based link to divisions, and are subdivided into those for which the divisions can reasonably be expected to pay (e.g. VC/Registrar, PRAS), and those for which divisions cannot be expected to pay (e.g. University Parks, museums). In a slightly Douglas Adams-esque fashion, the 123 Infrastructure Charge includes a fourth category of “service to service” charge where the costs of a service which is only consumed indirectly by divisions (such as BSP) is redistributed across the directly consumed services prior to the apportionment calculation.

The Worst Website in the World made extensive – almost exclusive – use of audience participation to describe the most annoying web site we could cumulatively imagine. By suggesting how to achieve the opposite of each annoyance we then discovered how to create something which isn’t the worst website in the world. The notes that we typed up will be published soon on a …website… near you!

What’s Religion and Belief got to do with me? This workshop, offered by the Equality and Diversity Unit, raised awareness of legislation changes introduced in the Equality Act 2010, and ongoing work to explore perceptions of how well the University understands accommodates those with particular religions and belief systems. An interesting session from which I took away two points: one of the areas most commonly raised as a problem by survey respondents was college food, and the Equality and Diversity Office is an excellent source of advice and assistance.

Sharing Services: what do you think we can do? was another facilitated discussion, with the audience encouraged to think about the opportunities for greater sharing of services within the University, what could be stopped in order to release resources to do this, and how to get the ball rolling. What made this particularly interesting for me – coming from OUCS – was hearing about the opportunities for non-IT services to be shared.

The day was rounded off with a panel Q&A session. The questions were the sort of thing you might expect – “What do you see as the biggest challenges / opportunities facing us”, “How can a democratic governance ensure that every voice is heard”, … However it was two of the panels responses that I will remember.

The first was in response to a question about colleges and the university working together – the push and pull of centralisation. Tim Gardam provided one of the best expressions of the value in Oxford’s organisational (un)structure. He said that the University structure is two-dimensional – it could be viewed as a vertical structure within the divisions, cross-cut by the colleges. This is a very modern (matrix) structure, and provides great resilience because a vulnerability to current conditions in one structure can be bolstered by strength in the other. He went on to say that this also enables the Collegiate University to provide students, tutors, and researchers, with an experience that is both “big and small” – vast yet personal.

The second was from a panel member who joined the University just 7 weeks ago. Her comment was that Oxford “is like no other place”. Yes indeed, there’s no cookie-cutter for Oxford, despite what some consultants would have us believe – long live innovation!

Request Tracker is an open source issue ticketing software which we use extensively in OUCS, and Sysdev are responsible for managing it. In recent weeks, I found myself discussing some of the ways we use RT with people from the RT user and developer community, and I thought it might be interesting to share some details here.

I’m not going to provide a review or detailed description of RT as such, but the basis is a database-driven application which creates and manages tickets (the basic unit which which is analagous to a bug report, user enquiry, incident, as you prefer). The primary interface for users is incoming and outgoing email, and a web interface is provided to privileged (staff) users of the system. (It also offers a self-service web interface to end-users, which we haven’t deployed.)

RT has been around for a relatively long time, and OUCS began using it relatively early, too. The first ticket still in our main RT installation dates from September 2001, not far off 10 years ago. Initially, RT was deployed as a way for the help desk to track incoming enquiries, but since then its use has grown to all sorts of other areas, with various systems teams including ourselves using it for internal incident management, and so on. RT now handles a fairly significant volume of email, critical to OUCS’s business (providing IT services to the University).

When I first started working at OUCS, the RT system was already well-established, using RT 2, which was by that point fairly long in the tooth (the first release of RT 3 was four years previously in 2003). It had already handled more then a million tickets.

As RT is implemented in Perl, and Sysdev has always been fairly comfortable with hacking Perl, various customisations were implemented to handle specific requirements. Unfortunately, those changes were not in general made in collaboration with the developer community, largely because RT had moved on so much (some “interesting” hacks were made in order to overcome some severe performance problems, for example). When we finally managed to get time to bring our installation up to date around two years ago, a fair bit of software archaeology was required to find out what changes we had, and why!

One of the pleasant side-effects of Sysdev working on RT has been the chance to work on, and improve the packaging of RT in Debian (our Linux distribution of choice for deploying services on). Since I am a registered Debian Developer, I was in a good position to join the packaging team in Debian which had suffered from lack of sustained interest. This provided an excellent excuse to get stuck into updating the RT packages to 3.8, which I did, based on preparatory work by others. Interestingly, I wasn’t the first Sysdev team member to be involved in RT packaging for Debian; inspection of the older changelogs reveals work done by one of my predecessors. Being involved in the official packaging project has also meant that I’ve become reasonably familiar with RT4, which has been out for a few months now, even though we’re a little way off deploying that at OUCS.

Once we started to analyse how an upgrade to RT 3.8 would look at OUCS, and discussed the system with others around the department, we found that many of the tweaks added either weren’t important, or had been superceded by other changes in RT, but there were still quite a few customisations required. Since the migration, I’ve tried hard to make sure that issues we face are tracked in the upstream bug tracking system, and that patches we write are both manageable (so that they don’t present such an obstacle to future upgrades) and where possible, general enough to be fed back upstream. Inevitably this doesn’t always happen, of course — at the time of writing, we have 13 patches to the core (on top of 18 applied by Debian). These cover trivial things like adding extra convenience links to ticket display pages, to bugfixes which only arise because of some of the “interesting” ways we use RT, to a patch set originally taken from the “QueueDeactivatedScrips” extension.

One example of the “interesting” ways we use RT is in our handling of bounces; RT3, unlike RT2, doesn’t by default support routing bounce messages relating to tickets back into RT, but because of the range of internal users of the system, we needed to be able to continue to support this functionality. What we’ve ended up with is an exim filter rule, together with a scrip based on this one from the RT wiki (another very useful community resource) with some additions which email our staff users to let them know about the incoming bounce, and with a patch to ensure that bounces get routed to the right place.

It’s worth mentioning that RT supports quite a powerful overlays system which allows for plugins to cleanly implement a certain amount of new functionality without having to patch the core codebase. Some of our work is done this way, but a few more changes have been made directly, as discussed above. Nowadays we have much better ways for managing patchsets on top of official Debian packages, which makes this a less worrying prospect than it was in our RT2 days.

Our migration project also involved a fair bit of work to get the rt2tort3 scripts up and running with RT3.8 (I suspect we were one of the first to do so — after all, most people hopefully got round to the migration long ago!). That work is available as a forked github project.

The nice thing about working on RT at OUCS is that it’s an important tool used by a significant proportion of OUCS staff for their daily work, all day every day. This results in a large number of detailed change requests, which we have been diligently collating and assessing for implementation. Due to other workloads we haven’t been as responsive to those requests as we’d like, but a team internal ‘hackfest’ is planned for next year in order to hopefully address some of those (and probably also to move to RT4).

Another nice thing about RT is working with a great bunch of individuals both at Best Practical, who are the core maintainers of RT, and the external user and developer communities, both on IRC, mailing lists and wikis. This has made it especially rewarding working on the official Debian packages, as well as deploying RT here at OUCS. It’s an excellent example of Free/Libre Software working for all parties, with the users contributing valuable code back to the project, and I’m pleased to be able to participate in the free exchange of ideas and code, with the support of the OUCS OSS policy, which makes releasing code under Free/Open licences fairly trouble-free.

Coincidentally, I was able to meet up with the Best Practical team in Cambridge, MA, USA last year whilst attending the MIT Kerberos Conference; those sort of face-to-face meetings with collaborators in free software are an excellent way of getting to know the people at the other end of an IRC client or mailing list.

Crucially, they recommended a bar serving excellent beer.

OUCS has initiated a project to investigate a central Active Directory for Oxford University. This is intended to bring enhanced support for integration of Microsoft AD based environments with central identity and access management services. At a very basic level it is hoped that this will offer a Microsoft-supported mechanism for providing single sign-on to systems running a wide variety of Microsoft products.

The ability to offer initial sign-on to Microsoft workstations (through the Welcome screen / GINA) and seamless subsequent access to domain-based resources has been available for several years (see https://wiki.oucs.ox.ac.uk/itss/KerberosADTrust). Active Directory uses the Kerberos (version 5) protocol for user authentication  – this has been the case since Active Directory was launched in Windows 2000 (see, e.g. http://technet.microsoft.com/en-us/library/bb742516.aspx), and Microsoft recommends Kerberos rather than NTLM (http://msdn.microsoft.com/en-us/library/aa378749%28VS.85%29.aspx).

Microsoft’s Kerberos implementation makes use of some optional features – worthy of note is the inclusion of a Privilege Account Certificate (PAC) in the service ticket to convey information about SIDs and group memberships – but is still fully compatible with other popular Kerberos implementations such as MIT-Kerberos. As such, it is relatively straight forward to configure a “cross-realm trust” so that users can login using their credentials in the Oxford Kerberos realm (OX.AC.UK – used for Oxford SSO authentication) to login to an Active Directory domain. There are currently around 35 Active Directory domains registered for cross-realm trust.

Technically then, a working solution is already in place – applications that are designed to run in an Active Directory environment appear to work smoothly in this configuration and the user gets the single sign-on experience that they want. There are, however two issues that crop up:

• There is very little up-to-date information about the bigger picture in which these trust relationships are implemented – an end-to-end analysis needs to look at server roles & applications, client systems, and the federated nature of IT service provision at Oxford.
• Although Microsoft officially supports Kerberos trust relationships between two Microsoft Active Directory domains, they are far less clear about their position regarding trust relationships with other Kerberos implementations. There is a risk that the cross-realm trust could stop working following a hotfix, service pack, or platform upgrade.

Project MADDOX is a two-stage project designed to address both of these issues.

The first stage of the project involves building numerous environments incorporating Active Directory, server, and client components in differing configurations to find out what does, and does not, work. Tests will compare the results of Active Directory to MIT-Kerberos cross-realm trusts against Active Directory to Active Directory trusts (which are fully supported by Microsoft). It is expected that this stage will be completed towards the end of August 2011.

Following this, it is expected that a top-level Active Directory “authentication” domain will be created to support user authentication to other Active Directory domains deployed around the University (see http://technet.microsoft.com/en-us/library/cc757352%28WS.10%29.aspx for an outline of domain trust models). This will form an addition to the suite of identity and access management services offered by OUCS, offered initially as a pilot to establish demand, expected to be available towards the middle of Michaelmas Term 2011.

Service Level Descriptions for sysdev-owned services have been updated on the OUCS web site. This is an annual activity, typically falling in May/June, when SLDs for all OUCS services are reviewed, updated, and published. It is an opportunity for us to ensure that any changes to the service level offered are reflected in the SLD so that users and potential users of the service can assess its suitability for any given purpose.

Summary of Changes to Existing SLDs

Oak LDAP · Webauth · Shibboleth · OxFile · Web Publishing

This year we have focussed on making the SLDs more useful to end-users. With this aim we have improved several of the introductions/overviews to clearly state the purpose (supported business process) that the service is designed for, and to clarify the user support arrangements. The introductions typically include a link to a more thorough description of the service, and identifies (in our case) sysdev as the service owner.

Most of our SLDs now sport a clearer statement of the “Hours of Operation” to make it clear when the service is intended to be available for use, when technical support is available to deal with faults, and the usual schedule for any maintenance work.

Information about major outages is now contained in a “Disaster Recovery” section which provides two key details: (1) the state in which the service will be once it has been recovered (i.e. what data will have been restored), and (2) how long this might take. We hope that these will prove more useful than some dry technical details about what RAID levels and hardware models have been used to deliver this.

New SLDs

Maillist · News · Mirror · GNU/Linux Computing

Service Levels Descriptions have been written for a number of services that have actually been available from OUCS for quite some time, but until now have not been decorated with the appropriate documentation. In some cases this was because the facility had not actually be recognised as a service, in others it was a simply case of priorities leaving this work far down the list of to-dos.

Service Descriptions

We have also started to provide a more complete description of each service. The Service Description and Service Level Description documents are complementary, with the first providing information about the purpose of the service and what it offers, while the second  details the warranty under which the service is offered so that you can assess whether it is fit to use.

Further Improvements

Our work here is not yet complete. During the updates of the SLDs this year we identified further areas for improvement, and there are still some inconsistencies that we are yet to iron out. Although we have written service descriptions for several of our more popular services, some are still missing. Perhaps the largest change is that we are planning to present the suite of services that support identity and access management as just that – a suite of IAM Services – rather than a piecemeal listing of the various technologies on which these sit.

As already referred to in recent blog posts, and thanks to OUCS, some members of Sysdev, myself included, were fortunate enough to attend the UKUUG 2011 Spring Conference. Among the many talks one which particularly stood out for me was a talk given by Simon Wilkinson from the University of Edinburgh about the Git Distributed Version Control System (DVCS). Although this was a 90 minute introductory talk its level was well judged, introducing the fundamental topics that a user currently familiar with a Centralised VCS would need to understand to start using the Git VCS by stepping through a set of basic workflows, while explaining some of the different distributed development file versioning scenarios using directed graphs.

In case this is a bit unfamiliar to you, a VCS is invaluable to developers as it’s where all the shared resources for a project are stored for safekeeping. It records not only source code, configuration, and documentation, but also each and every change in those project resources, so changes can be tracked and reverted if necessary. It’s called ‘Version Control’ because it allows, and promotes, application of your chosen versioning scheme to files. It’s also sometimes called ‘Source Control’, because it is commonly used by software developers to manage changes in their source code. This blog post considers the VCS from the multiuser point-of-view but a VCS can also be invaluable for a single-developer project.

My own experience with Version Control traces a path through RCS, CVS, and Subversion, each of which build on the previous to develop the sense of working with a single shared, centralised repository. In this sense they (particularly CVS and Subversion) are Centralised VCSs. Each user working with a Centralised VCS typically checks out a particular current set of files from the central repository before making changes. After making changes locally the user then accesses the shared repository across the network for VCS interactions such as comparisons, commits, updates, etc. For any activity that needs access to the central repository the user is competing with others for access to the same shared resource. Since the resource is shared it carries with it all the baggage and overheads of locking to prevent leaving the resources in an inconsistent state.

Git has a handy solution to this: when you ‘check out’ a project repository using Git you are actually ‘cloning’ it – you get a backup of the repository, lock, stock and barrel, including all the history. This neatly side-steps some of the contention and locking overheads, achieving some noticeable performance gains since it makes many operations local which would, with Subversion, have been remote repository interactions. This is one way in which Git achieves some significant performance gains but this is not the only difference. There are other features that Git includes to make VCS users happier that seem to have been missing from Centralised VCSs, such as squashed commits, simpler and less resource-heavy branching, and simpler repository file formats. Yet another benefit that Git brings to the table is that equivalent repositories are much smaller than other VCSs.

Sysdev currently run a Subversion VCS service for use by OUCS staff and OUCS-run projects. We also count ourselves among the users of this service, since it is used by our home-grown configuration management system rb3, as well as for the projects undertaken by Sysdev staff. Subversion was originally designed as a largely compatible replacement for its predecessor, CVS, and as such, it shares much in common with CVS syntax. In turn, there is some familiarity to Subversion users in the syntax that the Git developers have chosen for the Git toolset. Tools are available to migrate repositories from Subversion to Git (git-svn), just as tools were developed for migration from CVS to Subversion (cvs2svn). However, apparently Git’s design goals were not primarily to be a direct Subversion replacement, and when you are used to a Centralised VCS using a DVCS can take a bit of getting used to. I’m no expert Git user, but as a long-standing user of other VCSs I believe that Git has speed and efficiency improvements, along with other convenience benefits, that are well worth consideration by our users. Naturally, for a team such as ours to consider providing a DVCS service such as Git the demand for it would have to be expressed by our users, and assessed.

Git is an example of a project that was born from Linux kernel developers wanting a better tool, and as such it fits well in a UKUUG Conference, but at its core Git has much wider appeal to developers in general. This seems to me to be typical of the UKUUG Conference. If you’re not part of a UNIX team you may think that a conference organised by a UNIX users group is too specialised to be relevant, and perhaps not really your thing. If you think along those lines I’d like to suggest some reconsideration as these are useful events, with talks likely to be relevant and of interest to a broad range of IT professionals. As if to highlight that a recent announcement was made: the UKUUG have changed their name to FLOSS UK, which stands for Free and Libre Open Source Software, the name change reflecting a broader scope and a correspondingly wider appeal.