Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: At a Glance
NCR, Dayton Ohio, USA
Top End, version 2.02.02
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: The Ovum verdict
Top End is an extraordinary product. It appears to have everything a customer needs from a middleware product (let alone a distributed transaction processing product). There are superb load balancing and failover capabilities, well developed security, a well thought out architecture geared towards enterprise level use, excellent administration tools, support for a large number of platforms, a committed vendor, good worldwide support and consultancy services, the ability to customise the product, support for version control, internationalisation support, good development tools and support for a range of popular third-party development tools.
Top End (coupled with the companion LifeKeeper) probably has the best automatic support for load balancing, restart, failover and recovery facilities of any of the middleware products we have seen. NCR has struck just the right balance between high performance and high availability. Top End is not just best of class in this area, it is the best on the middleware market.
Top End could justifiably compete with DCE in its support for enterprise level computing with networks consisting of thousands of nodes, and it provides far more facilities than DCE. It does not support the vast number of platforms that DCE does, but its directory service is likely to be less time consuming to administer, the API the developer uses is simpler, both synchronous and asynchronous communication are supported, the security is as good as DCE's and easier to implement, and the performance is likely to be better because of the load balancing and automatic restart capabilities. If NCR added a time service, Top End would not just be equal with DCE in its support for large scale global computing, but a long way ahead.
Another key strength of Top End is its unique 'veneer' capability. Other vendors of DTPM products are reliant on being able to persuade the vendors of DBMSs and resource manager packages to provide support for XA. NCR, by providing its own veneer, can support any resource manager its customers require. We remain sceptical about whether the veneer would work with all resource managers, but in general this is a good solution to a tricky problem.
The one significant weakness in Top End is its support for guaranteed delivery. Generally, guaranteed delivery and distributed transaction processing are almost incompatible objectives, as to fully guarantee that a message reaches its destination (whatever failure occurs and without needing the application processes to re-send on failure), the middleware has to employ mechanisms that may slow performance to unacceptable levels and may, in addition, cause significant performance degradation of the DBMS. NCR should have added more programmer-controlled options to allow developers to add more safeguards en route. For example, the memory-based sending or receiving queues could be logged. There could also be more hand-shaking between network interfaces and node managers to track the course of a message to determine its status and position.
Minor weaknesses in Top End are that it only supports TCP/IP (it really needs to provide full support for SNA, not just a gateway) and that many of the enhancements and products described in this issue have only just been released (with the normal risk associated with new products). We would also like to see LifeKeeper on more platforms.
Support for some key platforms is also missing, in particular Digital's OpenVMS. Furthermore, despite claims to the contrary by NCR, there have been reports that not all platforms install with equal ease, and that the functionality on some platforms is different (though being harmonised in the next release).
NCR also relies on third parties to port some of its key software to platforms such as AIX, MVS, HP-UX and Solaris. A company such as NCR should not be reliant on third parties for ports to these important platforms. The development of MVS support in particular, needs to be brought in-house. In general, support for the MVS platform has been weak up to now, but is being corrected with the use of the remote server. However, more development work is needed to provide the full Top End base services.
Top End is NCR's only strategic middleware product (it does have a port of DCE to NCR systems, but NCR only invests its development resources in Top End). As such, it is wholly committed to its success. There are no divided loyalties within its development groups and there is no need for the company to split its resources. Furthermore, the marketing message is as clear as a bell - whatever application you are developing, use Top End!
NCR tends to focus on selling to the 'enterprise solution' market, but we wonder why it has limited itself - it could capture a far larger market share with its product.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: When to use
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Product overview
Top End is a message-based distributed transaction processing monitor, which supports the XA standard. It provides disk- and memory-based queues and uses NCR's own high performance memory-based, self replicating, self-registering directory.
Top End was developed from the start to be infrastructural software that could be used across an enterprise to support large, high volume, high performance, mission critical, distributed applications.
Top End is geared towards support for transactions that involve resource managers using the XA standard. 'Resource managers' can (in theory) be queue managers, DBMSs, or even print queue managers, but in practice, support for the XA standard tends mainly to be found in DBMSs and file managers. Top End (like all DTPM products that support XA) is oriented towards supporting transactions involving distributed units of work that update XA compliant databases. It achieves the co-ordination of the transaction and rollback actions by communicating with the resource manager (DBMS) using the XA standard.
The DBMSs Top End supports through the XA standard are Oracle, Informix, Gresham-XA, DB2/6000 and Microsoft's SQL Server.
Top End differs from the other DTPM products in this issue as it is the only product to provide 'veneers' to DBMSs that do not have a XA capability of their own.
A XA veneer is like a translation tool. It takes the XA commands issued by the transaction manager and translates them into the commands expected by the DBMS. Through this approach, NCR has been able to support DBMSs that do not support the XA standard.
NCR currently use its XA veneers to access Sybase, Teradata and Ingres.
As shown in Figure E4.1, an application communicates with the resource manager using SQL, while the transaction manager uses the TX standard and the communications resource manager uses the client-server interface (CSI). CSI is Top End's own API to this component. The transaction manager and communications resource manager communicate using the 'XA plus' standard. This interface is invisible to the programmers.
Although CSI is the interface normally used by the programmer, NCR is starting to use 'veneers' to CSI that enable the developer to use other APIs. It has already provided a veneer for DCE's TXRPC, and one for ActiveX objects is planned. The developer uses TXRPC or the ActiveX interface and the calls are translated to equivalent Top End calls. In the case of TXRPC, the veneer provides users with a migration path from DCE-based products. In the case of ActiveX, the veneer enables any tool that supports ActiveX components to be used with Top End.
Top End uses the concepts of a product (or server) and services (or functions).
A server is the name given to a collection of functions. For example, the server name bank account could be used to group a set of functions which open, close, debit or credit the account and so on. In object-oriented terms this would correlate to the idea of an object with its methods.
Each server is defined by its functions, but a function can be used by more than one server.
Top End clients invoke functions by using the name of the server and then the name of the function they wish to invoke.
Top End can be used to develop two-tier client-server applications, peer-to-peer applications and several other topologies. It is especially suited, however, to the development of three-tier applications with 'thin' clients accessing business logic servers, which in turn access database servers using the transaction processing services. This approach to distribution is shown in Figure 2, along with the DBMSs and file systems that Top End can support using XA, and the tools that can be used to develop Top End applications.
Top End can be used in three-tier applications that invoke a database server to handle the database calls. Business logic server can also send SQL calls directly to the DBMS. In this case, the developer can use a component (not shown in Figure E4.2) called the RDA agent. The RDA agent's purpose is to receive SQL calls from an OSI network, pass them to the DBMS for processing and then return the response.
Top End consists of three main types of component:
Each machine normally has one copy of the Top End Base services software, but it is possible to run additional Base servers on the same machine, to support for example, test and production systems, or even different versions of the software.
An example configuration is shown in Figure E4.3.
The Top End Base service component runs on the Unix and Windows NT platforms, but there are some differences in support between the Unix and Windows NT platform. The Windows NT platform does not support message sensitive routing, the non-XA-compliant database veneer, OSI, managed servers for Cobol, login clients, extended recovery or support for special devices. NCR states that support for the additional services - security, remote servers and the LU6.2 open gateway - will be available on Windows NT 'as the market dictates'.
The Top End Base Server core components are the Top End node manager, Sysmon (the System Monitor) and the network interface component. Customers can add other optional services, such as security.
The node manager monitors applications, supports the sign-on and sign-off of applications, manages transactions, and handles logging and failure recovery, client and server requests, security, run-time administration and application component control, as well as co-ordinating transmissions.
The network interface handles the transmission and 'listening' services. Each network interface on a node interfaces directly with a network interface on the other server nodes it is connected to.
On the start up of a node, the node manager initiates the network interface and other services. Servers can then register themselves with the node manager. When they are loaded, they sign-on using their server name and advertise the functions they offer. The node manager stores the details of the servers that have registered with it in a memory-based directory. This directory is reconstituted from scratch each time a node is re-booted or a new node is added.
Once the servers have signed on, the node manager advertises the servers it has on its node to all the other nodes it is connected to via the network interface. The node managers on these nodes then add the servers, their functions and physical addresses to their directory. The propagation of changes is then passed on to the other nodes in the network, so every node has a directory containing the physical addresses of all the registered servers (including its own), together with their functions. Even if a server is found on more than one node, the directory is able to distinguish between the copies on each node and between copies of the same server on the same node.
By using 'heartbeat' communication to keep in touch with other nodes, the node managers are able to detect a node failure. They will then update their respective directories accordingly, as well as informing the other node managers of the change. The same procedure occurs if servers fail, are removed or close themselves down.
New nodes and servers can be added at run-time, as and when needed. Top End Base Server nodes communicate with one another in a peer-to-peer arrangement to ensure that the directories are all up-to-date, the services are all registered and messages are managed around the network.
Two types of memory-based queue are allocated to each server - the 'shared queue' and the 'private queue'.
Both these queues are used for messages going to the server, in other words, the queue is at the receiving end and is not employed if the server end is used in client mode.
The queue used to store messages that are to be sent from a node, is called the 'CSP queue'. It too is memory-based and is managed by the CSP component of the node manager. It is effectively a transmission queue, used to store all the messages from several clients. This queue is also used, however, to store messages that were received prior to them being passed to the shared and private queues of server processes. It is also used to store the replies that are to be passed back to the client.
When a client sends a message, it is first placed on the CSP queue by the node manager. The node manager passes the message to the network interface components, which transmit the message across the network. The network interface component at the receiving end passes the message to the node manager, which in turn places the message temporarily on the CSP queue. The message is then distributed to the appropriate shared or private queue of the server.
When the server replies, the CSP component of the Node Manager places the reply temporarily in its CSP queue. The Node Interface components then transmit the message across the network, where the reply is stored in the CSP queue of the receiving Node Manager. Once the client issues the receive command, the reply is passed to the client.
Developers can choose between 'robust path' message queues and 'express path' message queues. Express path message queues are shared memory implementations. On Unix platforms, robust path queues are based on IPC queues. On Windows NT, NCR has developed its own IPC-equivalent queuing mechanism, called Top End IPCQ: this is a multi-threaded queuing service.
Most developers initially use the robust path queue and if performance needs to be improved they switch to express path. NCR estimates that performance improvements of about 10% are achieved using the express path.
The recoverable transaction queue (RTQ) is used to safe store messages over unreliable lines, or provide a staging post for messages travelling over a WAN.
This queue is disk-based and can be located on any node chosen by the administrator - its own secure node, a client node or a server node. The only proviso is that the RTQ service must be able to support the Top End Base services component, of which it is an optional part.
Any number of clients or client copies on any node can place messages onto the queue and any number of servers or server copies on any node can get messages from it. The administrator can choose to have as many RTQs as the applications need.
The RTQ service consists of two components - a 'server', which puts the messages received onto the queue, and a 'worker', which takes them off.
If messages are to be sent via the RTQ, the client sends a message addressed to the final server, but with instructions to enqueue it on the RTQ. The message is first placed on the CSP queue by the node manager. The node manager passes the message to the network interface components, which transmit the message across the network. The network interface component at the RTQ end passes the message to the RTQ server, which then places the message on the RTQ.
At this point, the RTQ worker takes the message and attempts to deliver it (via the remote network interface component of the application server's node) to the remote node manager. Once the line or node is available, the network interface component delivers the message to the remote node manager where the message is stored temporarily on the CSP queue. The message is then distributed to the appropriate shared or private queue of the server. If a reply is issued via the RTQ, the reverse process is followed.
RTQs are under transaction control. When a client issues the commit command, the messages sent within the transaction must have been stored safely on the queue. If not, all the other actions in the transaction are rolled back and any messages put on the queue successfully (up to the point of a failed enqueue) are removed and passed back to the client.
A second transaction is used to dequeue the messages. This too can be under transaction control, so that if the dequeue process fails, the rest of the transaction is deemed to have failed and other actions are rolled back. The sequence of transactions is always exactly replicated so that the order the transactions were enqueued and dequeued in is preserved.
The purpose of the remote server software is to provide support for platforms not supported by the Top End Base services software. It is used to connect Unix and Windows NT machines to proprietary machines such as MVS using TCP/IP. The remote server machines supported by NCR are MVS and OS/2, but support is planned for other machines (Tandem Guardian, AS/400 and Unisys 2200). This software is provided by a third party, EnterSoft Corporation.
Although called remote server software, the application on the mainframe can be a client or server or both. Where the application is acting as a client, Top End is able to support transactions initiated from the mainframe. The remote server includes the ability to register transaction control APIs used by the remote application and database. This enables the remote application to control the local transaction by overriding the standard tx_begin (), tx_commit () and so on.
As the number of mainframe DBMSs that support XA are limited (Oracle does, but not CA-IDMS, IMS, or DB2), Top End client transactions based on Unix or Windows NT cannot generally include the mainframe server portion as part of the client transaction. The developer can build Unix or Windows NT clients that invoke servers accessing mainframe DBMSs but the server portion will not be under transaction control.
The remote server software component consists of two components - the remote server software itself (which resides on the remote server) and the remote server network agent.
The remote server software includes the RCS component (remote client services) to enable the server to act as client as well. The remote server software does not include a directory, but does include components to handle transactions, to support security and enable the CSP and shared or private memory queues to be maintained on the server.
The remote server network agent acts like a proxy on behalf of the server. It will channel any messages to the client using its own address mechanism and receive messages from the remote server (in this case acting as a client) and channel them on its behalf.
Top End supports IBM 3270 terminal access using existing terminal networks to Top End server applications. It also supports other devices, such as TTY type terminals and support for personal computers. A special component within the node manager, consisting of a library of terminal support routines and screen formats, is used to support communication between a character mode terminal and a Top End application. The Top End device support component handles screen management, translates screen formats and can support field level validation.
Top End enables a client to communicate with an IMS or CICS-based application, or a CICS or IMS client to communicate with a Top End server (in peer-to-peer communication) using the LU6.2 communication protocol. Security and transaction control is supported by Top End by a mapping to the protocol (transactional, synch, no-response, conversational and so on) and any user ID or password.
Remote clients can be Windows 3, Windows 95, Windows NT, MS-DOS, OS/2 and UnixWare. Remote client services are also available for all of the Unix platforms supported by the Base services component. Remote client services enable an application client using the Top End API (CSI) and TM API to run remotely on a networked workstation that does not have a node manager running on it.
Clients do not advertise their services with the directory services. Instead, the network agent acts like a proxy on their behalf - client messages are all funnelled through the agent, which then accesses the directory to determine the address of the service the client requires. All the client has to know is the address of the agent acting on its behalf.
Java remote client is aimed at organisations wanting to support commercial transactions on the Internet using the Web.
The Internet and its protocols are normally unable to recognise the state of a transaction, which makes the multi-step, often complex interactions that take place within a transaction largely impossible to support. Java remote client solves this problem by combining the distributed transaction processing capabilities of Top End with software that can sustain transaction interactions over the Web using browsers.
Although Web browsers and servers can be used directly with Top End, NCR believes that the combination of the Java remote client with the Web browser is a more robust and high performance solution for transaction processing and other applications.
Java remote client is written in the Java language and is supplied as a Java applet. The architecture for Java remote client is shown in Figure E4.4.
The Top End Java remote client applet contains a Top End Java remote client application and a Top End Java remote client class library. The Java remote client applications are written in Java by the application developers. The class library provides all the facilities normally supplied by a Top End remote client. The Java remote client packages requests and their associated parameters into a message buffer and delivers them over the network to a special network agent on the Top End server.
Users accessing the services of Java remote client, first access the Web using the browser. They then download a HTML page containing the tag pointing to the Top End remote client applet. Once the page is downloaded, the Web browser activates the applet, which then becomes an autonomous program running in the browser.
Java remote client replaces the need for HTTP and CGI programs, and also avoids the use of Web servers. Instead, it opens a direct connection to the Top End server, which runs on the company's host server. Communication between the client and server is via TCP/IP, but the protocol used is not HTTP.
The Top End network agent server component accepts requests from clients, unpacks the data in the message, performs any data format conversion and then maps those requests onto Top End service requests using the Top End API. The network agent then makes the appropriate service request to the node manager. Once this has happened, it is executed in exactly the same way as any request from a normal client.
Once the server has executed the request, the response is passed from the node manager to the network agent which packages the results and any error information into a message that is sent to the Java Remote Client. The client then maps the contents of the message into the client interface and passes this to the Web browser.
Security for users of the Internet (as opposed to intranet) has to be provided by corporate firewalls, primarily because Java cannot yet use the facilities available over the Internet such as SSL and signed applets (signed applets use public key encryption techniques to encrypt Java applets), but of course once the request enters the Top End environment it can be checked using Top End's own facilities.
The service interface repository (SIR) enables 'objects' to be registered. Objects in this context can be any service available over the network and may include applications or even database stored procedures. You select an object in SIR and Top End generates the code to connect to Java or ActiveX applications. This means that client applications built using ActiveX or Java can access remote applications as though they were local business 'objects'. Top End hides the interface to, and the location of, the server code.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Product background
Top End's roots go back over 25 years to the early development of TP monitors. When NCR first started to ship mainframe computers, its staff in countries such as the UK and Switzerland decided that the machines needed to be supported by a mainframe class TP monitor. The UK developed a TP Monitor called OLCP and staff in Switzerland developed one called Transact. NCR, however, decided that it needed one TP Monitor that worked across the range of machines and therefore developed a TP Monitor called TranPro, which it provided for all its proprietary systems.
TranPro evolved to become MultiTran, a TP Monitor that could support the NCR 9800 and clustered environments with added functionality such as failover.
When the NCR 3000 range was launched in 1990 and NCR decided to move away from its proprietary operating systems to Unix, it realised that MultiTran would need to be adapted and re-developed to support not only the Unix operating system, but large networks of distributed computers. MultiTran was consequently re-developed to support distributed environments and parallel architectures and the resulting product was renamed Top End. It was released in late 1991.
Although development of Top End continued steadily after its release, the team were undoubtedly affected by AT&T's ownership of the competing Tuxedo product. Rivalry was (and still is) fierce and the effect of having two competing DTPMs being sold by the same company, not only created customer confusion, but also affected morale.
Despite the rivalry between Tuxedo and Top End, development and enhancement of Top End continued. Once AT&T sold USL to Novell in 1993, however, a notable increase in the pace of Top End development took place. The version 2.02.02 update on Unix was available towards the end of 1995 with extended recovery and RFC enhancements to the Base services.
Top End on Windows NT (release 2.03.01) was available at the beginning of 1996. NCR invested significant effort in the development of the Windows NT version, which was developed from scratch to take advantage of Windows NT's facilities. Although the resulting Top End component has a different code base from the Unix versions, NCR stated that the result is a product that is tuned to the Windows NT environment.
The announcement of AT&T's notice to allow NCR to be an independent public company was matched by a flurry of announcements from the Top End team. In March 1996 it announced:
In June 1996, NCR announced the first of its Internet-based tools. These included the Java and CGI remote client capability and Top End on the Web. In August 1996, NCR announced its Java and ActiveX remote client component, as well as its service interface repository (SIR).
NCR also announced that over 80,000 copies of Top End were to be distributed on SCO's Toolware CD, released in November. This CD ships with every UnixWare system and contains a variety of tools for use on SCO platforms. DCE integration was also added, this time at the request of a specific customer - Wal-Mart.
Top End is NCR's only strategic middleware product. It offers a complementary product called LifeKeeper, but unlike the other hardware vendors, does not offer competing MOM, ORB or RPC middleware. The only other middleware offered by NCR is a port of DCE on NCR machines. As such, Top End is an extremely important product for NCR and one that it channels considerable resources in - people, time and money. Top End is intended to be a product that can span the multiple markets of MOM, ORB and DTPM.
It is also a product that NCR's overall hardware and company strategy depend on. NCR has (since the release of its Tower range of minicomputers) concentrated on the 'open' (Unix) distributed computing market. More recently it has also added support of Windows NT. Its range of computers go from Intel-based office machines to massively parallel processing (MPP) and symmetric multi-processing (SMP) machines. These machines are aimed at the customer needing high performance, scalable, distributed 'enterprise level' computing; Top End was specifically developed by NCR to support this type of architecture.
NCR's policy is to release all 'tier 1' platforms together. Other platforms may be released later. Tier 1 platforms are all NCR machines, IBM AIX, HP-UX, Solaris and Windows NT. The release of some platforms at later dates to the tier 1 platforms has resulted in different version numbers being used for different platforms, as the functionality on all platforms is not necessarily the same. Although tier 1 platforms were on release 2.02.02 (with Windows NT on 2.03.02) at the time of writing, other platforms were on earlier releases.
NCR's forthcoming version 2.04 is intended to bring every platform up to the same level of functionality and version number.
NCR is in partnership with EnterSoft and Bay Technologies to provide ports to third party hardware. EnterSoft, for example, worked with NCR to provide the SCO UnixWare port, and it builds the remote server component. Once EnterSoft and Bay Technologies have done the ports, NCR does the testing and integration work.
The most immediate plan for Top End involves the release of the remote node services version 2.04 on Unix and Windows NT towards the beginning of 1997. This release will harmonise the functionality on all these platforms. Additional enhancements include:
Version 2.05 of Top End on Windows NT and Unix platforms is due towards the middle of 1997. This release will include:
Version 3.01 on Windows NT and Unix will then be the next edition with a release date that has still to be announced. This release will include:
Other plans NCR has for Top End include the addition of more plug-and-play type tool support via ActiveX components. Tempo (NCR's tool support and service tool integration programme) is being expanded to provide more emphasis on this aspect.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Company background
NCR was founded in 1884 by John Patterson. As the National Cash Register Company, it was the first maker of mechanical cash registers and the first maker of cash registers powered by an electric motor. NCR specialised in cash register technology and related business equipment machines right up to 1952, when it acquired Computer Research Corporation and started its move towards computing technology. In 1953, NCR established its Electronic Division to pursue this diversification into 'electronic business machines'. In 1982, NCR launched its first NCR Tower supermicrocomputer system.
NCR was acquired by AT&T in 1991. At the time AT&T bought NCR, there was a strong feeling within telecommunications companies that telecommunications technology and computing technology would eventually merge. AT&T had already lost over $3 billion trying to establish its own computer division, and had given up trying to establish its own computer manufacturing division. However, it had considerable success with computer software such as Unix, which, of course, originated at AT&T.
AT&T was in 1991 largely a US-based company. In the early 1990s, there was a strong push by most telecommunications companies to free themselves from their 'national' image and start to push into international markets. NCR had offices, as well as support, sales, marketing and consultancy staff in numerous non-US countries worldwide.
On its acquisition, AT&T renamed NCR 'AT&T GIS' (global information systems). This turned out to be something of a portent for the future, as AT&T seemed unaware that the new name caused considerable confusion in the market - many customers wondered what NCR had to do with geographic information systems!
AT&T's ownership of NCR was not a success. Resources were wasted as they were diverted to pointless projects such as the combined telephone and computer (which never reached product status), and NCR's traditional customers started to back away, disaffected by NCR's new parent.
NCR had always specialised in the finance, telecommunications and retail sectors. Telecommunications companies were the first to abandon NCR, stating that they did not wish to do business with a division of their main competitor. AT&T also succeeded in alienating other large companies. For example, its adverts implying that it was better to use the telephone for business than fly, succeeded in annoying some of NCR's airline company customers.
Perhaps most harmful to NCR's Top End product was AT&T's acquisition of USL and the Tuxedo product. AT&T then owned two distributed transaction processing monitors, which had previously been (and remained) in fierce competition. In the time that AT&T owned NCR, they succeeded in turning an $8 billion company making a profit, into an $8 billion division making a loss.
AT&T eventually realised that there was no synergy between the companies. In September 1995, it announced that it would spin off three of its divisions - AT&T Communications, AT&T Network Systems and AT&T GIS. This was planned to be complete by the end of 1996.
AT&T GIS was renamed in January 1996, back to NCR. AT&T Network Systems was renamed to Lucent Technologies in February 1996. Lucent was floated in September 1996 and is now a separate company. Lucent was in the black at the time and the conditions for flotation were met on the date planned.
In order to float NCR, the division had to have made a profit for two consecutive quarters. NCR had a poor financial year in 1995 with a loss of over $850 million. This loss was partly caused by a massive re-structuring exercise completed in 1994/5, but was also a result of the disarray caused by AT&T's decision to spin off the companies. This meant that assets were split between them, with all the disruption this inevitably causes. By the second quarter of 1996, NCR had managed to pull its finances round and it made a profit of $11 million. At the time of writing, NCR's published results for the third quarter were better than the previous quarter (with a profit of $29 million), thus satisfying the pre-conditions for flotation.
NCR supplies a range of hardware, business equipment, software and services, and specialises in providing products to the retail, communications and financial markets. Its products and services include the following:
NCR also provides solutions that cover subsets of the products above. Its scalable data warehouse programme, for example, includes its MPP and SMP range of computers; the Teradata database, Top End, partner relationships to deliver specialised services with the likes of Informix, Oracle, Information Builders and Microsoft; and its own consultancy offerings.
NCR employs about 37,900 people worldwide with approximately 19,000 based in the US. It has five main divisions:
Revenue in the 1995 calendar year was $8.16 billion, with a loss of over $37 million. By the second quarter of 1996, NCR had made a profit of $11 million. At the time of writing, NCR fully expected the published results for the third quarter to be better than the previous quarter. NCR's top five international countries in terms of revenue are Japan, Germany, Switzerland, UK and France.
Since NCR was being divested from AT&T at the end of 1996, there were SEC restrictions on any financial information put into the public domain. Ovum estimates that Top End revenues were around $20 million in 1996. This figure only represents software sales and does not include associated service revenue.
NCR stated that, in general, its only competitor has been Tuxedo. It rarely comes into competition with the other DTPM vendors and does not usually compete with the DBMS vendors. This last point is an interesting one as it reflects the different profile Top End customers have, as opposed to some of the customers of other DTPM products. Database vendors are often capable of providing reasonable solutions in the departmental segment of the market, but once the applications reach a certain size in terms of the volume of data and number of client and server nodes, the vendors tend to seek out the services of a DTPM product. NCR thus tend to team with DBMS vendors rather than compete with them.
NCR tends to concentrate on its existing or potential hardware customer base and its key vertical market customers - finance, telecommunications and retail, with secondary vertical target markets being the transport and manufacturing sectors.
Its direct sales force tends to target the larger customer, whereas its distributors and partners may target the smaller one.
Top End's traditional end-user customer has been the larger company needing an 'enterprise level' (as opposed to departmental level) solution for its infrastructure. NCR defines 'enterprise' as any company having at least 10 server machines and at least 100 clients, although in practice, NCR's clients have far larger configurations than this.
NCR has also recently targeted companies needing the combined solution of a middleware infrastructure with Internet access, and has also concentrated on companies wanting to off-load processing from the mainframe onto Unix or Windows NT.
Key customers include AT&T, Belgacom, BellSouth, Delta Air Lines, EDS, IRS, Italian PTT, Mastercard, NTT, Reuters, Sears, Sony, Swissair, Telstra and Wal-Mart.
The server product is grouped (for pricing purposes) into a base set of services (Top End Base Server software) and a series of value added services (Top End Server Options and Administrative Tools). The value added services include RTQ, global administration, LU6.2 connectivity, security services, interactive system definition software and the Oracle Tools interface. These services are split from the Base Server software mainly so that the customer can choose which additional modules they require.
NCR has a tiered pricing model. Machines are split into groups based on their theoretical relative processing power. All machines within a category (whatever make and operating system) are then priced the same. Altogether there are seven categories of machine. Typical prices for the services are as follows:
Prices for the other value added services:
The client product is not split into modules, but comes as one packaged item of software - the Top End Client software. Non-Unix pricing is based on the number of copies sold (1, 32, 64, 128 and over). Typical prices for OS/2 or Windows NT clients vary from $200 for a single copy to $120,000 for 1,024 users. MS-DOS and Windows 3 clients are cheaper.
Pricing for Unix clients is based on the same server size classification used for servers. Typical prices for a HP-UX client range from $375 for class 1 to $10,700 for a class 7 server.
Site licences can be obtained and discounts are offered.
Software houses and application package vendors can purchase Top End under a separate licensing arrangement equivalent to a one off site licence, so that they do not have to keep track of every copy of Top End they deliver with their software.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Customer support
NCR provides a comprehensive set of user guides and reference manuals aimed at each job, for example, there are sets for designers and for administrators. Guides are also provided for each product/component.
Courses can be provided on site or in NCR's own training facilities. NCR has its own global training package with the following standard courses:
Training can also be provided by Top End distributors and OEMs, such as EnterSoft.
NCR provides a Top End 'home page', which provides general product information, press releases, analysts comments and customer support telephone numbers. The Web address is http://www.ncr.com/product/topend. There is also a Top End special interest group and a NCR newsletter.
Support is provided by NCR or its distributors. Both provide 'how to' support, product assistance and problem diagnosis services via a telephone hotline, e-mail and an Internet help desk. The customer has the choice of two support programmes with different charging structures - five days a week with eight hours of support each day and seven days a week with 24-hour support.
First line support is provided locally in each country, with second line support being provided by the Top End development team based in San Diego. San Diego's global support centre operates 24 hours a day, seven days a week.
Consultancy services can be obtained from NCR, its distributors and several partner companies. A large range of consultancy services can be provided, including architectural design, rapid prototyping, application development monitoring, implementation and deployment, tuning and on-the-job training.
NCR offers a product and service programme called Tempo. NCR staff go to the customer site and implement the application development tools the customer will be using, as well as Top End itself. The staff then build a distributed computer prototype. If the customer decides to deploy the solution across the company, the NCR staff will provide assistance to fully develop the system.
Demonstration copies can be obtained, but NCR prefers to get involved in any evaluation.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Distribution
NCR distributes Top End from its offices worldwide and via several selected distributors. NCR has over 1,100 offices in more than 130 countries. The distributors provide the same level of support and consultancy services as NCR.
Headquarters
NCR Corporation
1700 South Patterson Boulevard
Dayton
OH 45479
USA
Tel: +1 513 445 5000
Europe
NCR Ltd
206, Marylebone Road
London
NW1 6LY
UK
Tel: +44 (0) 171 723 7070
Fax: +44 (0) 171 725 8224
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: The communication layer
The communication layer uses the network software to transport instructions and data across the network. It is responsible for ensuring that data and instructions are passed from a sending process to one or more receiving processes, and that the information sent is correctly handled at the receiving end.
Top End scores 54 out of 70 for the communication layer, which we normalise to a score of 8 out of 10.
Messages are limited to 30K in length, although there is no limit on the content. There are plans for large message support. A message consists of a header, together with a data part provided by the program.
The header is set up automatically by Top End from internal data and the information provided by the program. The header contains information on which client sent the message, when it was sent, the user ID and password, whether the type of communication is conversational or not and whether the message is subject to transaction control.
The data portion of the message is also set up by Top End, although the programmer must clearly create the message contents first. The program provides a pointer to the buffer that contains the message to be sent, and Top End transfers this data into the buffer.
Top End does not support data format translation, however, it is planned for release 2.04. NCR was unable to provide precise details of how this would work, but indicated that a message format database will probably be used where the developer sets up a description of the message, which is then used for conversion.
Top End handles all the buffering for the programmer, the program does not have to do anything.
Top End does not support the broadcast or multi-cast of messages, although administration messages (for example, connect to the DBMS, disconnect from the DBMS, allocate more shared memory) can be broadcast to the queues of servers via the administration tools.
All communication is essentially one-to-one, but variations upon this basic method are supported. One or more clients or copies of clients can communicate with a server instance, and a client instance can also communicate with one or more server instances.
A client can issue a send with no reply expected, as a request/reply pair; or in the conversational style of processing, the client can issue multiple requests with perhaps only one reply. Where a conversational style of communication is being used, the messages need to be placed on the server's private queue in order to maintain context.
The programmer uses the logical name of the server, the service required and optionally the target (for more about targets see The management layer: load balancing). The directory services look-up the name in the memory-based directory and get the physical address of that instance of the server. In reality, the directory services are getting the name of the shared memory or private queue of the server.
If, after directing a message to a server, the server or its node are found to be out of action, the node manager will re-direct the message to the next most suitable server until one that is up and running is found.
The only time a message is routed to a specific server instance is when a dialogue has been established between two processes - the client instance and server instance - and the context of the conversation must be maintained. This may occur, for example, when database information is to be returned to the client in chunks of data, and the database cursor must be maintained by the server as well as the context of the query.
Top End supports TCP/IP and OSI TLI on Unix and TCP/IP only under Windows NT. It also supports gateways to non-Top End systems using IBM's LU6.2. Remote clients can be connected to the Top End servers using any WinSock compliant network protocol.
Top End uses the concept of a dialogue to identify the send and reply conversation that takes place between two process instances. Each dialogue is uniquely identified, so sends and replies can be matched. A dialogue is equivalent to a session.
The developer can use a sequential model of processing where each send and reply pair have to be completed before the next send and reply are issued, whether this is a send and reply to a different process or the same one. Parallel dialogues can be supported as long as the developer establishes each one using the commands 'tp-client-signon' (which establishes the dialogue) and 'tp-terminate' (which ends the dialogue).
Top End establishes the connection, co-ordinates the transmission and synchronises with the network, the programmer does not have to program this.
Transmission of both client and server can be synchronous or asynchronous.
When the client 'sends' a message, it is blocked until the send has been completed. An acknowledgement is issued to the client that the message was successfully received by the node manager. The client can then continue with other tasks.
When the client receives the reply using the separate 'receive' command, it can issue a time-out value.
If the time-out value is null, the receive command is treated as a 'polling' action. If no reply message is present on the CSP queue, processing can continue. If a time-out value has been set, the client will block for that time, after which it will time-out.
In order to synchronise the send of a message and its reply, the send and receive commands must therefore be placed together in the logic, so that a send is issued immediately followed by a blocking receive command.
An almost identical set of actions occurs with the server. When the server issues a 'receive' to get the client's message and the time-out value is a null value, the receive command is treated as a 'polling' action and if no reply message is present on the server's shared or private queue, processing can continue. If a time-out value has been set, the server will block for that time, after which it will time-out.
When the server 'sends' a reply message, the send is completed asynchronously and the server can continue with other tasks. The server will wait and block until a confirmation is received from Top End.
In version 2.04, NCR has added the ability to use call-backs from either clients or servers. The client or server can provide a pointer to a call-back function and when a send or receive is complete, the function is triggered which then interrupts processing.
As described in Product overview, there are essentially two types of queue provided by Top End, the memory-based queues (private, shared and CSP) and the disk-based recoverable transaction queue (RTQ).
Only when the program is using the RTQ is the queue specifically named. The developer writes enqueue and dequeue commands to put messages on this queue and take them off. The other queues are invisible to the programmer, who writes a program to access services and servers, not queues.
The CSP/node manager queue is used to store messages that are to be sent prior to transmission, and messages which have been received prior to them being handed to the client (in the case of a reply) or prior to them being placed on the shared or private queue of the server. Node manager queues are located on every node where the Base Server services are provided. Remote clients use the node manager queue to store messages prior to transmission and replies. Remote server components also use a node manager queue.
The servers' shared and private queues are located on server nodes. A shared queue can be accessed by one or more server instances, but not by more than one server. The private queue is accessed by one and only one server instance. Shared and private queues are used by the Top End Base Server services and remote servers.
The recoverable transaction queue (RTQ) can be located anywhere on the network and is shared by one or more clients and/or client instances and one or more servers and/or server instances. RTQs work in conjunction with the CSP/node manager queue and the servers' private and shared queues. The RTQ is not a replacement for these queues, but an addition used to provide safe store and forward capability for messages.
Messages cannot be prioritised. All messages are placed on the queues in a first-in-first-out order to preserve the transaction sequence. It is worth pointing out that message prioritisation and transaction processing, when combined, have the potential to destroy the integrity of the database because the order of transactions is usually important. NCR has adopted the 'safe' approach by assuming all messages may be under transaction control. However, if Top End is used simply to pass messages that are not under transaction control, the order is then less crucial and the ability to support message prioritisation becomes more relevant.
The only time some form of prioritisation is possible is when RTQs are used: messages can then be time-stamped for sending at a specific time (see The management layer: Advanced queuing services).
Messages cannot be selected from private or shared queues by the server, the queues are read by the server and messages are passed to it in the order they were stored in.
Memory-based queues have no special protection from Top End. If a node goes down, or the node manager fails, all the messages in the node manager queue and the server's private and shared queues are lost.
NCR pointed out that as the administrator can set the parameters for queue length (CSP, shared and private queues) such that a minimal number of messages are stored on the queue at any one time (if more messages than the maximum allowed start to build up, Top End will immediately start new server instances), any loss of messages is likely to be small. It also pointed out that as long as the client was using the request/reply mode of conversation, it would know the message had been lost as no reply would have been received.
If a server instance fails when it is using a private queue, that private queue is also lost. If all the server instances of a type on one node fail, the shared queue is lost (the failure of the last remaining server instance on that node causes the loss of the shared queue). Again, as long as the client was using the request/reply mode of conversation, the client would know the message had been lost as no reply would have been received.
If any queue becomes corrupted, Top End is able to detect this immediately. The queue corruption cannot be corrected, but a message is sent to the administrator's console and the node manager itself is shut down until the fault is repaired. Again, this will result in a loss of messages, detectable by the client as long as it used the request/reply mode of processing.
RTQ, on the other hand, is protected against failure of the node or node manager by being on disk. It is not protected against loss of the disk or corruption of the disk, for example, by the use of logs.
No limits are placed on the memory-based queues. The administrator can control the size of queues using the load balancing tools, where the maximum number of messages in a queue can be specified. When the maximum is reached, new server instances or CSP instances are created.
The administrator can also set threshold limits on the memory-based queue sizes (again in terms of the number of messages), so that alerts are posted to the administrator's console when these thresholds are reached. Once alerted, the administrator has the option of changing the load balancing parameters dynamically - queue sizes and the positioning of servers.
The administrator cannot currently specify a maximum size for the queue in absolute terms ( that is, in bytes). As the message length is restricted to 30K, this does not generally create a problem. However, NCR is planning to allow messages of unlimited length, meaning that several very large messages could (in theory) exceed the memory limitations of a small machine, even if they were still within the maximum message number. Currently, the user would get the standard Unix message if they ran out of memory.
Similarly, if the number of server instances created exceeded the size of the machine to support them, the user would get the standard Unix message.
Where RTQ is being used, the administrator defines the size of the file to hold the messages. The administrator can define a threshold point - say 80% full - that when reached causes an alert to the console. The administrator can dynamically expand the size of the RTQ without needing to power down Top End or the RTQ itself.
Messages stay on all the queues until they are removed by server instances.
In general, memory-based queues store the message for as long as is needed, but special circumstances may arise when using the RTQ. Messages in the RTQ remain there until the server node and its node manager are up and running, and the message is transferred. The RTQ may thus be storing messages that are 'stale', because the server and its node remain inactive. The administrator can use administration tools to search and select from the RTQ and remove messages. This has to be done extremely carefully. It is more likely that the administrator will use the search to determine whether nodes or servers need to be re-activated to process these old messages, rather than using it to find messages that need to be deleted. There are no automatic alerts to highlight stale messages.
When the client sends a message to the node manager, the node manager puts the message on its local CSP queue. The network interface will then attempt to send the message to the network interface of the server node.
If no alternative nodes have been set up, the client is sent a return code indicating that the requested service is not there and the message itself is returned to the client. Top End does not store the message in the CSP and attempt retries automatically.
Communication failures are reported to the node manager by the network interface.
If a node fails and alternatives exist, however, Top End will automatically re-route the user request to another copy of the application on another node. Re-routing is an option the developer can use for servers that are not database dependent and as such can be placed on any node in the network. It is also an option that can be used with servers that are dependent on a database, as long as they have been deployed on a 'paired server' basis. A paired server configuration uses two machines that share a common I/O device. If one machine fails, a fail-over agent detects it, switches all of the shared data to the alternative machine, then re-starts the applications on it.
NCR provides a complementary product to Top End called LifeKeeper. One of LifeKeeper's functions is its ability to 'failover' applications from a failed server to one or more surviving systems. LifeKeeper uses 'heartbeat' monitoring as its primary method of failure detection. If a node fails, LifeKeeper will detect the failure and:
LifeKeeper is also able to recover RTQ queues.
When a node fails, the software that could have failed on the node may include system software, DBMS software or packaged software - not just the Top End application components. NCR provides pre-packaged 'recovery kits' for Informix, Oracle, Sybase, MS SQL Server, MS LAN Manager, MS Exchange, Lotus Notes, PeopleSoft and SAP.
By combining LifeKeeper's application failover capability with Top End's ability to replicate services and dynamically respond to their changing location, a company has a set of products that automatically respond to server node failures with no operator intervention needed (other than that needed to start the alternative node).
LifeKeeper includes a standard interface for error reporting, a recovery system and interfaces for recovery control. LifeKeeper is only available on NCR's Unix SvR4 and Windows NT.
If the server has failed and no alternative servers on other nodes (or the same node) have been set up, the client is sent a return code indicating that the requested service is not there. The message itself is returned to the client. Again, Top End does not store the message and attempt retries automatically.
If an application server fails and alternatives have been set up, however, Top End will automatically re-route the user request to another copy of the application on the same or another node.
Top End can also re-start failed applications, but it can only re-start server instances when the node itself has not failed. If the node has failed, a product such as LifeKeeper must be used. Top End is able to re-start failed servers and clients. Where any sort of failure in communication has occurred, however, the dialogue between client and server is dissolved and a new one must be established to re-send the message.
If the client issues a receive of the reply, it will be informed that the line or node is down. Again, Top End does not store the message and attempt retries automatically. The server will also be informed that the line is down when it tries to issue a send back to the client.
If the client attempts to send a message to a RTQ service component and either the node or the component itself are down, Top End will inform the client that the node cannot be reached.
Once the message has been safe stored on the RTQ, the 'worker' will then attempt to pass the message on to the server's node. If the node is unavailable or the line is down, the worker will not abandon the attempt but will continue to try to send the message. Node failure or line failure is not the subject of any 'retry limits', the worker simply continues to try and send the message until it eventually reaches the server's node. If the node is available and the message cannot be delivered, however, the worker will continue to retry, but this time under the control of retry limits. The administrator can set both a time interval for the retries and the number of retries that should be attempted.
Once the retry limit has been exceeded, the message is marked as 'held' and an alert is sent to the administrator. Once the problem has been fixed, the message can be moved back into the active state through the administrative tools.
One of the tools provided within the Administrator's toolset enables recoverable transaction queues to be relocated on other nodes, so that if a line failure does occur between a client, for example, and a RTQ server, the administrator can transfer the queue to another node at run-time and re-connect the client to the new node. Similarly, if the line between the RTQ and the server is down, the RTQ can be relocated dynamically to a node with a line between the server and RTQ node. As Top End itself handles all the addressing and directory updates needed, the administrator only has to move the file.
API commands are statically bound into the program and one executable is produced. However, support is provided for DLLs and shared libraries. Directory look up is dynamic at run-time.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: The translation layer
Top End does not support translation of APIs such as ODBC.
As with the other DTPM evaluations, we have not scored Top End for translation layer support.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: The management layer
As messages and procedure calls fly between one piece of software and another, there is plenty of potential for message corruption, loss and error. Additional controls are needed to check, for example, that messages are not lost, that errors are trapped and handled correctly, and that security is maintained.
The management layer of middleware takes over the responsibility of providing these services from the developer, so that they do not need to be duplicated in each program. Support for the management layer is one of the key differentiators between middleware products.
Top End scores 64 out of 90 for the management layer, which we normalise to a score of 7 out of 10.
The emphasis in Top End is for support of multi-tasking rather than multi-threading (which NCR considers easier to program). The APIs and components in Top End are thread-safe in the Windows NT platform. Thread-safe components are being developed for the Unix platform, so programmers can use threads in an application but must create them themselves.
Top End's considerable support for multi-tasking is described under Load balancing.
Top End supports multi-tasking of the application (server) and multi-tasking of its own components - RTQ services, CSP services and so on. All multi-tasking and creation of new tasks, as well as removal of unwanted tasks, is automatic and dynamically controlled by Top End, based on loading information monitored by Top End itself at run-time. Creation and removal of tasks is controlled via administrator-defined parameters.
Top End has a component called the automatic throughput optimisation (ATO) module which is part of the node manager. This module monitors the requests for service - whether they are application services or internal Top End services. If it sees that a bottleneck has started to form, it will automatically spawn additional copies of the service. ATO spawns copies of the application servers on the same node. It can also spawn copies of components such as the CSP and recoverable transaction queuing service when their loads become too high.
Once the traffic had died down, the ATO will then automatically shut dormant copies down.
ATO monitors the ratio of messages on the shared queue (CSP queue, RTQ etc) to the number of server (or other process) instances. If this ratio exceeds a given maximum value then ATO starts up another instance. When this ratio falls below a given minimum value then ATO shuts down a server instance. The administrator sets these parameters of maximum and minimum values. Other parameter values that the administrator can set, to control the way the load is balanced using ATO are:
All these parameters can be dynamically changed at run-time by the administrator using the GUI-based administrator's tools.
Before a message is placed on a queue, the directory service also has to decide which of several server instances on different nodes are to receive the message. The directory uses one of three methods (chosen by the administrator) to decide.
The Message sensitive routing component analyses the contents of a message to decide which server is to receive the message. Although the module can be used for other purposes - for example, where the data has been geographically distributed by content and access needs to be provided across the spread - it is primarily used within Top End to help balance the load on large databases.
The administrator sets up a table that describes the server, the service and the routing method to be used. The method can be quite complex and is defined using a 4GL-type language. Using the 4GL, the developer can specify a route for a message based on any of the contents of the message in combination, using ranges, absolute values and so on. For example, a very large database may be partitioned to ease performance problems so that the data for the London location is in one file on one node, the data for Winchester, Brighton and Luton are in another database. The values London, Brighton, Luton and so on would each be associated with a node address so that any messages were routed to the correct one.
If necessary, the administrator can change the routing algorithm on-line at run-time, though clearly, this has to be done with care.
The server instances supporting these partitions of the database are known as 'targets'. When the client sends a message, the parameters of the call can optionally specify the target as well as the server and function. This means that not only can Top End automatically route a message based on the content, it is also possible for the client program to route a message under its control to a server target. It is worth noting that targets can be the subject of ATO optimisation as well as the normal server instances.
The administrator can designate one shared queue to be the default destination for requests with values that are out of range. They can also specify a default to be used in case the shared server queue, which should have been targeted, has failed.
Message sensitive routing can be used within a node and across nodes. Message sensitive routing is not yet supported on the Windows NT platform.
Top End uses Kerberos to provide authentication and validate the identity of users. Kerberos authentication information can be set up with the same TP Secure tool used to set up Top End's own authorisation information. This tool has a graphical and a command line interface and can be accessed via a central console by the administrator.
Authorisation to use the services once the user has been authenticated is provided by Top End's own internal security services. A table is set up using Top End's TP Secure tool which describes the server, the functions and the permissions granted to access each server and service. Permissions can be allocated to groups of users and each user has a profile that describes the groups he or she is in and the permissions that have been allocated.
The table is propagated across the network in the same way the directory is, so that the permissions are available on all server nodes. Access to this table is protected.
Authorisation and authentication checking is provided by the network agent where remote clients and remote servers (acting as clients) are used.
The administrator (or developer) can also choose to have network traffic encrypted. If the developer chooses the 'clear' option, encryption is not performed, but if they choose the 'safe' option, all traffic across the network is encrypted (including internal Top End communication). A further option, 'private', provides encrypted check-sums so that any alterations to the message can be detected. Encryption is handled by the network interface components.
Top End uses the DES standard - currently the subject of US government restrictions (as is all encryption software). Two versions of DES are supplied to conform to the regulations - a US domestic version and an international one. As many non-US banks, financial institutions and airlines are either exempt from the restrictions or have applied for a licence, NCR has not found the restriction to be a particular problem.
There is no encryption of traffic from a remote client, or remote server acting as a client, to a Top End node, but support for this is planned. It should also be noted that security services are not provided on Windows NT.
Messages are not grouped, but where large messages are being processed (this will particularly be the case when the restrictions on message length are removed in the next release), the buffer is split and reconstituted automatically by Top End.
Optional message compression is due to be provided in version 2.04 along with the introduction of unrestricted message lengths.
No particular additional features are provided to manage memory other than those already described under load balancing, where server instances can be created and removed as needed, which helps to conserve memory.
As was explained in the introduction, application services/functions can be grouped into one server 'program' with a single Top End core, to save memory.
When dialogues are terminated, Top End frees all the resources and manages the memory for the programmer. The 'terminal handling module' employs caching to save frequently requested formats. The maximum cache size is tuneable.
Delivery cannot be guaranteed in the sense that a message is guaranteed to reach its destination, regardless of failures and without needing any client or server intervention to re-send or re-read messages. With Top End it is possible that failures can result in a message not reaching its destination. However, although delivery is not assured, the client or server is always made aware if a message is not delivered.
If the application just uses memory-based queues, a message may not reach its destination because:
Even using RTQ, there are still circumstances where the message may not reach its destination:
Support for deferred delivery in a transaction processing product may seem incompatible, but as Top End is a messaging product at heart, it can be used simply to pass messages without the need for transaction control.
Support for deferred delivery in this case is provided using RTQ. A client can send a message to a server even if that server and its node is not up and running, providing that the RTQ node is up and running. Similarly, the server can receive the message from RTQ even if the client and its node are not up and running.
Both client and server can pull and poll on a receive command. This was described in the section on synchronous and asynchronous processing. The action taken depends on the setting of the time-out signal, which if set to null causes the process to poll for the message, or, if set to a value causes the process to request and then block processing.
Server processes generally have to be active and up and running in order to receive a message, they cannot be loaded when a message is received for them.
The recoverable transaction queuing service offers a special form of triggering service based on a time-stamp, which can be relative (that is, after a set period has elapsed) or absolute (at a specific date and time). Messages sent to recoverable queues are stored there and then dequeued by the worker component of this service. If no time-stamp has been placed on the message, the worker will try to send the message the moment it has been enqueued. If the message has been time-stamped, it will be held, either until the absolute time has been reached, or until the relative time period has elapsed. The worker will then send the message. Thus although the server process is not triggered as such (that is, it is not loaded into memory by the arrival of the message), it is triggered in the sense that there is some control over when the server is activated.
The next version of Top End is due to support call-backs, which can be used to trigger specific functions within a server process. It will be possible to set up specific 'listening' servers that only activate a service when a message arrives for it.
Top End enables messages to be bridged between TCP/IP and LU6.2 using the third-party product, Open Gateway. Top End can also bridge between TCP/IP and OSI TLI.
Top End provides message sensitive routing but it does not provide routing across a network of nodes - this feature is planned. Currently, if a message is to travel from, say, node 1 to node 4 via nodes 2 and 3, either a hardware-based solution (for example, routers) must be used or the programmer must develop intermediate programs that pass the message on.
Top End has no timing services.
No specific support.
No specific support.
Top End uses the XA model of distributed transaction processing support. The application communicates with the DBMS using normal DML/SQL commands and uses the TX interface to communicate with the transaction manager. The DBMS and transaction manager inter-communicate using the XA interface, hence the application does not need to know the interface exists. All synchronisation, commit and rollback actions are co-ordinated by the transaction manager via the XA interface.
NCR also provides 'veneers' to DBMSs that do not have an XA capability of their own. It currently uses XA veneers to access Sybase, Teradata and Ingres.
An XA veneer is really a translation tool that takes the XA commands issued by the transaction manager and translates them into the commands expected by the DBMS. By using this approach, NCR has been able to support DBMSs whose vendors do not support the XA standard.
Due to the modular nature of Top End, NCR has been able to separate the specific TM components that deal with the XA interface and drive the process via a translation table where the XA veneer is being used.
The DBMSs that Top End currently supports are Oracle, Informix, Sybase, Teradata, CA-Ingres, Gresham's ISAM-XA, Microsoft SQL Server and DB2/6000. MQSeries also supports the XA standard as it can act as a resource manager, so can be used with Top End to handle cross-product transactions.
The non-XA compliant database veneer facility is not available on Windows NT.
It is important to note that there are no conformance tests for the X/Open specification at the moment and as such it is not possible for any vendor to state that it is XA 'compliant'. All it can claim is that it has used the specification and produced an XA implementation. NCR has worked with the DBMS and other suppliers of XA implementations to ensure that its products worked with Top End and provide the correct interpretation of the specification.
Transactions are handled using two-phase commit. A set of TMS (transaction manager servers) modules within Top End use the XA interface to co-ordinate the DBMS/server interaction. One TMS exists for each resource manager, but copies of the same type of TMS can be created to improve throughput by the ATO service. Each TMS component is linked to the database library. The XA library is then provided either by the DBMS vendor, or the veneer to the DBMS is provided by NCR.
Where communication is via the mainframe using the LU6.2 gateway, co-ordination of the two-phase commit process is supported, but is not completed using the XA model. If one of the processes in the transaction fails and the actions on the mainframe must be rolled back, Top End co-operates with CICS to initiate a rollback of the actions on the mainframe. Confirmation for each service can be set at the message, sync level or connection level. (It is the mapping of transaction control onto sync points that provides data integrity.)
Transactions are identified by a unique number. The client issues the command tx_begin to denote that the next sequence of commands marks a transaction and Top End allocates the global transaction identifier to that piece of work. Every item of work that comes within that transaction is tagged with the identifier from then on.
Top End can distinguish between each client copy and server copy (instance), hence a set of sends and replies within a transaction will be to the specific server and client instances required.
A transaction can contain a series of sends and replies to server services and a set of enqueues or dequeues to the RTQ. Top End does not support nested transactions.
Unlike a DTPM based on remote procedure calls, the client does not make a series of 'calls' to the server. Instead, the client sends a set of messages that it expects a reply to. This reply is then used to determine whether the service was successful or not.
In a three-tier architecture, for example, clients access a business logic server that contains the transaction logic together with the limit commands of tx-begin and tx-commit. The database servers contain SQL statements. It is possible (using Top End for one database server) to contain SQL instructions that access more than one DBMS. Top End co-ordinates all the DBMSs.
It is also possible for the business logic server to send SQL calls directly to the database by embedding them within the transaction.
The first stage of the process is the initial issue of the send and reply commands bound by the tx-begin command. When the transaction statement tx-begin is issued, an asynchronous call is made to the server node manager warning it that a transaction is about to begin.
The business logic server issues a series of sends and replies to the database server. Each send/reply pair will result in the server performing some actions on the database, which are co-ordinated through the XA interface. If the work is successful, the server will reply with a success message. If the server action is unsuccessful, it will reply with an error message denoting what went wrong. If all the replies are a success, the client will issue a tx_commit command.
If any of the replies are unsuccessful, the client does not issue a commit. Instead, it will issue a roll-back or abort command and all those servers that were visited by the transaction are commanded to roll-back. Top End keeps details of the commands and actions (including any requested database updates to be made) in a protected log file, which it maintains. If the client commits, the log file is used to update the databases, if the client aborts, the log file is rolled back.
If all the replies denote a successful outcome to the process, the client will then issue the command to commit.
Where a commit has been issued, Top End takes over and co-ordinates the subsequent actions with the Top End components on the servers.
Vital to the success of this stage is the transaction log - a disk-based log of all the transactions that have to be completed at the node, in the order they must be completed in. In effect, the transaction manager keeps its own log of database updates, which it uses to co-ordinate with the DBMS.
The transaction manager first informs all resource managers that they are no longer working on the transaction. Top End then calls through the XA interface to ensure all DBMSs are ready to commit.
The transaction manager collects the responses from all the resource managers to the 'prepare' request.
If the resource manager can commit, it will reply to the transaction manager that it is ready and will then lock the data to be updated, ready for the second phase of the two-phase commit process.
If all the DBMSs are ready to commit, Top End issues the command to commit to each DBMS through the XA interface and the databases will be updated. Once updated, the locks will, of course, be released.
If the DBMS cannot commit because, for example, the disk that the data is held on has crashed or gone off line, or the file has run out of space, it will report that it cannot commit.
If any of the replies the DBMS gave denote that they cannot commit, Top End will initiate a rollback of those servers that have been 'visited' via the XA interface. The work to be completed is held in readiness in the 'commit' areas of each DBMS and not run against the actual database until the final command to commit is received from Top End. The command to rollback simply removes the 'updates in waiting' from the buffer area.
A transaction may fail for other reasons - such as the line going down before it was fully completed, the DBMS may fail, or the node fails. In this instance, the DBMS has not replied with the message that it could not commit; no message is sent back to Top End at all.
In this case, Top End uses the 'presumed rollback' model of the X/Open model. If, after a failure, there is no record indicating that a transaction completed the 'prepare' phase, it is assumed that it can be rolled back.
On the second phase of the commit, if any node, DBMS or similar fails, Top End keeps a list of pending transactions in its log. Since the log is protected, when the node is restored the contents of the log will still be available for processing.
Top End will wait until the node has been restored and the DBMS is up and running once more and will try again. The DBMS and transaction manager keep a list of pending transaction IDs, which they compare to ensure that all the transactions that should be committed have been committed. If the DBMS has already committed, it will know and tell the transaction manager. If the DBMS has not yet committed, then it will do at that point. Situations can arise, however, where the DBMS simply does not know. This can cause a problem - one common to all products that support XA - and the administrator has to sort these problems out manually. DBMSs such as Oracle, for example, may not be able to tell (after a restore) that transactions are pending, as some of its optimisation procedures overwrite the data.
Top End also has a facility called extended recovery, which provides the same level of transaction recovery services when a server and its DBMS are switched over from one node to another. This service is provided on all Unix-based machines; support is planned for Windows NT.
If the RTQ service is used, this transaction commit process is applied in exactly the same way (RTQ is XA compliant.). As explained in Product overview in the introduction, messages to the recoverable queue are only tracked to the point where they are stored on the queue. Thus a call may be made within a transaction to store the message on the file; if the store action fails the transaction is rolled back. If the store action succeeds, the transaction is deemed to be successful.
When the client needs to access the queue to dequeue the message, the dequeue action can itself be made part of a transaction. If the dequeue action fails, then the transaction is deemed to have failed; likewise, if the dequeue succeeds, the transaction is said to have succeeded. If a transaction involving a dequeue fails for some other reason, the message is put back on the queue.
None.
None.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Directory services
See Product Overview.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Development services
Both clients and servers can be developed using this model and the application has full control over all events. The programmer locates control in the application, not in Top End, from the moment the process starts until it fails or completes. Usually this type of application is more complex as the programmer has to build in code to handle timers, timing, signals and events. The programmer can use all the TX and CSI commands, but processes have to be written in C. In this case, Top End acts like a service library. This development environment tends to be used by tool vendors or developers needing tight control over resources (and C gurus!).
This provides a simpler interface and application code can be written in C, Cobol or C++. The developer is provided with a series of wrappers for their code, which provide ease-of-use features and 'short-cuts', such as easier initialisation. It operates primarily on the server side. In this model, Top End owns the 'main' function, and is in control of all the events. In general, this is the model most developers would use to build their applications, as they are able to concentrate on creating the business logic rather than having to worry about events and so on. Managed server for Cobol is not available under Windows NT, but the C and C++ versions are.
Top End has its own workbench, a Motif-based development environment that includes a debugger, editor and software for version control. Separate versions of the workbench are available for the languages supported (C, Cobol and C++).
Developers using the Remote Server for MVS component to develop mainframe applications are provided with C header files and a pre-linked library, which enable an application to use Top End CSI/TM function calls. The developer then uses a normal text editor or tool to develop the application code.
There are no limitations on the application, which can be written in C, Cobol or even Natural, and use DBMSs such as Adabas or TP Monitors such as IMS DC or CICS.
Top End Remote Server consists of C language 'include' files, as well as static linkable API library objects. The latter are linked with the application to produce a run-time executable program. During execution, library routines handle the mechanics of communicating remote CSI (client server interface - the Top End API) or transaction manager requests for processing to the Top End node manager. The library routines also handle the return of the associated data.
Where the developer wants to include terminal or other device access to Top End applications, a special tool called Format Management is used to enable screen layouts to be created and customised for a wide variety of devices. An interactive screen builder is included in the toolset. There is also a Format interface facility API that can be used to define the interfaces. The screens are compiled and stored in format libraries, which are then used at run-time to control terminal/device interaction. These tools are not available under Windows NT.
Top End can be used with any tools that support the C language, but it has tight integration with the following tools, using, for example, DLLs:
The integration with the Oracle development environment deserves special mention.
The developer uses the Top End generator to develop an application. The generator reads Top End distributed service information and generates Oracle Form objects. A list of Top End services is displayed to the developer for selection and once a service is selected, a default Form module is generated, with objects corresponding to the Top End distributed service input and output parameters. The developer can then further adapt the generated form modules using the Form Designer tool within Oracle. Once the Form has been generated, the Top End interface is used with Oracle FFI (Oracle foreign function interface) to translate the Top End APIs into Oracle PL/SQL calls accessible to the Oracle Forms developer. The Top End interface also supports other Developer/2000 tools such as Oracle Reports and Oracle Graphics.
A process can act as both a client and a server, but the commands used are context dependent. Top End uses its own API to define the interaction between the client and server.
The developer or administrator uses the Interactive System Definition Tool to generate the application. It is composed of two parts:
The system generation utility is called TPGEN. The developer can select which parts of the definition are transferred to each node and TPGEN will then generate only the components needed on that node. ISD is a Windows-based tool with a graphical interface that takes the developer/administrator step-by-step through the process. All distribution transfer of components to relevant hosts can be managed from this central tool.
The developer can use Top End's own workbench, which includes a debugger and editor, or uses third-party tools such as Micro Focus Cobol or the debugger in Visual C++.
Security and password controls are placed on all the global and local administration tools and TP Secure itself (the tool used to set up security for applications).
Version control is included within the Top End workbench and provided by a third-party tool, Sablime.
There are four ways that Top End can be customised:
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Administration services
NCR provides two tools for the administrator - the Global Administrative Console and the Interactive System Definition tool. In addition to these NCR tools, third-party suppliers have also added tool support: Bay Technologies with TPCharger and Abaci with the Tool Performance Monitor.
The Top End Global Administration tool provides an integrated environment for administering Top End and DCE components. It provides a single operational view of the entire network. The Console can be run from anywhere on the network, and Top End can support more than one concurrently operating console.
The Global Administration Console can be used to manage all server and client applications and the Top End Base Server software. The administrator can also manage a series of servers with different software versions on each. Remote servers and remote clients, however, cannot be managed from the administration tools.
The tools are Unix-based and run under the Motif interface, but Top End/NT offers the same enterprise administration capabilities and user interface semantics as the Unix-based tool.
Top End Windows NT servers can be managed from a Unix-based console and vice versa. The only difference between the Windows NT- and Unix-based versions of the Console are that software distribution (carried out automatically by the Unix Console) is implemented using Microsoft's SMS product under Windows NT.
The Global Administration tool is GUI-based and provides a single operational view of the entire network. Each managed object appears as an icon. Node icons turn red if there is a component failure and by clicking on the icon, the administrator can tunnel down to determine which particular component has failed. The actual component will also be shown as a red icon. By clicking on this icon, the administrator is also able to see the actual failure or alert message. Icons turn green when the Top End Administration tool receives an information auditing message.
If the Global Administration view is unavailable, Top End also provides local control as administrative back-up. Local administration on Unix uses a character-based interface to control the local objects, but the functions available are identical to those using the Global Console. The native Win32 GUI is used for Windows NT. From the Console, the administrator can:
From the same administrative console there are also configuration parameters associated with each server instance that can be dynamically tuned to determine, for example, how many server instances Top End will start-up automatically and at what rate.
A new Top End feature is to be added to the administration tools in the next release called the Systems Management API or 'SMAPI'. SMAPI will provide a programmatic interface to most of Top End's administrative functions and is intended for use in scripts or administrative applications. The aim of the SMAPI is to enable customers to build customised administration programs which, for example, start-up and shut-down application servers based on time, system performance or other events.
Although alerts can already be sent to third-party tools, so in theory Top End's administrative tools can be integrated with third-party system management software, NCR also plans to build in SNMP compliance.
The ISD tool is used during system generation by the developer, but is also a tool used by the administrator to define all the parameters used in the application - such as the load balancing parameters, routing tables and so on. ISD is a Windows-based tool developed using Gupta's (now Centura's) tools, which takes the administrator step-by-step through the process of defining the parameters. It can be used for Unix and Windows NT systems.
In addition to Top End's own tools, Bay Technologies provides a tool called TPCharger. TPCharger is a Unix- (SunOS, Solaris, HP-UX and Pyramid) or PC- (Windows 3, 95 or NT) based tool that captures the service calls from Top End to an application. An agent performs this collecting activity and the data can be collected across nodes.
The data collected can be used to monitor performance, provide costing data based on user profiles, or can be used for auditing. All calls to the TP environment are registered. Certain functions will also trigger the output of resource usage data such as CPU time, elapsed time, returned buffer length and so on. Access to the tool and the data collected is protected by an authentication server. TPCharger also works with Tuxedo.
Code/Title - DTM/Distributed TP Monitors
Chapter E4 - NCR - Top End
Section: Platform support
The remote server for MVS software supports the 32-bit IBM MVS XA or ESA operating system in either basic mode or in TSO mode. IBM TCP/IP for MVS and a C compiler are required.
The Windows NT platform does not support LU6.2 connectivity, remote server support and security services.
The development environment is the same as the run-time environment.
