Partners header

ECOSIM Telematics Applications Project:
Deliverable D04.05

Architecture Assessment Results (updates)

Keywords:
Software, networking, client-server architecture, performance, reliability.
Release 10 November 1997
Author: Kurt Fedra




Synopsis
Programme name Telematics Application Programme
Sector Environment
Project Acronym ECOSIM
Contract number EN 1006
Project title Ecological and environmental monitoring and simulation system for management decision support in urban areas
Deliverable number D04.05
Deliverable title Architecture Assessment Results (updates)
Deliverable version number 0.4
Work package contributing to deliverable 4
Nature of the deliverable Report
Dissemination level Project Deliverable
Contractual date of delivery PM20 (August 1997)
Actual date of delivery PM20 (online)
10 November 1997 (hardcopy)
Author Dr.Kurt Fedra
Environmental Software & Services GmbH
Project technical co-ordinator Dr.Kurt Fedra, Environmental Software & Services GmbH
tel: +43 2252 633 050
fax: +43 2252 633 059
E-mail: kurt@ess.co.at





Executive Summary:

This document summarizes the results of the ECOSIM demonstrator architecture assessment.

It presents a short discussion of the methodology adopted, defines the assessment criteria and evaluation procedures, and then discusses the ECOSIM architecture in terms of the criteria selected.

Contents:





Assessment methodology and criteria

The assessment of the ECOSIM systems architecture is based on a qualitative multi-criteria evaluation approach. The systems architecture (and possibly a set of design alternatives) is evaluated in a peer review process along a set of criteria (dimensions), where the assessment consists in awarding a score between 0 and 100 (representing relative or percentage achievement in a normalized scale from best to worst).

This allows to position a given solution (architecture or design alternative) in an n-dimensional (multi-criteria) performance space, that is normalized to give each of the criteria equal weight within the range of potential achievement (utopia).

The distance of the evaluation (a point in the normalized n-dimensional evaluation space) to the utopia point of the evaluation space defines a single-attribute measure for the evaluation of an architecture or the comparative evaluation of alternatives.

The underlying methodology is described in detail in the ECOSIM Validation Plan.

However, for reasons of practicality and also the constraints imposed by the resources available for these tasks, and in order to simplify the approach, a range of five symbolic labels, ranging, depending on the nature of the variable measured, from

very small small medium large very large
very poor poor average good very good
very low low medium high very high

was chosen.

Assessment Criteria

The criteria for assessment have to be related to both user requirements (performance, ease of use, support of the required functionality, cost of development and maintenance, etc.) as well as generic attributes such as interoperability, platform independence, etc. that may or may not be explicit components of the user requirements.

The following primary criteria have been chosen:

  • Systems flexibility
    measures or describes the ease with which the system can be adapted to different applications (the three validation sites), the ease of inclusion of new and alternative components (different models at the three validation sites, the linkages to legacy software (existing models) and external data structures (existing predefined data structures such as the BLUME data base formats).

  • Ease of use
    measures or describes, from a users point of view, the implications of the architecture on the ease of use of the system; elements include complexity, transparency, and user interface characteristics, as well as issues of interactive use (responsiveness) as far as they are determined by the system's architecture.

  • Development costs
    measures or describes the effort required in systems development, i.e., design, coding, debugging, and operational testing (see also: Integration Plan: Testing and Validation Issues).

  • Maintenance costs
    related to the development costs, measures or estimates the expected cost of system maintenance including issues such as porting to new architectures or operating system releases for the software; and data maintenance and updates (as far as these are affected by the architecture).

  • System performance
    measures the (interactive) responsiveness and the overall computational and display performance of the integrated system as far as this is affected by the system architecture.

  • System reliability
    measures the reliability of operations, for example the ratio between successful and unsuccessful user requests, errors, time-outs, system crashes and forced rebooting, overall availability or up-times.

Architecture Assessment Results and Conclusions

For the assessment, and based on the results of the CEO report on Internet Bandwidth and Latency (see: Deliverable D04.02

The basis for the architecture assessment are:

  • the generic ECOSIM systems architecture (as described in Deliverable D04.04, Architecture description);

  • the Validation Architecture for Berlin (as described in D10.01, Validation Plan);

  • the Validation Architecture for Athens (as described in D10.01, Validation Plan);

  • the Validation Architecture for Gdansk (as described in D10.01, Validation Plan);

The preliminary assessment is summarized below; where appropriate, alternatives for the network connectivity (Internet, dedicated ISDN, LAN) have been considered. Please note that the validation phase is expected to lead to new data and updates to this assessment.

Generic Architecture, Internet connectivity

The assessment is based on average Internet performance as experienced by normal operational practice at ESS, based on a 128 Kb leased line connection (DLL-S) to EUnet as the Internet provider, and a number of experiments between ESS and the GMD.

ASSESSMENT CRITERIA

RANKING

COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent.

Development costs

very high

due to the innovative elements, few ready-made components can be utilized for the system development

Maintenance costs

average

depends on whether the systems is maintained by an institutional end user with internal resources or through outsourcing to a service provider; the latter case should lead to a lower cost on average

System performance

average

here the main issue is the choice of a benchmark: compared to the user expectation based on standard set MS Office tools, the system is slow; compared to the alternative of using all its components as separate, stand alone tools, performance increases of several orders of magnitudes can be achieved.

System reliability

poor

depending on network reliability

Generic Architecture, ISDN connectivity

Please note that maintenance costs (including communication costs for routine operation) depend on the local telecommunication costs, e.g., ISDN phone rates compared to Internet access.

ASSESSMENT CRITERIA

RANKING

COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent.

Development costs

very high

due to the innovative elements, few ready-made components can be utilized for the system development

Maintenance costs

high

assumes that the telephone costs for a long-distance ISDN connection with a 64 Kb transfer rate exceed the cost of local connection to an Internet provider. average

System performance

good

even at 64 Kb the sustained throughput of a dedicated ISDN connection should exceed the average Internet bandwidth considerably.

System reliability

high

depending on quality of ISDN service

Generic Architecture, LAN connectivity

ASSESSMENT CRITERIA RANKING COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent.

Development costs

very high

due to the innovative elements, few ready-made components can be utilized for the system development

Maintenance costs

low

however, the capital costs of the computational resources now required in the LAN have to be considered

System performance

very good

the main constraint here is imposed by the computational resources within the local area network.

System reliability

very high

depending on network reliability





Validation Architecture Berlin, Internet connectivity

Since the atmospheric simulation model MEMO, acting as a pre-processor for the DYMOS modeling system can not be installed in Berlin locally, the Validation Architecture in Berlin depends on external connectivity to MEMO. Also access to the BLUME network database server may require outside connections. However, since the primary model for Berlin, REGOZON and DYMOS, run locally (at the GMS's parallel computers or workstation cluster, respectively), a mixture of dedicated ISDN lines (within the Berlin area) and long-distance connections using the Internet (for the MEMO server) could provide another mixed strategy alternative.

ASSESSMENT CRITERIA

RANKING

COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models and the linkage to the BLUME monitoring network as a legacy system.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent.

Development costs

very high

due to the innovative elements, few ready-made components can be utilized for the system development

Maintenance costs

low

the close cooperation of the Stadtsenat Berlin and the GMD as well as the primarily local connections to the HPC computing resources at the GMD should lead to relatively low maintenance and operational costs.

System performance

good

due to the availability of true HPC (parallel) computers at the GMD, excellent overall performance should be achievable.

System reliability

good

depending on network reliability for access to MEMO.

Validation Architecture Berlin, ISDN connectivity

As above, with improvements expected in systems performance.

Validation Architecture Berlin, LAN connectivity

As above, with additional improvements in systems performance. However, this option would also potentially lead to significantly higher operational costs as it foregoes the use of the HPC equipment at the GMD. This trade-off is likely to make this option infeasible not only because of the purely financial considerations, but because of the manpower requirements for systems maintenance.





Validation Architecture Athens, Internet connectivity

The validation architecture for Athens depends on external linkage to the monitoring network HELLASPAC, the model server POM (University of Athens) and the model server for MEMO/MUSE (Aristotle University of Thessaloniki). Here a combination of dedicated ISDN connections (within the Athens area) and Internet connectivity to Thessaloniki could be evaluated as an additional, intermediate alternative.

ASSESSMENT CRITERIA

RANKING

COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models for the domain air, groundwater, and coastal water.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent.

Development costs

very high

due to the innovative elements, few ready-made components can be utilized for the system development

Maintenance costs

medium

the primary consideration will be the support provided by at least two, possibly three external University partners.

System performance

average

depending on the hardware platform used for the computer servers (e.g., a SG four processor PowerChallenge available at the University of Athens) and the reliability and bandwidth of connectivity.

System reliability

low

depending on network reliability but also Universities as service providers.

Validation Architecture Athens, ISDN connectivity

The ISDN scenario is similar to the Internet scenario above, with expected improvements in performance and reliability, traded against increased operation (communication) costs.

Validation Architecture Athens, LAN connectivity

The feasibility of the LAN solution (institutional and administrative constraints may make the purchase and maintenance of local models difficult) is unclear at the moment. Technically, it would obviously be the most attractive solution until a commercial version of the system with low-bandwidth client support becomes available.



Validation Architecture Gdansk, LAN connectivity

Constrained by the poor Internet connectivity on one hand, and the restricted data availability for use of the complex models DYMOS and MEMO, the Validation Architecture for Gdansk only foresees the use of local area network connections.

ASSESSMENT CRITERIA

RANKING

COMMENTS

Systems flexibility

high

applicability of the same basic architecture to all three case studies indicates a high degree of flexibility, as does the possibility to integrate a range of different simulation models. Restrictions in Gdansk are primarily due to data constraints.

Ease of use

high

the fully interactive and menu driven is easy to use, access to external models and the underlying complexity is completely transparent. The ability to accommodate alternative interface languages would contribute to ease of use by city officials.

Development costs

very high

as above

Maintenance costs

high

since only the less compute intensive models like REGOZON and POM (possibly also the groundwater models) are foreseen for use in Gdansk, the availability of more expensive high-performance computers and their costs is less of an issue.

System performance

very good

even a standard 10 Mb Ethernet would provide more than enough bandwidth for all foreseeable model applications.

System reliability

very high

depending on overall systems reliability; use of a UPS unit for the systems hardware elements might be advisable.





Conclusions

The ECOSIM architecture has its main advantages in its flexibility, and ease of use through the transparent integration of different tools and information resources. Disadvantages are the comparatively high cost of the solution, which implies expensive customization.

The system depends in its performance and reliability both on the network connections, and the computational resources of its external information resources. A clear trade-off, as expected, between costs and reliability can be expected. However, cost and performance are not at all clearly related, since the access to shared high-performance computing resources is cost effective, its effectiveness however constrained by the network reliability and bandwidth.

However, with the rapid growth of the Internet and related services, this problem is likely to disappear by the time commercial solutions derived from ECOSIM will reach the market.

Manpower constraints and administrative problems likely to be encountered in public administrations may lead to outsourcing of a complete set of services including the provision of the necessary computational infrastructure as an attractive option and commercial opportunity for information service providers.


Copyright 1995-2002 by:   ESS   Environmental Software and Services GmbH AUSTRIA