Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.
Difference between revisions of "Interop Testing Plan"
(→Test Broker) |
(→Test Broker) |
||
| Line 49: | Line 49: | ||
As the broker is executed, it notes when it encounters any embedded conformance statements, and will display at the end of its run coverage figures, like this: | As the broker is executed, it notes when it encounters any embedded conformance statements, and will display at the end of its run coverage figures, like this: | ||
| − | {{ | + | {{{Code| |
| − | INFO 20140203 131616 exceptions 2 out of 17 11% | + | INFO 20140203 131616 exceptions 2 out of 17 11%<br> |
| − | INFO 20140203 131616 coverages 35 out of 55 63% | + | INFO 20140203 131616 coverages 35 out of 55 63% |
| − | }} | + | }}} |
Revision as of 10:47, 3 February 2014
The test material is based upon the current OASIS consultation draft of the specification.
The test material described below can be found in the Paho testing git repository.
Conformance Statements
The specification contains a list of identified conformance statements, which are labelled with sequence numbers. Although the statements are to a certain extent arbitrary, due to the use of language in the specification (wherever the words MUST or SHOULD are used), nevertheless they are a good place to start in determining conformance. I anticipate adding more statements to this list in due course, but for now we will go with the list as identified in the current version of the specification.
I have extracted the list into a spreadsheet where tests for both MQTT clients and servers are briefly described. I am using this as a coverage checklist to view progress of the current test material.
Model Based Testing
I have taken the approach of what can be described as model based testing. A fully automated test at its core is comprised of two sets of data:
- input sequences
- expected results, or outputs.
MQTT is a client/server protocol, meaning that in general there are two components, clients and servers to test.
Test Material Components
I'll start by describing the components of the test material.
Test Broker
A test or "model" broker is in the package mqtt/broker. The first question I will be asked (thanks Ian Skerrett) is why write another broker, when Mosquitto, RSMB or others already exist. The reasons are:
- I already had one written before (but that is not really important)
- a model broker need not have all the implementation details of a production server
- the model is seeded with conformance statements - it knows when they have been executed
- the model is written in Python and is intended to be simple and serve as an illustration and clarification for the specification.
I hope the source can be scrutinized and discussed by the community, both for understanding the specification, and discussion of MQTT server implementation. The source consists of the following files:
- start.py - Python socket server wrapper for the MQTT code
- MQTTV311.py - MQTT packet formatting
- MQTTBrokers.py - MQTT packet interactions of the broker
- Brokers.py - core broker behaviour, without MQTT packet details
- SubscriptionEngines.py - subscription and retained message processing
- Subscriptions.py - handling of individual subscriptions
- Topics.py - validation and matching of topics
I've put these into a literate programming system for a pretty printed display. I intend to add descriptions to this source, so that the reasons for the implementation are explained, and also to allow comments on the source pages.
As the broker is executed, it notes when it encounters any embedded conformance statements, and will display at the end of its run coverage figures, like this:
INFO 20140203 131616 exceptions 2 out of 17 11%
INFO 20140203 131616 coverages 35 out of 55 63%