Configuration in Complex Networks¶
Default Operation using ZMQ core¶
The starting place for many HELICS-based co-simulation is running on a single computer using the default core, ZMQ. The ZMQ core provides a lot of advantages but its behavior on the localhost network is more complex and this complexity is often hidden when a user is only running the co-simulation on a single computer.
When starting up a HELICS-based co-simulation using the ZMQ core, HELICS opens two ports: one for high-priority traffic (default to port number 23405) and the other for low priority traffic (defaults to 23406). As federates join the co-simulation they connect to the broker on the high-priority port and the broker assigns the federate a dedicated port that all further communication between the broker and federate takes place.
Again, when all federates are running on a single computer, this prolific use of port numbers (at high federate counts) is generally not a problem. There may need to be permission granted on the local computer to open those ports in a firewall for localhost traffic but no traffic ever leaves the local machine.
Speciality Cores for Complex Networks¶
When a co-simulation grows to the point where it starts spanning multiple compute nodes, the default ZMQ core may start running into networking problems. In situations where all the compute nodes are still in the same subnet or administered by the same organization, IT policies may easily accommodate the ZMQ core’s need for ports. This may not always be the case, though, and to allow HELICS to operate in these environments, the HELICS developers have created two speciality cores to simplify the impact of HELICS in a networked environment.
The zmq_ss core is a version of the ZMQ core with modified behavior to only use a single socket. (A socket is the combination of ip address and port.) The core has been designed to accommodate a large number of federates where there is a possibility of running out of available ports on a single compute node (ip address). By using a single socket, it has the side-affect of also simplifying the required networking and/or firewall configuration.
The tcp_ss core is similar in nature to the zmq_ss core in that it uses a single socket but is based on the tcp core. This core removes the extra complexity of the zmq core and just uses the tcp protocol directly and has been designed as the go-to core when needing to work in complex networking environments. In addition to only using a single socket, the tcp core allows the broker to initiate connections with federates which can be important when trying to work in networking environments when firewalls prevent connections to be initiated in particular directions.
Regardless of which core you’re using, there are a few specific networking options that allow for changes to default values to enable working in a more restrictive networking environment.
broker_port- for sub-brokers or federates, defines the IP address (
broker_address) and port (
broker_port) which should be used to connect to a parent broker
local_port- defines the IP address (
local_interface) and port (
local_port) where a broker, sub-broker, or federate will look for connections to the federation.
broker_port is typically defined as a command line switch when instantiating the broker.
broker_port are also used as options when configuring federates to define the socket they should connect to. These options can be set as part of the
core_init_string or part of the JSON configuration for that federate. The
local_port options can be similarly configured. Further details on these configuration methods can be found in the Configuration Options Reference.
It is also possible to include the port number when defining
broker_address (effectively defining the broker’s socket) in addition to the IP address such as in the format:
126.96.36.199:23400. Doing so would then not require the
broker_port option to be defined.
As of this writing there is a generic
port option supported in HELICS that tries to be all things to all people. Experience has shown that though well-intentioned, the feature of it being generic has become a bug in that it causes confusion among users. You may still see it lurking in examples or documentation but it is recommended that its use be avoided and the more explicit
local_port be used instead.
Broker Hierarchies and Multi-Computer Federations¶
Once a federation reaches a certain size, it is not unusual for it to end up deployed across multiple compute nodes and often this results in establishing a broker hierarchy to reduce traffic between compute nodes (and help the co-simulation to run faster). In complex networking environments, this will likely entail the use of the
tcp_ss core and the specification of the broker and sub-broker sockets.
The good news is that we already have two other pages of documentation devoted to this and both include running examples that show how these features can be put to use. Here’s the documentation on broker hierarchies and here’s the one on running across multiple compute nodes.