Distributed computing

This article outlines the process of running a large simulation job across several computers on a local network or cluster. This feature is available for FDTD solver and varFDTD solver in MODE.

distributed_computing_overview.png

Note:
  • Configure your system first, as shown in the Getting started with HPC page. 
  • This scenario, "running 1 single simulation across several computers or nodes" will only require 1 FDTD license.
  • Before trying to distribute a single simulation across multiple computers, ensure that simulations run properly on each computer individually.

Using the CAD/GUI

  1. Create or open your simulation project
  2. Open "Resources"
  3. Specify the HostName or IP of the machine where you want to remotely distribute your jobs, separated by a 'comma'
  4. Indicate the number of processes for each of the machines, separated by a 'comma'.
  5. And the number of threads per process to use for each resource on the list.
  6. In the advanced configuration use the following 'Job launching preset';
    • Windows
      • Remote: MPICH2 - for machines with less than 64 cores and only 1 CPU socket.
      • Remote: Intel MPI - any machine, especially with 2 or more CPU sockets or more than 64 cores.
      • Custom - if using your own MPI installation (e.g. clusters or HPC)
      • Do Not Use Remote: Microsoft MPI or Local Computer as the job launching preset
    • Linux and macOS
      • Remote: MPICH2 - works for most machines and networks (default)
      • Custom - if using your own MPI installation (e.g. clusters or HPC)
      • Do Not Use Local Computer as the job launching preset
  7. In the example shown, the simulation is split between 2 computers, the local computer (localhost) and 2nd computer (otus).
  8. Both computers use 2 processes and 1 thread per process. 

dist_comp_resource_config.png

dist_comp_resource_adv.png

Run the simulation directly from the command line

  • Open a command prompt (Windows) or terminal (Linux and macOS)
  • Run the simulation/project file using the command to execute
  • Examples are shown here for all three supported platforms running with different or the same number of processes/cores on each machine/node.

Note: If your network is not able to resolve the HostNames of your machines, use the IP_Address instead.

Windows:

"C:\Program Files\Lumerical\MPICH2\mpiexec.exe" -hosts 2 node1 4 node2 6 "C:/Program Files/Lumerical/2019b/bin/fdtd-engine.exe" -t 1 "C:\[path]\simulationfile.fsp"

This runs your simulation on 2 nodes as indicated by '-hosts 2', compute 'node1' using 4 processes, 'node2' with 6 processes.

Linux:

/opt/lumerical/fdtd/mpich2/nemesis/bin/mpiexec -hosts node01:4,hosts02:8,node03:16 /opt/lumerical/2019b/bin/fdtd-engine-mpich2nem -t 1 /[path]/simulationfile.fsp

This will run your simulation on 3 hosts, 'node01' with 4 processes, 'hosts02' using 8 processes and 'node03' with 16 processes.

macOS:

/opt/lumerical/mpich2/bin/mpiexec -hosts IP_node1:4,IP_node2:4 /Applications/Lumerical\ 2020a.app/Contents/Applications/FDTD\ Solutions.app/Contents/MacOS/fdtd-engine -t 1 /[path]/simulationfile.fsp

This runs your simulation on 2 nodes using 4 processes each.

See also

Running simulations from the command line

Related articles