Distributed computing

This article outlines the process of running a large simulation job across several computers on a local network or cluster. This feature is available for FDTD solver and varFDTD solver in MODE.

mceclip0.png

Important

  • Configure your system first, as shown in the Getting started with HPC page. 
  • This scenario, running 1 single simulation across several computers or nodes will only require 1 FDTD license. 

Using the CAD/GUI

  1. Create or open your simulation project
  2. Open Resources
  3. Specify the HostName or IP of the machine where you want to remotely distribute your jobs, separated by 'comma'
  4. Indicate the number of processes for each of the machines, separated by a 'comma'.
  5. And the number of threads per process to use for each resource on the list.
  6. In the advanced configuration use the following 'Job launching preset';
    • Windows
      • Remote: MPICH2 - for machines with less than 64 cores and only 1 CPU socket.
      • Remote: Intel MPI - any machine, especially with 2 or more CPU sockets or more than 64 cores.
      • Custom - if using your own MPI installation (e.g. clusters or HPC)
      • Do Not Use Remote: Microsoft MPI or Local Computer as the job launching preset
    • Linux and macOS
      • Remote: MPICH2 - works for most machines and networks (default)
      • Custom - if using your own MPI installation (e.g. clusters or HPC)
      • Do Not Use Local Computer as the job launching preset
  7. In the example shown, the simulation is split between 2 computers, the local computer (localhost) and 2nd computer (otus).
  8. Both computers use 2 processes and 1 thread per process. 
    mceclip0.png
    mceclip2.png 

Note

Before trying to distribute a single simulation across two computers, ensure that simulations run properly on each computer individually.

Using the command prompt

  1. Open a command prompt (Windows) or terminal (Linux and macOS)
  2. The simulation runs on 2 computers using 4 processes/core on each computer/node. 
    Run the MPI and the solver/engine commands;
    • Windows 
      "C:\Program Files\Lumerical\MPICH2\mpiexec.exe" -hosts 2 node1 4 node2 6 "C:/Program Files/Lumerical/2019b/bin/fdtd-engine.exe" -t 1 "C:\[path]\simulationfile.fsp"
      The above runs your simulation on 2 nodes as indicated by '-hosts 2', compute 'node1' using 4 processes, 'node2' with 6 processes.
    • Linux   
      /opt/lumerical/fdtd/mpich2/nemesis/bin/mpiexec -hosts node01:4,hosts02:8,node03:16 /opt/lumerical/2019b/bin/fdtd-engine-mpich2nem -t 1 /[path]/simulationfile.fsp
      This will run your simulation on 3 hosts, 'node01' with 4 processes, 'hosts02' using 8 processes and 'node03' with 16 processes.
    • macOS  
      /opt/lumerical/mpich2/bin/mpiexec -hosts IP_node1:4,IP_node2:4 /Applications/Lumerical\ 2019b.app/Contents/Applications/FDTD\ Solutions.app/Contents/MacOS/fdtd-engine -t 1 /simulation.fsp 
      This runs your simulation on 2 nodes using 4 processes each.

For more information on the command-line options, see, Running simulations from the command line for your operating system.

 

Related articles