Diagnostics : EM fields on particles #5637

grobertdautun · 2025-02-04T15:25:17Z

This PR implements the EM option for particles diagnostics output. It is an extension of #4599 and #4839 and works in the same way. It allows the user to output the electromagnetic fields seen by particles.

I added a test case test_2d_particle_EM_diagnostics with its analysis. This case represents a single particle fixed in a box traversed by a laser pulse, and the analysis verifies that the fields are matching the theoretical ones.

I also tested the diagnostics on bigger cases, and I noticed that when writing a lot of timesteps with a lot of particles, reading the diagnostics with adios2 file-based encoding is very slow due to the high number of opening/closing files. Moreover, it appears that openpmd-viewer does not support reading variable-based encoding, so I wrote some scripts to read them directly using the adios2 python API, that I added in Tools/PostProcessing/read_variable_based_adios2.py

There is a small update in the documentation to add the description of both the diagnostics and the script

This diagnostics does not support mesh refinement at the moment, this should be added in a follow up PR

…particles

for more information, see https://pre-commit.ci

…particles_2

for more information, see https://pre-commit.ci

…rpX into diags_em_on_particles_2

Source/Diagnostics/ParticleIO.cpp

ax3l · 2025-02-04T18:38:55Z

Source/Diagnostics/ParticleIO.cpp

+    tmp.NewRealComp("Ex");
+    tmp.NewRealComp("Ey");
+    tmp.NewRealComp("Ez");
+    tmp.NewRealComp("Bx");
+    tmp.NewRealComp("By");
+    tmp.NewRealComp("Bz");
+
+    int const Ex_index = tmp.getParticleComps().at("Ex");


There will be a minimal change we need to add to this PR once we merged #5481 (simplification of named SoA bookkeeping), but it will be easy to update once your PR is ready/out-of-WIP.

Source/Diagnostics/ParticleIO.cpp

Source/Diagnostics/WarpXOpenPMD.cpp

Source/Diagnostics/ParticleIO.cpp

for more information, see https://pre-commit.ci

Co-authored-by: Luca Fedeli <[email protected]>

for more information, see https://pre-commit.ci

Removed unnecessary comments

for more information, see https://pre-commit.ci

RemiLehe · 2025-02-13T17:26:37Z

Docs/source/usage/parameters.rst

@@ -2916,9 +2916,9 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
 * ``<diag_name>.<species_name>.variables`` (list of `strings` separated by spaces, optional)
    List of particle quantities to write to output.
    Choices are ``x``, ``y``, ``z`` for the particle positions (3D and RZ), ``x`` & ``z`` in 2D, ``z`` in 1D,
-    ``w`` for the particle weight and ``ux``, ``uy``, ``uz`` for the particle momenta.
+    ``w`` for the particle weight, ``ux``, ``uy``, ``uz`` for the particle momenta, and ``EM`` for the electromagnetic fields.


Could we instead allow the user to pass Ex, Ey, Bx, etc.

ax3l · 2025-02-13T17:10:51Z

Source/Diagnostics/ParticleIO.cpp

+                Ex_particle_arr[ip] = 0._rt;
+                Ey_particle_arr[ip] = 0._rt;
+                Ez_particle_arr[ip] = 0._rt;


alignment is off by one whitespace 😅

ax3l · 2025-02-13T17:22:54Z

Tools/PostProcessing/read_variable_based_adios2.py

@@ -0,0 +1,145 @@
+import numpy as np
+from adios2 import Stream


As discussed: we could use openPMD-api.

I think for openPMD-viewer support, we rely on this upcoming feature: openPMD/openPMD-api#1706

Current ADIOS2 support status:
https://openpmd-api.readthedocs.io/en/0.16.1/backends/adios2.html

@franzpoeschel, @guj @pnorbert can you remind me if we can do variable-based encoding without steps (i.e., can we already set something during write that allows-us to read variable based encoding with random access)?

(i.e., can we already set something during write that allows-us to read variable based encoding with random access)

Not yet, that is what #1706 (and follow-ups for changing metadata) will bring (setting a flag during write will not be needed though).
Until that point, it's better to continue using file encoding for random access.

RemiLehe · 2025-02-13T17:26:59Z

Docs/source/usage/parameters.rst

@@ -2784,7 +2784,7 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
 * ``<diag_name>.openpmd_encoding`` (optional, ``v`` (variable based), ``f`` (file based) or ``g`` (group based) ) only read if ``<diag_name>.format = openpmd``.
     openPMD `file output encoding <https://openpmd-api.readthedocs.io/en/0.16.1/usage/concepts.html#iteration-and-series>`__.
     File based: one file per timestep (slower), group/variable based: one file for all steps (faster)).
-     ``variable based`` is an `experimental feature with ADIOS2 <https://openpmd-api.readthedocs.io/en/0.16.1/backends/adios2.html#experimental-new-adios2-schema>`__ and not supported for back-transformed diagnostics.
+     ``variable based`` is an `experimental feature with ADIOS2 <https://openpmd-api.readthedocs.io/en/0.16.1/backends/adios2.html#experimental-new-adios2-schema>`__ and not supported for back-transformed diagnostics. This format is also not supported by OpenPMDTimeSeries at the moment, the script :download:`read_variable_based_adios2.py <../../../Tools/PostProcessing/read_variable_based_adios2.py>` provides basic functions to read variable-based particle diagnostics directly with the ``adios2`` module.


This could go in a separate PR. Could you remove from this one?

RemiLehe · 2025-02-13T17:27:07Z

Tools/PostProcessing/read_variable_based_adios2.py

@@ -0,0 +1,145 @@
+import numpy as np


This file could go in a separate PR. Could you remove from this one?

RemiLehe · 2025-02-13T17:27:43Z

Source/Diagnostics/ParticleIO.cpp

+                Ez_particle_arr[ip] = 0._rt;
+                Bx_particle_arr[ip] = 0._rt;
+                By_particle_arr[ip] = 0._rt;
+                Bz_particle_arr[ip] = 0._rt;


Could we add the external fields as part of this call?

See for instance: https://github.com/ECP-WarpX/WarpX/blob/development/Source/Particles/ElementaryProcess/Ionization.H#L115

RemiLehe · 2025-02-13T17:28:47Z

Source/Diagnostics/ParticleIO.cpp

+    amrex::MultiFab const& Ex = *warpx.m_fields.get(FieldType::Efield_fp, Dir{0}, lev0);
+    amrex::MultiFab const& Ey = *warpx.m_fields.get(FieldType::Efield_fp, Dir{1}, lev0);
+    amrex::MultiFab const& Ez = *warpx.m_fields.get(FieldType::Efield_fp, Dir{2}, lev0);
+    amrex::MultiFab const& Bx = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{0}, lev0);
+    amrex::MultiFab const& By = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{1}, lev0);
+    amrex::MultiFab const& Bz = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{2}, lev0);


Suggested change

amrex::MultiFab const& Ex = *warpx.m_fields.get(FieldType::Efield_fp, Dir{0}, lev0);

amrex::MultiFab const& Ey = *warpx.m_fields.get(FieldType::Efield_fp, Dir{1}, lev0);

amrex::MultiFab const& Ez = *warpx.m_fields.get(FieldType::Efield_fp, Dir{2}, lev0);

amrex::MultiFab const& Bx = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{0}, lev0);

amrex::MultiFab const& By = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{1}, lev0);

amrex::MultiFab const& Bz = *warpx.m_fields.get(FieldType::Bfield_fp, Dir{2}, lev0);

amrex::MultiFab const& Ex = *warpx.m_fields.get(FieldType::Efield_aux, Dir{0}, lev0);

amrex::MultiFab const& Ey = *warpx.m_fields.get(FieldType::Efield_aux, Dir{1}, lev0);

amrex::MultiFab const& Ez = *warpx.m_fields.get(FieldType::Efield_aux, Dir{2}, lev0);

amrex::MultiFab const& Bx = *warpx.m_fields.get(FieldType::Bfield_aux, Dir{0}, lev0);

amrex::MultiFab const& By = *warpx.m_fields.get(FieldType::Bfield_aux, Dir{1}, lev0);

amrex::MultiFab const& Bz = *warpx.m_fields.get(FieldType::Bfield_aux, Dir{2}, lev0);

RemiLehe · 2025-02-13T17:46:52Z

Source/Diagnostics/WarpXOpenPMD.cpp

    if ( particle_diag.m_plot_phi ) {
        storePhiOnParticles( tmp, WarpX::electrostatic_solver_id, !use_pinned_pc );
    }
+    if ( particle_diag.m_plot_EM ) {
+        storeEMFieldsOnParticles( tmp, WarpX::electromagnetic_solver_id, !use_pinned_pc );
+    }


Suggested change

if ( particle_diag.m_plot_phi ) {

storePhiOnParticles( tmp, WarpX::electrostatic_solver_id, !use_pinned_pc );

}

if ( particle_diag.m_plot_EM ) {

storeEMFieldsOnParticles( tmp, WarpX::electromagnetic_solver_id, !use_pinned_pc );

}

// Detect whether this is a FullDiagnostic (use_pinned_pc = 0) or

// a BoundaryScrapingDiagnostic/BackTransformedDiagnostic (use_pinned_pc=1)

bool const is_full_diagnostic = !use_pinned_pc;

if ( particle_diag.m_plot_phi ) {

storePhiOnParticles( tmp, WarpX::electrostatic_solver_id, is_full_diagnostic);

}

if ( particle_diag.m_plot_EM ) {

storeEMFieldsOnParticles( tmp, WarpX::electromagnetic_solver_id, is_full_diagnostic);

}

RemiLehe · 2025-02-14T00:57:42Z

Source/Diagnostics/ParticleIO.cpp

+                Ez_particle_arr[ip] = 0._rt;
+                Bx_particle_arr[ip] = 0._rt;
+                By_particle_arr[ip] = 0._rt;
+                Bz_particle_arr[ip] = 0._rt;


See for instance: https://github.com/ECP-WarpX/WarpX/blob/development/Source/Particles/ElementaryProcess/Ionization.H#L115

grobertdautun and others added 16 commits January 17, 2025 14:43

add m_plot_EM flag

e9a44ff

create storeEMFieldsOnParticles function

bb1947c

First particle EM fields diagnostics draft (not working)

3638175

Merge remote-tracking branch 'upstream/development' into diags_EM_on_…

7179fde

…particles

fixed nan

dafb2ac

added CI tests for EM diags on particles

c010ec5

[pre-commit.ci] auto fixes from pre-commit.com hooks

1ae4b55

for more information, see https://pre-commit.ci

deleted unused variables

0142f2e

Merge remote-tracking branch 'origin/diags_EM_on_particles' into HEAD

536756f

Merge remote-tracking branch 'upstream/development' into diags_em_on_…

a0d50b5

…particles_2

Fixed example lambda function and inputs name

bb5a285

[pre-commit.ci] auto fixes from pre-commit.com hooks

addaf67

for more information, see https://pre-commit.ci

removed unused variable geom

4de0408

Merge branch 'diags_em_on_particles_2' of github.com:grobertdautun/Wa…

3f5062a

…rpX into diags_em_on_particles_2

Fixed undefined identifier

3c08a8b

fixed no member error

18275a2

github-advanced-security bot found potential problems Feb 4, 2025

View reviewed changes

Source/Diagnostics/ParticleIO.cpp Fixed Show fixed Hide fixed

ax3l added the component: diagnostics all types of outputs label Feb 4, 2025

ax3l reviewed Feb 4, 2025

View reviewed changes

ax3l requested review from atmyers and RemiLehe February 4, 2025 18:39

lucafedeli88 reviewed Feb 5, 2025

View reviewed changes

Source/Diagnostics/ParticleIO.cpp Outdated Show resolved Hide resolved

lucafedeli88 reviewed Feb 5, 2025

View reviewed changes

Source/Diagnostics/WarpXOpenPMD.cpp Show resolved Hide resolved

lucafedeli88 reviewed Feb 5, 2025

View reviewed changes

Source/Diagnostics/WarpXOpenPMD.cpp Outdated Show resolved Hide resolved

lucafedeli88 reviewed Feb 5, 2025

View reviewed changes

Source/Diagnostics/ParticleIO.cpp Outdated Show resolved Hide resolved

grobertdautun and others added 5 commits February 5, 2025 11:22

Removed adios2 from test

8c50b0b

[pre-commit.ci] auto fixes from pre-commit.com hooks

c381d56

for more information, see https://pre-commit.ci

Update Source/Diagnostics/ParticleIO.cpp

800494e

Co-authored-by: Luca Fedeli <[email protected]>

Update Source/Diagnostics/ParticleIO.cpp

041e9a8

Co-authored-by: Luca Fedeli <[email protected]>

Update Source/Diagnostics/WarpXOpenPMD.cpp

4a7f71e

Co-authored-by: Luca Fedeli <[email protected]>

grobertdautun and others added 5 commits February 5, 2025 17:36

Added postprocessing script and updated documentation

8500064

[pre-commit.ci] auto fixes from pre-commit.com hooks

3b11e01

for more information, see https://pre-commit.ci

removed comment

a0cae43

Update analysis_2d.py

1aea782

Removed unnecessary comments

[pre-commit.ci] auto fixes from pre-commit.com hooks

cab9f6c

for more information, see https://pre-commit.ci

grobertdautun changed the title ~~[WIP] Diagnostics : EM fields on particles~~ Diagnostics : EM fields on particles Feb 6, 2025

ax3l assigned RemiLehe Feb 11, 2025

grobertdautun and others added 4 commits February 13, 2025 14:02

Merge branch 'development' into diags_em_on_particles_2

f8b9335

added output docstring in script

7644c3a

fixed merge conflict

6c601be

[pre-commit.ci] auto fixes from pre-commit.com hooks

86f86d8

for more information, see https://pre-commit.ci

lucafedeli88 requested a review from ax3l February 13, 2025 13:11

grobertdautun added 2 commits February 13, 2025 14:17

updated syntax for runtime components

028b139

fix args in GetRealCompIndex

dccd20c

RemiLehe reviewed Feb 13, 2025

View reviewed changes

ax3l reviewed Feb 13, 2025

View reviewed changes

RemiLehe reviewed Feb 13, 2025

View reviewed changes

RemiLehe reviewed Feb 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Diagnostics : EM fields on particles #5637

Diagnostics : EM fields on particles #5637

grobertdautun commented Feb 4, 2025 •

edited

Loading

ax3l Feb 4, 2025

RemiLehe Feb 13, 2025

ax3l Feb 13, 2025

ax3l Feb 13, 2025

franzpoeschel Feb 14, 2025

RemiLehe Feb 13, 2025

RemiLehe Feb 13, 2025

RemiLehe Feb 13, 2025

RemiLehe Feb 14, 2025

RemiLehe Feb 13, 2025 •

edited

Loading

RemiLehe Feb 13, 2025

RemiLehe Feb 14, 2025

		@@ -0,0 +1,145 @@
		import numpy as np
		from adios2 import Stream

Diagnostics : EM fields on particles #5637

Are you sure you want to change the base?

Diagnostics : EM fields on particles #5637

Conversation

grobertdautun commented Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RemiLehe Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grobertdautun commented Feb 4, 2025 •

edited

Loading

RemiLehe Feb 13, 2025 •

edited

Loading