Computing architectures are becoming increasingly complex and potent, as we reach new computing capacities. Currently the first three machines in the TOP500 list are exascale systems. To be able to take full advantage of these machines, and even run on such heterogeneous architectures, it has become imperative for computational simulations to be massively parallelized and hardware portable. The IPPL framework is a C++ library which allows users to write Particle-Mesh simulations in a performant and portable way, as it contains all the building blocks required for such simulations, such as meshes, particles, interpolation, and field solvers. We present a new solver implemented inside IPPL, a matrix-free FEM-based Poisson solver working from 1 to 3 dimensions, implemented in a fully portable way using Kokkos. The Finite Element framework implemented in IPPL is itself done in a modular way, allowing one to build on top of it to solve other PDEs. We showcase portability and performance by running scaling studies on the JEDI machine at Jülich Supercomputing Center, which has the new Nvidia GH200 GPUs, as well as the JURECA machine of the same center, with Nvidia A100 GPUs.