Skip to main content

This website uses cookies to ensure you get the best experience.

Virtasant and our selected partners use cookies and similar technologies (together “cookies”) that are necessary to present this website, and to ensure you get the best experience of it. If you consent to it, we will also use cookies for analytics purposes.

See our Cookie Policy to read more about the cookies we set.

You can withdraw and manage your consent at any time, by clicking “Manage cookies” at the bottom of each website page.

Select which cookies you accept

On this site, we always set cookies that are strictly necessary, meaning they are necessary for the site to function properly.

If you consent to it, we will also set other types of cookies. You can provide or withdraw your consent to the different types of cookies using the toggles below. You can change or withdraw your consent at any time, by clicking the link “Manage Cookies”, which is always available at the bottom of the site.

To learn more about what the different types of cookies do, how your data is used when they are set etc, see our Cookie Policy.

These cookies are necessary to make the site work properly, and are always set when you visit the site.

Vendors Teamtailor

These cookies collect information to help us understand how the site is being used.

Vendors Teamtailor
Technology Operations · USA, Canada, Latin America · Fully Remote

Senior Serverless Spark Migration Engineer

Senior Platform Engineer, Cloud Infrastructure

Type: Remote (Brazil, Mexico)
Coverage: Pacific Hours (8:00 AM – 5:00 PM PST)

About Virtasant

Virtasant is a global cloud and technology services company helping organizations modernize, optimize, and build at scale. We work with enterprise customers on complex cloud, data, infrastructure, and AI initiatives, bringing together deep technical expertise and hands-on delivery.

About the Role

We’re looking for a Senior Serverless Spark Migration Engineer to help modernize a large-scale enterprise data platform.

You’ll lead the migration of production Apache Spark workloads from on-premise Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP. This is a hands-on engineering role spanning workload assessment, architecture, application refactoring, migration execution, performance optimization, automation, and production readiness.

The goal is not simply to lift and shift existing workloads. You’ll determine the right target architecture for each workload and establish repeatable patterns that can eventually support migration at significant enterprise scale.

What You’ll Do

  • Lead migrations of enterprise Spark workloads from on-premise environments to AWS and GCP.

  • Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.

  • Determine the right migration approach across rehost, replatform, refactor, modernize, or retire.

  • Modernize traditional cluster-based workloads for serverless Spark where appropriate.

  • Design and implement architectures using technologies such as AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery.

  • Refactor legacy PySpark/Scala/Spark SQL applications for cloud portability, scalability, and reliability.

  • Migrate workloads from environments using Hadoop, HDFS, YARN, Hive, and on-prem Spark clusters.

  • Troubleshoot and optimize Spark workloads across partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.

  • Benchmark performance and optimize serverless workloads for performance, reliability, and cloud cost.

  • Build reusable migration tooling, automation, templates, and frameworks.

  • Implement CI/CD and Infrastructure as Code using tools such as Terraform.

  • Define testing, validation, cutover, rollback, observability, and production-readiness patterns.

  • Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.

What We’re Looking For

  • 8+ years of experience across data engineering, distributed systems, cloud engineering, or platform engineering.

  • 5+ years of hands-on Apache Spark experience in enterprise environments.

  • Strong PySpark and/or Scala development experience.

  • Proven experience migrating large-scale Spark workloads between infrastructure platforms.

  • Hands-on experience with both AWS and GCP.

  • Experience with on-premise Hadoop/Spark ecosystems, including technologies such as HDFS, YARN, and Hive.

  • Deep understanding of Spark internals and distributed processing.

  • Strong SQL and data engineering fundamentals.

  • Experience with cloud data lakes and object storage.

  • Strong production troubleshooting and performance-tuning experience.

  • Experience with CI/CD, Git, and Infrastructure as Code.

  • Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization.

Nice to Have

Experience with EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is valuable.

What Success Looks Like

You can take ownership of the complete migration lifecycle:

Discover → Assess → Design → Refactor → Migrate → Validate → Optimize → Operate

You understand both the legacy Hadoop/Spark world and modern cloud-native data platforms, and can make pragmatic architecture decisions based on workload characteristics rather than simply reproducing an existing environment in the cloud.

Our team
Technology Operations
Locations
USA, Canada, Latin America
Remote status
Fully Remote
Picture of Kathleen Aponte
Contact Kathleen Aponte Senior Technical Recruiter
Technology Operations · USA, Canada, Latin America · Fully Remote

Senior Serverless Spark Migration Engineer

Loading application form

Already working at Virtasant?

Let’s recruit together and find your next colleague.