Job Details

Solutions Architect: Databricks & Big Data

Who Are We?

SDG Group is a global consulting powerhouse with over 30 years of experience as a Market Leader in Data & Analytics. As we continue to push the expansion of our US business, we are seeking a Solutions Architect to lead technical delivery and innovation for our most ambitious client engagements.

Why Join Us?

In this role, you will be the technical backbone of our Databricks offering . You will design sophisticated, large-scale Big Data architectures and actively engage with the Databricks organization in close collaboration with our Alliances Lead to showcase SDG's premier technical capabilities. You are not just building pipelines; you are architecting the future of data intelligence for the world's leading brands.

• Innovation-First Culture: We don't just use technology; we evaluate and adopt the most disruptive tools in the market before they become mainstream.
• Global Reach, Local Impact: Benefit from the resources of a 2,500+ person global firm while playing a pivotal role in our targeted US expansion.
• Continuous Evolution: Enjoy a meritocratic career path where your technical contributions are recognized and your growth is supported through official certifications and specialized training.

What You'll Be Doing

• Advanced Big Data Architecture: Design and define end-to-end, high-performance architectures for processing massive, complex volumes of data in distributed environments. Establish standards for Data Lakehouse and Modern Data Warehouse patterns to ensure scalability and efficiency.
• Storage Strategy: Implement advanced storage strategies using Delta Lake , Apache Iceberg , and optimized formats like Parquet.
• Databricks Solution Design: Lead the technical design of solutions on the Databricks Data Intelligence Platform, utilizing Unity Catalog for unified governance.
• Future-Proofing: Demonstrate deep experience with Delta Live Tables (DLT) and maintain awareness of the upcoming Lakeflow ecosystem (Spark Declarative Pipelines) to ensure SDG remains at the cutting edge of the roadmap.
• Performance Tuning: Provide expert guidance on performance tuning for large Spark clusters and optimizing SQL workloads within Databricks.
• Client Engagement: Act as the "Technical North Star" in client workshops, translating strategic business needs into viable, high-impact technical blueprints.
• Alliance Collaboration: Collaborate with the Alliances Lead to engage with Databricks' technical teams, demonstrating SDG's excellence through deep-dive architectural reviews and Proofs of Concept (PoCs).
• Technical Excellence: Support project execution from a technical standpoint, ensuring the team adheres to DataOps best practices, CI/CD, and robust data quality frameworks.

What We Look For

• Architecture Experience: 6+ years of professional experience in data engineering, with a focus on building solutions on distributed platforms and managing complex ETL/ELT pipelines.
• Lifecycle Management: Proven experience in architecting solutions that manage the full data lifecycle-ingestion, cleaning, and governance-at enterprise scale.
• Databricks Expertise: 3+ years of hands-on experience designing and deploying solutions within the Databricks ecosystem.
• Cloud Platforms: Deep experience with at least one major cloud provider ( Azure, AWS, or GCP ), particularly their data-related services.
• Languages: Proficiency in Python, Spark, PySpark, and/or their variants such as SparkSQL.

Desired Skills & Experience

• Orchestration: Hands-on knowledge of orchestrators such as Airflow or native Databricks tools like Databricks Workflows .
• Processing Frameworks: Advanced experience with frameworks such as dbt .
• Communication: Exceptional interpersonal and communication skills, with a proven ability to lead technical discussions with both developers and executive stakeholders.
• Innovation: Collaborate with our global innovation labs to implement proprietary accelerators and artifacts that reduce time-to-market for our clients.
• Mindset: A proactive, "Growth Mindset" with a passion for staying at the forefront of emerging data technologies.

Benefits of Working With Us

At SDG, we're constantly striving towards building a culture where people come first by creating an environment of transparency, collaboration, and meritocracy. We take care of our global employees by making sure we offer the following:

• Competitive Compensation: Full-time employment in an exciting and growing high-tech industry with a competitive salary.
• Career Growth: A professional career with opportunities for personal/professional growth and rapid advancement based on meritocracy.
• Feedback Loop: Regular meetings with your manager to share feedback, discuss new challenges, and review your career trajectory.
• Flexibility: A creative and innovative work environment with flexible working options (flexible working hours, good mix of on-site and remote).
• Health & Wellness: Robust Healthcare and Benefits package including Medical, Dental, Vision, Disability coverage, and various other benefit options.
• Retirement: 401k Retirement plan with Employer match .
• Time Off: 3+ weeks paid vacation and flexible holiday schedule.

Team Data Technologies Role Big Data Engineer Locations Chicago, Massachusetts, New Jersey Remote status Hybrid Employment type Full-time Employment level First /Mid-Level Officials