Torchlight Analytics LLC

Senior Data Engineer

Remote - Full Time

This is a 100% remote, TS/SCI required, mission-critical role focused on designing, building, and optimizing high-throughput, stateless data processing pipelines. The successful candidate will develop a portable, scalable architecture that produces high-quality data streams from complex rule sets. This data will be deployable on an infrastructure that is containerized, scales dynamically, and strictly adheres to government security protocols and quality standards.

Clearance: An active TS/SCI Security Clearance is required

Responsibilities

● Pipeline development and orchestration: Design, implement, and maintain complex data workflows that use an "Extract, Load, and Enrich" (ELE) or a similar methodology to create high-fidelity data streams from foundational rules. You'll need to use distributed processing frameworks to achieve high computational efficiency and throughput. This includes an expert-level proficiency in orchestration systems like Apache Airflow.

● Quantitative fidelity engineering: Collaborate with quantitative analysts and data scientists to operationalize complex probabilistic models and statistical distribution rules. This includes a critical understanding of geospatial trajectories and activity patterns.

● Architecture and scalability: Develop and maintain a resilient, portable, and infrastructure-agnostic data architecture. The design requires a stateless approach, optimized for in-memory processing. Expertise in containerization technologies like Docker and Kubernetes is mandatory for building single-container deployable solutions that can scale dynamically.

● DevSecOps and quality assurance: Implement robust Continuous Integration/Continuous Deployment (CI/CD) pipelines to automate deployment and testing. You will also need to define source-to-target mapping directly as pipeline code to ensure auditable and verifiable software baselines.

● Security and compliance integration: Seamlessly integrate data solutions with host-managed security, authentication, and authorization systems. Familiarity with relevant compliance frameworks for classified government operations (e.g., NIST 800-171) is a must.

Requirements

  • Education: A Bachelor's degree in Computer Science, Data Engineering, Mathematics, or a related quantitative or technical field is required.
  • Experience: 5+ years of demonstrated progressive experience in large-scale data engineering, specifically in pipeline development within secured or classified environments.
  • Programming: Expert proficiency in Python is mandatory for pipeline development. Advanced proficiency in other high-performance compiled languages like C, Go, or Rust is preferred.
  • Distributed data processing: Deep expertise in Apache Beam (mandatory), Apache Spark, or Dask for scalable stream/batch processing is required.
  • Workflow orchestration: Advanced proficiency in Apache Airflow (mandatory), Prefect, or Dagster for complex scheduling and monitoring is required.
  • Data modeling & storage: Expert SQL development and optimization skills are required, along with hands-on experience with both relational (e.g., PostgreSQL) and NoSQL databases (e.g., MongoDB).
  • Containerization & deployment: Expert-level experience with Docker and orchestration tools like Kubernetes or ECS/Fargate is mandatory.
  • Cloud/Infrastructure: Strong familiarity with cloud primitives (e.g., GCS, S3) or experience managing portable, secure on-premise/bare-metal environments.
  • Quality & CI/CD: Strong experience with CI/CD tools (e.g., GitHub Actions, Jenkins) and a commitment to unit and integration testing.

About Torchlight

Torchlight AI® provides Behavior Based Insights to decision-makers in government and corporate organizations to uncover emerging threats and new opportunities.

Torchlight is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran status, disability status, or any other protected class. U.S. Citizenship is required for positions requiring government security clearance and for certain other restricted positions.

Apply: Senior Data Engineer
* Required fields
First name*
Last name*
Email address*
Location
Phone number*
Resume*

Attach resume or Paste resume

Attach resume as .pdf, .doc, .docx, .odt, .txt, or .rtf (limit 5MB) or paste resume

Paste your resume here or attach resume file

The following questions are entirely optional.
To comply with government Equal Employment Opportunity and/or Affirmative Action reporting regulations, we are requesting (but NOT requiring) that you enter this personal data. This information will not be used in connection with any employment decisions, and will be used solely as permitted by state and federal law. Your voluntary cooperation would be appreciated. Learn more.
Gender
Race/Ethnicity
Invitation for Job Applicants to Self-Identify as a U.S. Veteran
  • A “disabled veteran” is one of the following:
    • a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or
    • a person who was discharged or released from active duty because of a service-connected disability.
  • A “recently separated veteran” means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.
  • An “active duty wartime or campaign badge veteran” means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.
  • An “Armed forces service medal veteran” means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.
Veteran status
I IDENTIFY AS ONE OR MORE OF THE CLASSIFICATIONS OF PROTECTED VETERAN LISTED ABOVE
I AM NOT A PROTECTED VETERAN
I DON’T WISH TO ANSWER
Voluntary Self-Identification of Disability
Voluntary Self-Identification of Disability Form CC-305
OMB Control Number 1250-0005
Expires 04/30/2026
Why are you being asked to complete this form?

We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.

Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp.

How do you know if you have a disability?

A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:

  • Alcohol or other substance use disorder (not currently using drugs illegally)
  • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
  • Blind or low vision
  • Cancer (past or present)
  • Cardiovascular or heart disease
  • Celiac disease
  • Cerebral palsy
  • Deaf or serious difficulty hearing
  • Diabetes
  • Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
  • Epilepsy or other seizure disorder
  • Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
  • Intellectual or developmental disability
  • Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
  • Missing limbs or partially missing limbs
  • Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
  • Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS)
  • Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
  • Partial or complete paralysis (any cause)
  • Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
  • Short stature (dwarfism)
  • Traumatic brain injury
Please check one of the boxes below:
YES, I HAVE A DISABILITY, OR HAVE HAD ONE IN THE PAST
NO, I DO NOT HAVE A DISABILITY AND HAVE NOT HAD ONE IN THE PAST
I DO NOT WANT TO ANSWER

PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.

Name Date
Human Check*