Data Engineer

Job Title: Data Engineer You will be responsible for: Data collection system architecture: Develop and implement a multi-source data collection framework that integrates with business systems (e.g., CRM, ERP, APP logs), third-party APIs (e.g., APIs, web crawlers), and offline files, ensuring real-time or near-real-time synchronization to guarantee data integrity and timeliness. Data storage architecture design: Based on business scenarios (such as real-time queries, offline analysis, and massive storage), select and deploy data storage systems (e.g., HDFS, HBase, ClickHouse, MySQL, Redis), optimize storage tiers (ODS, DW, DM), and balance storage costs with access efficiency. Data cleaning and preprocessing: Formulate data cleaning rules, process missing values, outliers, and duplicate data, complete data format conversion, coding unification, and field standardization, and output high-quality structured/semi-structured data to provide the foundation for downstream analysis and modeling. Data Pipeline Development and Maintenance: Utilizing tools like Spark, Flink, and Airflow, we develop automated ETL/ELT pipelines, write efficient data processing scripts (SQL, Python, Scala), conduct regular pipeline health checks, and resolve issues such as data delays and task failures. Data quality control: Establish a data quality monitoring system, define data quality indicators (such as accuracy, integrity, consistency, and timeliness), develop quality verification rules and alarm mechanisms, regularly output data quality reports, and promote business and technical teams to optimize data sources. Data Service Support: Provides data support for business units (e.g., operations, product, algorithms) by developing data query interfaces, reporting tools, and data visualization dashboards (e.g., Tableau, Power BI) as needed, assisting teams in data extraction, analysis, and interpretation. I want you to have: Education and Experience: Bachelor's degree or above in Computer Science, Data Science, Software Engineering, Statistics, or related fields. At least 3 years of experience as a data engineer. Candidates with experience in internet, finance, or big data platforms are preferred. Technical Skills: Proficient in SQL and familiar with at least one programming language (Python/Scala/Java); Expertise in mainstream data processing tools (Spark/Flink/Airflow) and storage systems (Hadoop ecosystem, ClickHouse, MySQL, Redis); Knowledge of data modeling techniques (dimensional modeling, entity modeling). Project Experience: Must have experience in building complete data systems (e.g., building data warehouses and real-time data platforms from scratch). Candidates with experience in large-scale data processing (TB/PB level) and data pipeline development in high-concurrency scenarios are preferred. Problem solving ability: Strong logical analysis and troubleshooting ability, able to quickly locate problems in data processing and pipeline operation and provide solutions, and keen insight into abnormal data quality. Collaboration and Communication: Have a good sense of cross-department collaboration, can clearly understand business needs and translate them into data solutions, efficiently connect with business, algorithm and R & D teams, and promote the implementation of data projects. Bonus points: Familiarity with cloud-native data architectures (e.g., AWS S3, Alibaba Cloud OSS, Kubernetes-based data services); Experience in building lakehouse systems and real-time data warehouses; Certification in big data (e.g., Cloudera CCA Spark & Hadoop Developer, AWS Data Analytics Specialty). We offer: Competitive salary and benefits. A challenging and growing work environment. #J-18808-Ljbffr